task 5.2 · chapter 5 of 8: regex and textfsm · 40 min
A template of your own
objective
ntc-templates has no template for show ntp associations on cisco_ios. Write one in TextFSM, so every server comes back as a dict with its address, reference clock, stratum and reach.
templates/ntp_associations.textfsmnetlab/parsers.pyopen a file in VS Code: Ctrl+P (Cmd+P on a Mac), type its name, Enter
| The output | checks/data/ntp-r1.txt and ntp-sw1.txt: show ntp associations from r1 and sw1 |
|---|---|
| Your template | templates/ntp_associations.textfsm: a starting skeleton that’s wrong on purpose |
| Your function | parse_ntp_associations() in netlab/parsers.py (TODO 5.2) |
| The keys | lower case, and at least address, ref_clock, stratum and reach |
new to this? step by stepbeginner
- Open
checks/data/ntp-r1.txtandtemplates/ntp_associations.textfsmside by side. - Write
parse_ntp_associations()from the follow-along: four lines that run your template. - Try it on r1’s output in the REPL: the skeleton finds the header and the legend as servers.
- Challenge: fix the template, a Value per column and a state for the server lines, then
make check T=5.2.
more in meet the lab
Values, states and rules
When there’s no template
parse_output() from ntc-templates raises an exception when it has nothing for a command:
ntc_templates.parse.ParsingException: Unable to parse command "show ntp associations" on platform cisco_ios - No template found for attributes: "{'Command': 'show ntp associations', 'Platform': 'cisco_ios'}"
ntc-templates’ own templates are TextFSM files like the one you’re writing, so textfsm can run yours the same
way.
A template
Lesson 5 used this one, for two columns of show ip interface brief:
Value INTERFACE (\S+)
Value STATUS (up|down|administratively down)
Start
^${INTERFACE}\s+\S+\s+(YES|NO)\s+\S+\s+${STATUS}\s -> Record
A template has two parts:
- Values:
Value NAME (regex), one per column you want.Requiredbefore the name drops any row without that value. - States: a name on its own line, then indented rules, each a regex starting with
^.${NAME}in a rule stands for that Value’s regex, and saves what it matches. Every template starts inStart.
TextFSM reads a line at a time and tries the current state’s rules in order; the first that matches wins, and a
line that matches nothing is skipped. After -> comes what to do: Record saves the values as one row,
a state’s name moves to that state, and Error stops with an error. $$ is a literal end of line (one $
starts a value’s name).
in one sentence Values say what a column looks like; states and rules say which lines are rows.
The output
address ref clock st when poll reach delay offset disp
*~172.80.111.12 .LOCL. 3 22 64 17 0.621 -0.159 0.002
~172.80.111.99 .INIT. 16 - 128 0 0.000 0.000 0.000
* sys.peer, # selected, + candidate, - outlyer, x falseticker, ~ configured
A header, one line per server, and a legend. Each server line starts with two characters, the tally: *
for the server the clock is synced to, then ~ for “configured”. 172.80.111.12 is r2, which serves time
(ntp master 3 in its config); nothing answers at 172.80.111.99, so it’s stuck at stratum 16 with a reach of 0.
in one sentence Skip the header, record each server line, and don’t mistake the legend for a server.
What the skeleton finds
The starting template has one Value and one rule:
Value ADDRESS (\S+)
Start
^\s*${ADDRESS} -> Record
With parse_ntp_associations() written, on r1’s output, it gives four “servers” for two:
>>> from netlab.parsers import parse_ntp_associations
>>> parse_ntp_associations(open("checks/data/ntp-r1.txt").read())
[{'address': 'address'}, {'address': '*~172.80.111.12'}, {'address': '~172.80.111.99'}, {'address': '*'}]
\S+ matches any word, so the header’s address is a row, the tally is stuck to each address, and the
legend’s * is a row too. The check says the same, more briefly:
checks/test_5_2.py::test_template_exists PASSED [ 20%]
checks/test_5_2.py::test_the_keys FAILED [ 40%]
checks/test_5_2.py::test_real_outputs FAILED [ 60%]
...
E AssertionError: each row needs the keys address, ref_clock, stratum, reach (Values in capitals become lower case). Missing: ['ref_clock', 'stratum', 'reach']
...
E AssertionError: ntp-r1.txt has 2 servers, the template found 4
Run the template from Python
Write
parse_ntp_associations(). It opens the template, runs it over the text, and turns each row into a dict with lower-case keys:editnetlab/parsers.pyCtrl+P → parsers.pydef parse_ntp_associations(text): """`show ntp associations` (CapeOS Bolt), parsed by your template templates/ntp_associations.textfsm. ntc-templates has no template for this command, so you write one. It returns a list of dicts, one per server, with keys in lower case. These four must be there: address, ref_clock, stratum and reach. The header line and the legend at the bottom ("* sys.peer, # selected, ...") aren't servers. """ with open(TEMPLATES / "ntp_associations.textfsm") as f: fsm = textfsm.TextFSM(f) keys = [h.lower() for h in fsm.header] return [dict(zip(keys, row)) for row in fsm.ParseText(text)]TEMPLATESis set at the top of the module: the repository’stemplates/folder.fsm.headeris the list of Value names,fsm.ParseText()a list of rows in the same order, andzip()pairs them up.Try it in a new REPL, and get the four wrong rows above:
>>> from netlab.parsers import parse_ntp_associations >>> parse_ntp_associations(open("checks/data/ntp-r1.txt").read())The function reads the template every time it’s called, so from now on you can edit the template, save it, and run the same line again, without restarting the REPL.
in one sentence The Python part is four lines; the parsing lives in the template.
The template
Rewrite templates/ntp_associations.textfsm so it returns one row per server, and nothing for the header or
the legend. It needs a Value for at least ADDRESS, REF_CLOCK, STRATUM and REACH, and the address
without the tally. With no servers configured, only the header and the legend are printed, and the result
should be [].
hint
Give ADDRESS a stricter regex than \S+: four groups of digits with dots, \d+\.\d+\.\d+\.\d+. Then
neither address nor * can match it.
hint
Use two states. In Start, a rule that matches the header line moves to a state of your own, say Servers,
and records nothing. In Servers, one rule matches a whole server line, column by column with \s+ between,
and ends -> Record.
hint
The tally is up to two characters before the address: *~, +~, -~, x~, or a space and ~. A Value
for it, like ([*#+\-x ]?~?), keeps them out of the address. Mark ADDRESS as Required, so a line
without one can never become a row.
show the solutiontemplates/ntp_associations.textfsm
or pull it into your lab: make solve T=5.2, then git diff --staged shows what changed
Value Required ADDRESS (\d+\.\d+\.\d+\.\d+)
Value TALLY ([*#+\-x ]?~?)
Value REF_CLOCK (\S+)
Value STRATUM (\d+)
Value WHEN (\S+)
Value POLL (\d+)
Value REACH (\d+)
Value DELAY (\S+)
Value OFFSET (\S+)
Value DISP (\S+)
Start
^\s+address\s+ref clock -> Servers
Servers
^${TALLY}${ADDRESS}\s+${REF_CLOCK}\s+${STRATUM}\s+${WHEN}\s+${POLL}\s+${REACH}\s+${DELAY}\s+${OFFSET}\s+${DISP}\s*$$ -> Record
^\s+\*\s+sys\.peer
^\s*$$
^. -> Error
>>> parse_ntp_associations(open("checks/data/ntp-r1.txt").read())
[{'address': '172.80.111.12', 'tally': '*~', 'ref_clock': '.LOCL.', 'stratum': '3', 'when': '22', 'poll': '64', 'reach': '17', 'delay': '0.621', 'offset': '-0.159', 'disp': '0.002'}, {'address': '172.80.111.99', 'tally': ' ~', 'ref_clock': '.INIT.', 'stratum': '16', 'when': '-', 'poll': '128', 'reach': '0', 'delay': '0.000', 'offset': '0.000', 'disp': '0.000'}]
Start skips everything until the header, then hands over to Servers. There, a server line is recorded,
the legend and blank lines match a rule that does nothing, and anything else is an Error. That last rule is
the strict part: if a new release adds a column or a kind of line, the template stops with
TextFSMError: State Error raised and names the line, instead of quietly returning rows that are wrong. WHEN
is \S+ because a server that’s never answered shows -. Every value comes back as a string, '3' and not
3: TextFSM finds text, and turning it into numbers is your code’s job. The check runs the template on 40
made-up outputs with up to five servers each, references that are addresses as well as .GPS. and .LOCL.,
and one with no servers at all.
5 check: run it, then prove it
$ make check T=5.2when it's right, you'll see
task 5.2 checks/test_5_2.py
checks/test_5_2.py::test_template_exists PASSED [ 20%]
checks/test_5_2.py::test_the_keys PASSED [ 40%]
checks/test_5_2.py::test_real_outputs PASSED [ 60%]
checks/test_5_2.py::test_unseen_outputs PASSED [ 80%]
checks/test_5_2.py::test_no_servers PASSED [100%]
============================== 5 passed in 0.01s ===============================
✓ task 5.2 done