online posts 16
python 101 for network engineers

task 5.2 · chapter 5 of 8: regex and textfsm · 40 min

A template of your own

objective

ntc-templates has no template for show ntp associations on cisco_ios. Write one in TextFSM, so every server comes back as a dict with its address, reference clock, stratum and reach.

you edit, in the labtemplates/ntp_associations.textfsmnetlab/parsers.py

open a file in VS Code: Ctrl+P (Cmd+P on a Mac), type its name, Enter

given
The outputchecks/data/ntp-r1.txt and ntp-sw1.txt: show ntp associations from r1 and sw1
Your templatetemplates/ntp_associations.textfsm: a starting skeleton that’s wrong on purpose
Your functionparse_ntp_associations() in netlab/parsers.py (TODO 5.2)
The keyslower case, and at least address, ref_clock, stratum and reach
new to this? step by stepbeginner
  1. Open checks/data/ntp-r1.txt and templates/ntp_associations.textfsm side by side.
  2. Write parse_ntp_associations() from the follow-along: four lines that run your template.
  3. Try it on r1’s output in the REPL: the skeleton finds the header and the legend as servers.
  4. Challenge: fix the template, a Value per column and a state for the server lines, then make check T=5.2.

more in meet the lab

1 explain

Values, states and rules

When there’s no template

parse_output() from ntc-templates raises an exception when it has nothing for a command:

ntc_templates.parse.ParsingException: Unable to parse command "show ntp associations" on platform cisco_ios - No template found for attributes: "{'Command': 'show ntp associations', 'Platform': 'cisco_ios'}"

ntc-templates’ own templates are TextFSM files like the one you’re writing, so textfsm can run yours the same way.

A template

Lesson 5 used this one, for two columns of show ip interface brief:

Value INTERFACE (\S+)
Value STATUS (up|down|administratively down)

Start
  ^${INTERFACE}\s+\S+\s+(YES|NO)\s+\S+\s+${STATUS}\s -> Record

A template has two parts:

  • Values: Value NAME (regex), one per column you want. Required before the name drops any row without that value.
  • States: a name on its own line, then indented rules, each a regex starting with ^. ${NAME} in a rule stands for that Value’s regex, and saves what it matches. Every template starts in Start.

TextFSM reads a line at a time and tries the current state’s rules in order; the first that matches wins, and a line that matches nothing is skipped. After -> comes what to do: Record saves the values as one row, a state’s name moves to that state, and Error stops with an error. $$ is a literal end of line (one $ starts a value’s name).

in one sentence Values say what a column looks like; states and rules say which lines are rows.

The output

  address         ref clock       st   when   poll reach  delay  offset   disp
*~172.80.111.12   .LOCL.           3     22     64    17  0.621  -0.159  0.002
 ~172.80.111.99   .INIT.          16      -    128     0  0.000   0.000  0.000
 * sys.peer, # selected, + candidate, - outlyer, x falseticker, ~ configured

A header, one line per server, and a legend. Each server line starts with two characters, the tally: * for the server the clock is synced to, then ~ for “configured”. 172.80.111.12 is r2, which serves time (ntp master 3 in its config); nothing answers at 172.80.111.99, so it’s stuck at stratum 16 with a reach of 0.

in one sentence Skip the header, record each server line, and don’t mistake the legend for a server.

2 show

What the skeleton finds

The starting template has one Value and one rule:

Value ADDRESS (\S+)

Start
  ^\s*${ADDRESS} -> Record

With parse_ntp_associations() written, on r1’s output, it gives four “servers” for two:

>>> from netlab.parsers import parse_ntp_associations
>>> parse_ntp_associations(open("checks/data/ntp-r1.txt").read())
[{'address': 'address'}, {'address': '*~172.80.111.12'}, {'address': '~172.80.111.99'}, {'address': '*'}]

\S+ matches any word, so the header’s address is a row, the tally is stuck to each address, and the legend’s * is a row too. The check says the same, more briefly:

checks/test_5_2.py::test_template_exists PASSED                          [ 20%]
checks/test_5_2.py::test_the_keys FAILED                                 [ 40%]
checks/test_5_2.py::test_real_outputs FAILED                             [ 60%]
...
E   AssertionError: each row needs the keys address, ref_clock, stratum, reach (Values in capitals become lower case). Missing: ['ref_clock', 'stratum', 'reach']
...
E   AssertionError: ntp-r1.txt has 2 servers, the template found 4
3 follow along

Run the template from Python

  1. Write parse_ntp_associations(). It opens the template, runs it over the text, and turns each row into a dict with lower-case keys:

    editnetlab/parsers.pyCtrl+P → parsers.py
    def parse_ntp_associations(text):
        """`show ntp associations` (CapeOS Bolt), parsed by your template templates/ntp_associations.textfsm.
    
        ntc-templates has no template for this command, so you write one. It returns a list of dicts, one
        per server, with keys in lower case. These four must be there: address, ref_clock, stratum and
        reach. The header line and the legend at the bottom ("* sys.peer, # selected, ...") aren't servers.
        """
        with open(TEMPLATES / "ntp_associations.textfsm") as f:
            fsm = textfsm.TextFSM(f)
        keys = [h.lower() for h in fsm.header]
        return [dict(zip(keys, row)) for row in fsm.ParseText(text)]
    

    TEMPLATES is set at the top of the module: the repository’s templates/ folder. fsm.header is the list of Value names, fsm.ParseText() a list of rows in the same order, and zip() pairs them up.

  2. Try it in a new REPL, and get the four wrong rows above:

    >>> from netlab.parsers import parse_ntp_associations
    >>> parse_ntp_associations(open("checks/data/ntp-r1.txt").read())
    

    The function reads the template every time it’s called, so from now on you can edit the template, save it, and run the same line again, without restarting the REPL.

in one sentence The Python part is four lines; the parsing lives in the template.

4 challenge

The template

Rewrite templates/ntp_associations.textfsm so it returns one row per server, and nothing for the header or the legend. It needs a Value for at least ADDRESS, REF_CLOCK, STRATUM and REACH, and the address without the tally. With no servers configured, only the header and the legend are printed, and the result should be [].

hint

Give ADDRESS a stricter regex than \S+: four groups of digits with dots, \d+\.\d+\.\d+\.\d+. Then neither address nor * can match it.

hint

Use two states. In Start, a rule that matches the header line moves to a state of your own, say Servers, and records nothing. In Servers, one rule matches a whole server line, column by column with \s+ between, and ends -> Record.

hint

The tally is up to two characters before the address: *~, +~, -~, x~, or a space and ~. A Value for it, like ([*#+\-x ]?~?), keeps them out of the address. Mark ADDRESS as Required, so a line without one can never become a row.

show the solutiontemplates/ntp_associations.textfsm

or pull it into your lab: make solve T=5.2, then git diff --staged shows what changed

Value Required ADDRESS (\d+\.\d+\.\d+\.\d+)
Value TALLY ([*#+\-x ]?~?)
Value REF_CLOCK (\S+)
Value STRATUM (\d+)
Value WHEN (\S+)
Value POLL (\d+)
Value REACH (\d+)
Value DELAY (\S+)
Value OFFSET (\S+)
Value DISP (\S+)

Start
  ^\s+address\s+ref clock -> Servers

Servers
  ^${TALLY}${ADDRESS}\s+${REF_CLOCK}\s+${STRATUM}\s+${WHEN}\s+${POLL}\s+${REACH}\s+${DELAY}\s+${OFFSET}\s+${DISP}\s*$$ -> Record
  ^\s+\*\s+sys\.peer
  ^\s*$$
  ^. -> Error
>>> parse_ntp_associations(open("checks/data/ntp-r1.txt").read())
[{'address': '172.80.111.12', 'tally': '*~', 'ref_clock': '.LOCL.', 'stratum': '3', 'when': '22', 'poll': '64', 'reach': '17', 'delay': '0.621', 'offset': '-0.159', 'disp': '0.002'}, {'address': '172.80.111.99', 'tally': ' ~', 'ref_clock': '.INIT.', 'stratum': '16', 'when': '-', 'poll': '128', 'reach': '0', 'delay': '0.000', 'offset': '0.000', 'disp': '0.000'}]

Start skips everything until the header, then hands over to Servers. There, a server line is recorded, the legend and blank lines match a rule that does nothing, and anything else is an Error. That last rule is the strict part: if a new release adds a column or a kind of line, the template stops with TextFSMError: State Error raised and names the line, instead of quietly returning rows that are wrong. WHEN is \S+ because a server that’s never answered shows -. Every value comes back as a string, '3' and not 3: TextFSM finds text, and turning it into numbers is your code’s job. The check runs the template on 40 made-up outputs with up to five servers each, references that are addresses as well as .GPS. and .LOCL., and one with no servers at all.

5 check: run it, then prove it

$ make check T=5.2

when it's right, you'll see

task 5.2  checks/test_5_2.py
checks/test_5_2.py::test_template_exists PASSED                          [ 20%]
checks/test_5_2.py::test_the_keys PASSED                                 [ 40%]
checks/test_5_2.py::test_real_outputs PASSED                             [ 60%]
checks/test_5_2.py::test_unseen_outputs PASSED                           [ 80%]
checks/test_5_2.py::test_no_servers PASSED                               [100%]
============================== 5 passed in 0.01s ===============================
✓ task 5.2 done