online posts 16
python 101 for network engineers

task 5.1 · chapter 5 of 8: regex and textfsm · 30 min

One table, two parsers

objective

Parse show ip interface brief with a regular expression of your own, then with ntc-templates, and make the two give exactly the same list of dicts.

you edit, in the labnetlab/parsers.py

open a file in VS Code: Ctrl+P (Cmd+P on a Mac), type its name, Enter

given
The outputchecks/data/brief-r1.txt and brief-sw1.txt: show ip interface brief from r1 and sw1
Your codenetlab/parsers.py: BRIEF_LINE (TODO 5.1a), parse_brief_regex() (5.1b), parse_brief_ntc() (5.1c)
The shape{"interface": ..., "ip_address": ... or None, "status": ..., "protocol": ...}, as in chapter 2
Librariesre (standard library), and parse_output from ntc-templates 9.3.0
new to this? step by stepbeginner
  1. Open checks/data/brief-r1.txt and netlab/parsers.py side by side.
  2. Write BRIEF_LINE, a regex with a named group per column, and try it in the REPL on one line.
  3. Write parse_brief_regex(): the regex on every line, a dict for each line that matches.
  4. Challenge: parse_brief_ntc(), the same rows from ntc-templates. Then make check T=5.1.

more in meet the lab

1 explain

A pattern per line, or a template per command

A regular expression

A regex is a pattern for text. \S+ is a run of non-space characters (a word), \s+ is a run of spaces, up|down is either one, and ^ and $ are the start and end of the line. Parentheses make a group, and (?P<name>...) a named group: the part of the line you want back, under that name.

m = BRIEF_LINE.match(line)      # a match object, or None if the line doesn't fit
m["interface"]                  # one named group's text

Write patterns as raw strings, r"...", so Python leaves the backslashes for re.

in one sentence One named group per column, and match() on each line: a line that doesn’t fit, like the header, gives None.

The tricky column

Status can be two words, administratively down, so \S+ isn’t enough there. List the three values it can have: (?P<status>up|down|administratively down). The OK? column is always YES or NO, and being that strict is what stops the header line from matching.

ntc-templates

ntc-templates is a library of TextFSM templates for common commands. parse_output() picks one from the platform and the command, and returns a list of dicts:

from ntc_templates.parse import parse_output
parse_output(platform="cisco_ios", command="show ip interface brief", data=text)

CapeOS Bolt prints this command the way the platform ntc-templates calls cisco_ios does, so that’s the platform. Its keys are its own, though, so you map them to the lab’s.

in one sentence Two parsers, one shape: that’s what lets you compare them, and swap one for the other later.

2 show

One line, the pattern, the rows

split() can’t tell where a two-word status ends:

>>> line = "GigabitEthernet4       unassigned      YES unset  administratively down down"
>>> line.split()
['GigabitEthernet4', 'unassigned', 'YES', 'unset', 'administratively', 'down', 'down']

A regex with a group per column can. With BRIEF_LINE and parse_brief_regex() written, r1’s whole table:

>>> from netlab.parsers import parse_brief_regex
>>> text = open("checks/data/brief-r1.txt").read()
>>> for row in parse_brief_regex(text):
...     print(row)
...
{'interface': 'GigabitEthernet0', 'ip_address': '172.80.111.11', 'status': 'up', 'protocol': 'up'}
{'interface': 'GigabitEthernet1', 'ip_address': '192.168.10.1', 'status': 'up', 'protocol': 'up'}
{'interface': 'GigabitEthernet2', 'ip_address': '10.0.12.1', 'status': 'up', 'protocol': 'up'}
{'interface': 'GigabitEthernet3', 'ip_address': '10.0.13.1', 'status': 'up', 'protocol': 'up'}
{'interface': 'GigabitEthernet4', 'ip_address': None, 'status': 'administratively down', 'protocol': 'down'}
{'interface': 'Loopback0', 'ip_address': '10.255.0.1', 'status': 'up', 'protocol': 'up'}

And what ntc-templates gives for the same text, before any mapping:

>>> from ntc_templates.parse import parse_output
>>> parse_output(platform="cisco_ios", command="show ip interface brief", data=text)[:2]
[{'interface': 'GigabitEthernet0', 'ip_address': '172.80.111.11', 'status': 'up', 'proto': 'up'}, {'interface': 'GigabitEthernet1', 'ip_address': '192.168.10.1', 'status': 'up', 'proto': 'up'}]
3 follow along

The regex parser

  1. Build the pattern a column at a time, in the REPL, on the hardest line:

    >>> import re
    >>> line = "GigabitEthernet4       unassigned      YES unset  administratively down down"
    >>> m = re.match(r"(\S+)\s+(\S+)", line)
    >>> m.groups()
    ('GigabitEthernet4', 'unassigned')
    

    Two columns. Keep adding a \s+ and a group until all six are there, then name them.

  2. Put the finished pattern in the module. It’s split over two lines: Python joins strings that sit next to each other inside parentheses.

    editnetlab/parsers.pyCtrl+P → parsers.py
    BRIEF_LINE = re.compile(
        r"^(?P<interface>\S+)\s+(?P<ip_address>\S+)\s+(?P<ok>YES|NO)\s+(?P<method>\S+)\s+"
        r"(?P<status>up|down|administratively down)\s+(?P<protocol>\S+)\s*$"
    )
    

    \s*$ at the end allows trailing spaces and nothing else. re.compile() builds the pattern once, so the loop doesn’t rebuild it on every line.

  3. Write parse_brief_regex(). Every line that matches becomes a dict with the four keys; unassigned becomes None, as in chapter 2:

    editnetlab/parsers.pyCtrl+P → parsers.py
    def parse_brief_regex(text):
        """`show ip interface brief` (CapeOS Bolt) with BRIEF_LINE, as a list of dicts (see the top of this file)."""
        rows = []
        for line in text.splitlines():
            m = BRIEF_LINE.match(line)
            if m:
                address = m["ip_address"]
                rows.append({
                    "interface": m["interface"],
                    "ip_address": None if address == "unassigned" else address,
                    "status": m["status"],
                    "protocol": m["protocol"],
                })
        return rows
    

    if m: is what skips the header and blank lines: match() returns None for them.

  4. In a new REPL, check r1’s table looks like the one above. make check T=5.1 now passes test_regex_unseen, and fails the other two on TODO 5.1c.

in one sentence A strict pattern and if m: together decide which lines are rows.

4 challenge

parse_brief_ntc()

Write parse_brief_ntc(text, platform="cisco_ios"): the same table through ntc-templates, returning exactly what parse_brief_regex() returns for the same text. Same four keys, same values, same order.

hint

parse_output(platform=platform, command="show ip interface brief", data=text) gives a list of dicts. Loop over it and build a new dict for each row.

hint

Compare one row of each in the REPL. Two things differ: ntc-templates calls the protocol column proto, and it leaves unassigned as it is.

show the solutionnetlab/parsers.py

or pull it into your lab: make solve T=5.1, then git diff --staged shows what changed

def parse_brief_ntc(text, platform="cisco_ios"):
    """The same table, parsed by ntc-templates' template for `show ip interface brief` on this platform.

    ntc-templates calls the protocol column "proto". Return the same keys and values as
    parse_brief_regex, so the two can be compared.
    """
    rows = []
    for r in parse_output(platform=platform, command="show ip interface brief", data=text):
        rows.append({
            "interface": r["interface"],
            "ip_address": None if r["ip_address"] == "unassigned" else r["ip_address"],
            "status": r["status"],
            "protocol": r.get("proto", r.get("protocol")),
        })
    return rows

The check runs the regex on outputs it makes up fresh each time, blank lines included, and ntc-templates on more of them, with sub-interfaces, loopbacks and VLAN interfaces. Then it runs both on the recorded r1, r2 and sw1 outputs in checks/data/ and expects them to agree row for row. Notice what each one needed from you: the regex, a pattern you had to get right; ntc-templates, two renamed fields. Someone else had already got the pattern right.

5 check: run it, then prove it

$ make check T=5.1

when it's right, you'll see

task 5.1  checks/test_5_1.py
checks/test_5_1.py::test_regex_unseen PASSED                             [ 33%]
checks/test_5_1.py::test_ntc_unseen PASSED                               [ 66%]
checks/test_5_1.py::test_they_agree_on_real_output PASSED                [100%]
============================== 3 passed in 0.09s ===============================
✓ task 5.1 done