task 5.1 · chapter 5 of 8: regex and textfsm · 30 min
One table, two parsers
objective
Parse show ip interface brief with a regular expression of your own, then with ntc-templates, and make the two give exactly the same list of dicts.
netlab/parsers.pyopen a file in VS Code: Ctrl+P (Cmd+P on a Mac), type its name, Enter
| The output | checks/data/brief-r1.txt and brief-sw1.txt: show ip interface brief from r1 and sw1 |
|---|---|
| Your code | netlab/parsers.py: BRIEF_LINE (TODO 5.1a), parse_brief_regex() (5.1b), parse_brief_ntc() (5.1c) |
| The shape | {"interface": ..., "ip_address": ... or None, "status": ..., "protocol": ...}, as in chapter 2 |
| Libraries | re (standard library), and parse_output from ntc-templates 9.3.0 |
new to this? step by stepbeginner
- Open
checks/data/brief-r1.txtandnetlab/parsers.pyside by side. - Write
BRIEF_LINE, a regex with a named group per column, and try it in the REPL on one line. - Write
parse_brief_regex(): the regex on every line, a dict for each line that matches. - Challenge:
parse_brief_ntc(), the same rows from ntc-templates. Thenmake check T=5.1.
more in meet the lab
A pattern per line, or a template per command
A regular expression
A regex is a pattern for text. \S+ is a run of non-space characters (a word), \s+ is a run of spaces,
up|down is either one, and ^ and $ are the start and end of the line. Parentheses make a group, and
(?P<name>...) a named group: the part of the line you want back, under that name.
m = BRIEF_LINE.match(line) # a match object, or None if the line doesn't fit
m["interface"] # one named group's text
Write patterns as raw strings, r"...", so Python leaves the backslashes for re.
in one sentence One named group per column, and match() on each line: a line that doesn’t fit, like the header, gives None.
The tricky column
Status can be two words, administratively down, so \S+ isn’t enough there. List the three values it can
have: (?P<status>up|down|administratively down). The OK? column is always YES or NO, and being that
strict is what stops the header line from matching.
ntc-templates
ntc-templates is a library of TextFSM templates for common commands. parse_output() picks one from the
platform and the command, and returns a list of dicts:
from ntc_templates.parse import parse_output
parse_output(platform="cisco_ios", command="show ip interface brief", data=text)
CapeOS Bolt prints this command the way the platform ntc-templates calls cisco_ios does, so that’s the
platform. Its keys are its own, though, so you map them to the lab’s.
in one sentence Two parsers, one shape: that’s what lets you compare them, and swap one for the other later.
One line, the pattern, the rows
split() can’t tell where a two-word status ends:
>>> line = "GigabitEthernet4 unassigned YES unset administratively down down"
>>> line.split()
['GigabitEthernet4', 'unassigned', 'YES', 'unset', 'administratively', 'down', 'down']
A regex with a group per column can. With BRIEF_LINE and parse_brief_regex() written, r1’s whole table:
>>> from netlab.parsers import parse_brief_regex
>>> text = open("checks/data/brief-r1.txt").read()
>>> for row in parse_brief_regex(text):
... print(row)
...
{'interface': 'GigabitEthernet0', 'ip_address': '172.80.111.11', 'status': 'up', 'protocol': 'up'}
{'interface': 'GigabitEthernet1', 'ip_address': '192.168.10.1', 'status': 'up', 'protocol': 'up'}
{'interface': 'GigabitEthernet2', 'ip_address': '10.0.12.1', 'status': 'up', 'protocol': 'up'}
{'interface': 'GigabitEthernet3', 'ip_address': '10.0.13.1', 'status': 'up', 'protocol': 'up'}
{'interface': 'GigabitEthernet4', 'ip_address': None, 'status': 'administratively down', 'protocol': 'down'}
{'interface': 'Loopback0', 'ip_address': '10.255.0.1', 'status': 'up', 'protocol': 'up'}
And what ntc-templates gives for the same text, before any mapping:
>>> from ntc_templates.parse import parse_output
>>> parse_output(platform="cisco_ios", command="show ip interface brief", data=text)[:2]
[{'interface': 'GigabitEthernet0', 'ip_address': '172.80.111.11', 'status': 'up', 'proto': 'up'}, {'interface': 'GigabitEthernet1', 'ip_address': '192.168.10.1', 'status': 'up', 'proto': 'up'}]
The regex parser
Build the pattern a column at a time, in the REPL, on the hardest line:
>>> import re >>> line = "GigabitEthernet4 unassigned YES unset administratively down down" >>> m = re.match(r"(\S+)\s+(\S+)", line) >>> m.groups() ('GigabitEthernet4', 'unassigned')Two columns. Keep adding a
\s+and a group until all six are there, then name them.Put the finished pattern in the module. It’s split over two lines: Python joins strings that sit next to each other inside parentheses.
editnetlab/parsers.pyCtrl+P → parsers.pyBRIEF_LINE = re.compile( r"^(?P<interface>\S+)\s+(?P<ip_address>\S+)\s+(?P<ok>YES|NO)\s+(?P<method>\S+)\s+" r"(?P<status>up|down|administratively down)\s+(?P<protocol>\S+)\s*$" )\s*$at the end allows trailing spaces and nothing else.re.compile()builds the pattern once, so the loop doesn’t rebuild it on every line.Write
parse_brief_regex(). Every line that matches becomes a dict with the four keys;unassignedbecomesNone, as in chapter 2:editnetlab/parsers.pyCtrl+P → parsers.pydef parse_brief_regex(text): """`show ip interface brief` (CapeOS Bolt) with BRIEF_LINE, as a list of dicts (see the top of this file).""" rows = [] for line in text.splitlines(): m = BRIEF_LINE.match(line) if m: address = m["ip_address"] rows.append({ "interface": m["interface"], "ip_address": None if address == "unassigned" else address, "status": m["status"], "protocol": m["protocol"], }) return rowsif m:is what skips the header and blank lines:match()returnsNonefor them.In a new REPL, check r1’s table looks like the one above.
make check T=5.1now passestest_regex_unseen, and fails the other two onTODO 5.1c.
in one sentence A strict pattern and if m: together decide which lines are rows.
parse_brief_ntc()
Write parse_brief_ntc(text, platform="cisco_ios"): the same table through ntc-templates, returning exactly
what parse_brief_regex() returns for the same text. Same four keys, same values, same order.
hint
parse_output(platform=platform, command="show ip interface brief", data=text) gives a list of dicts. Loop
over it and build a new dict for each row.
hint
Compare one row of each in the REPL. Two things differ: ntc-templates calls the protocol column proto, and
it leaves unassigned as it is.
show the solutionnetlab/parsers.py
or pull it into your lab: make solve T=5.1, then git diff --staged shows what changed
def parse_brief_ntc(text, platform="cisco_ios"):
"""The same table, parsed by ntc-templates' template for `show ip interface brief` on this platform.
ntc-templates calls the protocol column "proto". Return the same keys and values as
parse_brief_regex, so the two can be compared.
"""
rows = []
for r in parse_output(platform=platform, command="show ip interface brief", data=text):
rows.append({
"interface": r["interface"],
"ip_address": None if r["ip_address"] == "unassigned" else r["ip_address"],
"status": r["status"],
"protocol": r.get("proto", r.get("protocol")),
})
return rows
The check runs the regex on outputs it makes up fresh each time, blank lines included, and ntc-templates on
more of them, with sub-interfaces, loopbacks and VLAN interfaces. Then it runs both on the recorded r1, r2 and
sw1 outputs in checks/data/ and expects them to agree row for row. Notice what each one needed from you: the
regex, a pattern you had to get right; ntc-templates, two renamed fields. Someone else had already got the
pattern right.
5 check: run it, then prove it
$ make check T=5.1when it's right, you'll see
task 5.1 checks/test_5_1.py
checks/test_5_1.py::test_regex_unseen PASSED [ 33%]
checks/test_5_1.py::test_ntc_unseen PASSED [ 66%]
checks/test_5_1.py::test_they_agree_on_real_output PASSED [100%]
============================== 3 passed in 0.09s ===============================
✓ task 5.1 done