online posts 16
python 101 for network engineers

task 8.1 · chapter 8 of 8: capstone: an inventory to csv · 40 min

Every device, one CSV

objective

Write the three steps scripts/inventory.py runs: load the inventory, turn one device’s show version into a row, and write the rows with csv.DictWriter. Then make the row come out the same for r3, the Pulse router.

you edit, in the labnetlab/report.py

open a file in VS Code: Ctrl+P (Cmd+P on a Mac), type its name, Enter

given
The scriptscripts/inventory.py: given. It calls your functions in netlab/report.py
From earlier chaptersconnect() and DeviceError (6.1, 6.2), load_inventory() (7.1), get_devices() and api_to_inventory() (7.2)
The inventoryinventory.yml, or the API at LAB_API_URL with LAB_API_TOKEN (both in .env)
The outputinventory.csv: hostname, model, version, serial, uptime, ip_interfaces
new to this? step by stepbeginner
  1. Starting the lab at chapter 8? make skip CH=8 solves chapters 1 to 7 first: this task calls their functions.
  2. In netlab/report.py, write load_devices(), write_csv() and facts() (TODO 8.1a, 8.1c, 8.1b).
  3. Try facts() in the REPL on r1. Challenge: make it work on r3 (arista_eos) too.
  4. python scripts/inventory.py, cat inventory.csv, then make check T=8.1.

more in meet the lab

1 explain

Inventory in, one function per step, results out

The shape of the tool

scripts/inventory.py is given, and its main() is four lines of steps: read the options, load the devices, get a row from each, write the CSV. The steps are functions in netlab/report.py, and in this task you write three of them:

FunctionTakesGives back
load_devices(source)"yaml" or "api"the inventory: a list of dicts, one per device
facts(device)one of those dictsone row: a dict with the six keys in COLUMNS
write_csv(rows, path)a list of rows, a file namenothing: it writes the file

Most of the work is already done. Chapter 7 loads the inventory from YAML and from the API, and chapter 6 logs in. The new parts are shaping one row and writing the CSV.

in one sentence One function per step means you can try each one in the REPL before the script runs them all.

Two device types, one row

show version, parsed by ntc-templates, gives each device type different keys. r1, r2 and sw1 are cisco_ios: the release is version, and the serial is serial, a list (a switch stack has one per member). r3 is arista_eos: the release is image, the serial is serial_number, a string. Only uptime is the same on both. Normalizing is mapping each type’s keys onto one set of column names, so the CSV doesn’t care which device a row came from.

Two columns don’t come from show version at all. The hostname is conn.base_prompt, which Netmiko works out from the prompt (r1# gives r1), so it’s there on every device. The model is in the template’s output, but empty on these devices, so it comes from the inventory’s model.

in one sentence Pick the column names first, then find where each one is reliable on each device type.

csv.DictWriter

csv.DictWriter(f, fieldnames=COLUMNS) writes dicts as CSV rows: writeheader() writes the column names, writerows(rows) writes every row, each value in the column its key names. A missing key gives an empty field; a key that isn’t in fieldnames raises ValueError. Open the file with newline="", as the csv documentation asks, because the module writes its own line endings.

2 show

Two rows, two device types

facts() for r1 and for r3, in the REPL (devices is load_devices("yaml")):

>>> facts(devices[0])
{'hostname': 'r1', 'model': 'CapeOS-Bolt', 'version': '0.11.2', 'serial': 'CAPEHFI5RPW', 'uptime': '10 minutes', 'ip_interfaces': 5}
>>> facts(devices[2])
{'hostname': 'r3', 'model': 'CapeOS-Pulse', 'version': '4.99.0C', 'serial': 'CAPE4S5VG44', 'uptime': '10 minutes', 'ip_interfaces': 4}

And the whole script, with the CSV it wrote:

$ python scripts/inventory.py
wrote inventory.csv: 4 of 4 devices
$ cat inventory.csv
hostname,model,version,serial,uptime,ip_interfaces
r1,CapeOS-Bolt,0.11.2,CAPEHFI5RPW,0 minutes,5
r2,CapeOS-Bolt,0.11.2,CAPED9P9655,0 minutes,5
r3,CapeOS-Pulse,4.99.0C,CAPE4S5VG44,0 minutes,4
sw1,CapeOS-Bolt,0.11.2,CAPEEXIN1H1,0 minutes,2

Your serials are different: the lab makes new ones every time it starts.

3 follow along

Load, write, and a row for r1

  1. Open netlab/report.py. Each function has a docstring that says what it should do, and a raise NotImplementedError line where your code goes. Replace that line in each, and keep the docstring.

  2. TODO 8.1a. The inventory, from either source. Chapter 7 wrote both halves:

    editnetlab/report.pyCtrl+P → report.py
    def load_devices(source="yaml"):
        if source == "api":
            load_env()
            return api_to_inventory(get_devices(os.environ["LAB_API_URL"], os.environ["LAB_API_TOKEN"]))
        return load_inventory(ROOT / "inventory.yml")
    

    load_env() reads .env into the environment, so LAB_API_URL and LAB_API_TOKEN are there to read. ROOT is the repository’s folder, so the script finds inventory.yml from any folder you run it in.

  3. TODO 8.1c. The CSV:

    editnetlab/report.pyCtrl+P → report.py
    def write_csv(rows, path):
        path = pathlib.Path(path)
        path.parent.mkdir(parents=True, exist_ok=True)
        with path.open("w", newline="") as f:
            writer = csv.DictWriter(f, fieldnames=COLUMNS)
            writer.writeheader()
            writer.writerows(rows)
    

    mkdir(parents=True, exist_ok=True) makes the folder if it isn’t there, so --out reports/x.csv works in task 8.3, and does nothing if it is.

  4. TODO 8.1b. One device’s row. First the login, with Netmiko’s two exceptions turned into DeviceError as collect() did in task 6.2. Then the two commands, and disconnect() in a finally: so the SSH session closes even if a command fails:

    editnetlab/report.pyCtrl+P → report.py
    def facts(device):
        try:
            conn = connect(device["host"], device["device_type"])
        except NetmikoAuthenticationException:
            raise DeviceError(f"{device['name']}: login failed") from None
        except NetmikoTimeoutException:
            raise DeviceError(f"{device['name']}: no answer from {device['host']}") from None
        try:
            version = conn.send_command("show version", use_textfsm=True)[0]
            brief = conn.send_command("show ip interface brief", use_textfsm=True)
            hostname = conn.base_prompt
        finally:
            conn.disconnect()
        release, serial = version["version"], version["serial"][0]
        return {
            "hostname": hostname,
            "model": device["model"],
            "version": release,
            "serial": serial,
            "uptime": version["uptime"],
            "ip_interfaces": sum(1 for i in brief if i["ip_address"] != "unassigned"),
        }
    

    [0] after show version takes the one dict out of the list ntc-templates returns. sum(1 for ...) counts the interfaces that have an address: one for each row that passes the test.

  5. Try it on r1, the first device in the inventory. Start the REPL from the repository’s folder:

    run in the terminal
    python
    
    >>> from netlab.report import load_devices, facts
    >>> devices = load_devices("yaml")
    >>> facts(devices[0])
    

    You get r1’s row, as in the Show stage. If you change report.py, exit() and start the REPL again: Python reads a module once, when you import it.

in one sentence facts() logs in, runs two commands, disconnects, and returns a dict with the same six keys every time.

4 challenge

The same row from r3

devices[2] is r3, on CapeOS Pulse (arista_eos). Call facts(devices[2]): it stops with a KeyError. Make facts() give r3 a row with the same six keys, then run the whole script and look at the CSV:

run in the terminal
python scripts/inventory.py
cat inventory.csv
hint

Look at what r3’s parsed show version has. In the REPL:

>>> from netlab.devices import connect
>>> conn = connect("172.80.111.13", "arista_eos")
>>> conn.send_command("show version", use_textfsm=True)[0]

then conn.disconnect(). Which keys hold the release and the serial?

hint

The device’s own dict says which kind it is: device["device_type"]. An if on it can pick the keys.

hint

r3’s serial is already a string. Only the cisco_ios one needs [0].

show the solutionnetlab/report.py

or pull it into your lab: make solve T=8.1, then git diff --staged shows what changed

def facts(device):
    try:
        conn = connect(device["host"], device["device_type"])
    except NetmikoAuthenticationException:
        raise DeviceError(f"{device['name']}: login failed") from None
    except NetmikoTimeoutException:
        raise DeviceError(f"{device['name']}: no answer from {device['host']}") from None
    try:
        version = conn.send_command("show version", use_textfsm=True)[0]
        brief = conn.send_command("show ip interface brief", use_textfsm=True)
        hostname = conn.base_prompt
    except DROPPED as e:
        raise DeviceError(f"{device['name']}: the session broke ({type(e).__name__})") from None
    finally:
        try:
            conn.disconnect()
        except DROPPED:
            pass                    # already gone: nothing left to close
    if device["device_type"] == "arista_eos":
        release, serial = version["image"], version["serial_number"]
    else:
        release, serial = version["version"], version["serial"][0]
    return {
        "hostname": hostname,
        "model": device["model"],
        "version": release,
        "serial": serial,
        "uptime": version["uptime"],
        "ip_interfaces": sum(1 for i in brief if i["ip_address"] != "unassigned"),
    }

DROPPED, at the top of report.py, is the lab’s list of what Netmiko raises when a device it had already logged in to drops the session: a read that timed out, an SSH error, a closed socket. Catching it here means one device that falls over halfway through a command becomes a failure for that device, not the end of the run (task 8.2 is about that). The try around disconnect() is for the same case: there’s nothing left to close.

This is r3’s show version with the cisco_ios key, and then what it does have:

>>> version["version"]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 'version'
>>> version
{'model': '', 'hw_version': '', 'serial_number': 'CAPE4S5VG44', 'sys_mac': 'cae2.b268.c68e', 'image': '4.99.0C', 'total_memory': '8025148', 'free_memory': '5733076', 'uptime': '15 minutes'}

The if is on the inventory’s device_type, the same field that told Netmiko which CLI to expect, so one field picks both the dialect and the keys. The mapping lives in one place: a third device type later is one more elif, and nothing after facts() changes. ip_interfaces needs no if, because both templates call the address ip_address (r3’s includes the prefix, 10.0.13.2/30, which is still not unassigned).

5 check: run it, then prove it

$ make check T=8.1

when it's right, you'll see

task 8.1  checks/test_8_1.py
checks/test_8_1.py::test_inventory_csv PASSED                            [100%]
============================== 1 passed in 4.84s ===============================
✓ task 8.1 done