online posts 16
python 101 for network engineers

task 7.1 · chapter 7 of 8: yaml, json and rest · 25 min

The inventory in YAML and JSON

objective

Write load_inventory(), save_json() and load_json() in netlab/inventory.py: the lab’s devices from inventory.yml, out to a JSON file and back, unchanged. Then find the four values at the top of inventory.yml that YAML reads differently from you, and quote them.

you edit, in the labnetlab/inventory.pyinventory.yml

open a file in VS Code: Ctrl+P (Cmd+P on a Mac), type its name, Enter

given
The inventoryinventory.yml: a few site settings, then devices:, a list of r1, r2, r3 and sw1
Each devicefive keys: name, host, device_type, role, model
The librariesPyYAML 6.0 (import yaml) and the standard library’s json, both imported already
The ruleyaml.safe_load, never yaml.load: the check looks
new to this? step by stepbeginner
  1. cat inventory.yml, then in the REPL: import yaml, doc = yaml.safe_load(open("inventory.yml")), doc["devices"][0].
  2. Write load_inventory() (TODO 7.1a), save_json() (7.1b) and load_json() (7.1c), each with a with open(...).
  3. Challenge (TODO 7.1d): print doc without devices, find the four values that aren’t what the file says, and quote them in inventory.yml.
  4. make check T=7.1.

more in meet the lab

1 explain

Two spellings of one list

YAML and JSON

Both write down the same shapes: lists and dicts of strings, numbers, booleans and nothing. In YAML, key: value is a dict entry, - starts a list item, and indentation (spaces only) shows what’s inside what. JSON writes the same thing with {}, [] and double quotes. Loaded into Python, they’re identical.

in one sentence Load a YAML or a JSON file, and you get the same lists and dicts.

safe_load

yaml.safe_load(f) builds only plain values. yaml.load() with PyYAML’s full loader can build any Python object a file asks for, which means a YAML file can run code on your machine. An inventory needs only plain values, so there’s never a reason to use anything else.

json.dump and json.load

json.dump(data, f, indent=2) writes Python values to an open file as JSON, indented so a person can read it; json.load(f) reads them back. (With an s on the end, dumps and loads work on strings instead of files.)

What YAML guesses

YAML decides a value’s type from how it looks, and PyYAML follows YAML 1.1’s rules, which guess more than you’d expect: some words are booleans, a leading zero means octal, and colons can make a number. When a value has to stay a string, put it in quotes.

in one sentence If it must be a string, quote it.

2 show

One device, three ways

>>> import yaml, json
>>> doc = yaml.safe_load(open("inventory.yml"))
>>> doc.keys()
dict_keys(['site', 'country', 'rack', 'backup', 'devices'])
>>> doc["devices"][2]
{'name': 'r3', 'host': '172.80.111.13', 'device_type': 'arista_eos', 'role': 'router', 'model': 'CapeOS-Pulse'}
>>> print(json.dumps(doc["devices"][2], indent=2))
{
  "name": "r3",
  "host": "172.80.111.13",
  "device_type": "arista_eos",
  "role": "router",
  "model": "CapeOS-Pulse"
}

And the starter’s check, before you’ve started: one assertion, and two functions still to write.

________________________ test_the_surprises_are_quoted _________________________
checks/test_7_1.py:13: in test_the_surprises_are_quoted
    assert doc["country"] == "NO", f"country loads as {doc['country']!r}: Norway's code should stay the string 'NO'"
E   AssertionError: country loads as False: Norway's code should stay the string 'NO'
E   assert False == 'NO'
______________________________ test_lab_inventory ______________________________
TODO 7.1a is still open: write load_inventory() in netlab/inventory.py, then check again
____________________________ test_unseen_round_trip ____________________________
TODO 7.1a is still open: write load_inventory() in netlab/inventory.py, then check again
3 failed, 1 passed in 0.08s
3 follow along

Load, save, load again

  1. Open netlab/inventory.py and find TODO 7.1a. The devices are the list under the file’s top-level devices: key:

    editnetlab/inventory.pyCtrl+P → inventory.py
    def load_inventory(path):
        """...the docstring stays as it is..."""
        with open(path) as f:
            return yaml.safe_load(f)["devices"]
    
  2. TODO 7.1b writes the list to a JSON file, opened for writing with "w":

    editnetlab/inventory.pyCtrl+P → inventory.py
    def save_json(devices, path):
        with open(path, "w") as f:
            json.dump(devices, f, indent=2)
            f.write("\n")
    

    json.dump doesn’t end the file with a newline; the last line adds one, the way text files on Linux end.

  3. TODO 7.1c reads it back. That one’s yours: with open(...), and the json function that reads from a file.

  4. Try the round trip in the REPL:

    >>> from netlab.inventory import load_inventory, save_json, load_json
    >>> devices = load_inventory("inventory.yml")
    >>> save_json(devices, "scratch/inventory.json")
    >>> load_json("scratch/inventory.json") == devices
    True
    

    Open scratch/inventory.json in the editor: the same four devices, in JSON.

in one sentence YAML in, JSON out, JSON in: the same list of dicts all the way around.

4 challenge

Four values YAML reads its own way

TODO 7.1d is in inventory.yml itself. The block above devices: has four values that don’t load the way they read. Load the file in the REPL, look at what each of country, rack and backup became, and quote the four so YAML keeps them as the strings the comments describe. Don’t change what they say, only how they’re written.

hint

Print everything except the devices: {k: v for k, v in doc.items() if k != "devices"}. Compare each value with the line in the file. Look at the keys of backup as well as its values.

hint

Lesson 7 has a table of what YAML 1.1 does to NO, on, 0123 and 2:30, and why.

hint

A key can be quoted too: "on": sunday.

show the solutioninventory.yml

or pull it into your lab: make solve T=7.1, then git diff --staged shows what changed

site: lab
country: "NO"               # Norway
rack: "0123"                # the label on the rack
backup:
  "on": sunday              # which day the configs are backed up
  at: "2:30"                # and when
>>> yaml.safe_load("country: NO\nrack: 0123\nbackup:\n  on: sunday\n  at: 2:30\n")
{'country': False, 'rack': 83, 'backup': {True: 'sunday', 'at': 150}}
>>> yaml.safe_load('country: "NO"\nrack: "0123"\nbackup:\n  "on": sunday\n  at: "2:30"\n')
{'country': 'NO', 'rack': '0123', 'backup': {'on': 'sunday', 'at': '2:30'}}

NO and on are YAML 1.1 booleans, 0123 is octal (83), and 2:30 is base 60 (2×60 + 30 = 150). Quoting makes each a string. And load_json():

def load_json(path):
    with open(path) as f:
        return json.load(f)

None of the four affects the devices, which is why the round trip passed before you fixed them. That’s the danger: the file loads without an error, and the wrong value turns up later, in whatever reads country.

5 check: run it, then prove it

$ make check T=7.1

when it's right, you'll see

task 7.1  checks/test_7_1.py
checks/test_7_1.py::test_the_surprises_are_quoted PASSED                 [ 25%]
checks/test_7_1.py::test_lab_inventory PASSED                            [ 50%]
checks/test_7_1.py::test_safe_load_only PASSED                           [ 75%]
checks/test_7_1.py::test_unseen_round_trip PASSED                        [100%]
============================== 4 passed in 0.10s ===============================
✓ task 7.1 done