online posts 16
python 101 for network engineers

task 7.2 · chapter 7 of 8: yaml, json and rest · 35 min

The same inventory from an API

objective

Write get_devices(), which asks the lab’s API for its devices with the token and turns a 403 or no answer into an ApiError with one line, and api_to_inventory(), which reshapes the API’s nested JSON into the same list of dicts as inventory.yml.

you edit, in the labnetlab/inventory.py

open a file in VS Code: Ctrl+P (Cmd+P on a Mac), type its name, Enter

given
The APIhttp://api:8000/api/devices/: read-only, NetBox-style JSON (LAB_API_URL in .env)
The tokenLAB_API_TOKEN in .env, sent as the header Authorization: Token <token>
The answera dict with count, next, previous and results, the list of devices
Your errorApiError, already defined at the top of netlab/inventory.py
new to this? step by stepbeginner
  1. curl -i http://api:8000/api/devices/: a 403, because there’s no token.
  2. In the REPL, requests.get() the same URL with the Authorization header and timeout=5; look at .status_code and .json().
  3. Challenge: get_devices() (TODO 7.2a) with raise_for_status() and an ApiError, and api_to_inventory() (TODO 7.2b).
  4. make check T=7.2.

more in meet the lab

1 explain

Asking a server instead of a file

A GET request

A REST API holds data at URLs, and you ask for it over HTTP. GET means read. The answer has a status code (200 for OK, 403 for Forbidden, 404 for Not Found), headers, and a body, here in JSON. In Python, requests.get(url, headers={...}, timeout=5) sends the request and returns a response object: r.status_code is the code, r.json() parses the body.

in one sentence requests.get() asks; r.status_code says how it went; r.json() is the data.

The token

The lab’s API, like NetBox, wants a token in a header: Authorization: Token <token>. A token is a password for a program, so it lives in .env, not in the code. Your function gets it as an argument.

An error status isn’t an exception

A wrong token gets an answer: 403, and a JSON body that says why. requests doesn’t raise for that. Your code has to look, and r.raise_for_status() does it for you: it raises requests.HTTPError for any 4xx or 5xx status. A server that doesn’t answer at all raises requests.ConnectionError (or Timeout); all of them are requests.RequestException.

in one sentence Call raise_for_status() before you trust the body.

Nested JSON

The API answers the way NetBox does: a device’s role is {"name": "Router", "slug": "router"}, not "router", and its address is inside primary_ip. To compare with inventory.yml, you reshape each device into the same five flat keys.

2 show

200, 403, and r1 as the API sees it

With the token:

>>> r = requests.get("http://api:8000/api/devices/", headers={"Authorization": "Token 0123456789abcdef0123456789abcdef01234567"}, timeout=5)
>>> r
<Response [200]>
>>> data = r.json()
>>> data.keys()
dict_keys(['count', 'next', 'previous', 'results'])
>>> data["count"]
4
>>> data["results"][0]
{'id': 1, 'url': 'http://api:8000/api/devices/1/', 'name': 'r1', 'device_type': {'model': 'CapeOS-Bolt', 'manufacturer': {'name': 'Network Doodles', 'slug': 'network-doodles'}}, 'role': {'name': 'Router', 'slug': 'router'}, 'platform': {'name': 'cisco_ios', 'slug': 'cisco_ios'}, 'serial': '', 'site': {'name': 'LAB', 'slug': 'lab'}, 'status': {'value': 'active', 'label': 'Active'}, 'primary_ip': {'address': '172.80.111.11/24', 'family': {'value': 4, 'label': 'IPv4'}}}

With a wrong one:

>>> bad = requests.get("http://api:8000/api/devices/", headers={"Authorization": "Token nope"}, timeout=5)
>>> bad.status_code, bad.json()
(403, {'detail': 'Invalid token'})
>>> bad.raise_for_status()
Traceback (most recent call last):
  ...
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: http://api:8000/api/devices/

And what the check reports for a get_devices() that skips the status and goes straight for results:

________________________________ test_bad_token ________________________________
checks/test_7_2.py:29: in test_bad_token
    call(get_devices, env["LAB_API_URL"], "not-the-token")
checks/conftest.py:20: in call
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
netlab/inventory.py:47: in get_devices
    return r.json()["results"]
           ^^^^^^^^^^^^^^^^^^^
E   KeyError: 'results'

A KeyError about results, when the real problem was the token.

3 follow along

Your first request

  1. Ask the API with curl, without a token, and look at the status line:

    run in the terminal
    curl -i http://api:8000/api/devices/1/
    

    HTTP/1.0 403 Forbidden, and a body saying Authentication credentials were not provided.

  2. Now from Python, with the token read from .env, the way your code will get it:

    >>> import os, requests
    >>> from netlab.env import load_env
    >>> load_env()
    >>> h = {"Authorization": f"Token {os.environ['LAB_API_TOKEN']}"}
    >>> page = requests.get("http://api:8000/api/devices/", headers=h, params={"limit": 2}, timeout=5).json()
    >>> page["count"], page["next"], page["previous"]
    (4, 'http://api:8000/api/devices/?limit=2&offset=2', None)
    >>> [d["name"] for d in page["results"]]
    ['r1', 'r2']
    

    params= adds ?limit=2 to the URL. Two devices to a page, count says there are four, and next is the URL of the second page. Without limit, a page holds 50, so the lab’s four devices arrive in one.

  3. Try a device that doesn’t exist, and a method the API doesn’t allow:

    >>> requests.get("http://api:8000/api/devices/99/", headers=h, timeout=5).status_code
    404
    >>> requests.post("http://api:8000/api/devices/", headers=h, timeout=5).status_code
    405
    

in one sentence Every answer has a status code; only 2xx means the body is the data you asked for.

4 challenge

get_devices() and api_to_inventory()

Write the two functions in netlab/inventory.py:

  • get_devices(base_url, token) (TODO 7.2a): GET <base_url>/api/devices/ with the token header and a 5-second timeout, and return the JSON’s results list as it is. For an error status, raise ApiError with one line that includes the status code (the check looks for 403). For no answer at all, raise ApiError too.
  • api_to_inventory(results) (TODO 7.2b): each API device as the five keys of inventory.yml.

When they work, this is True:

>>> api_to_inventory(get_devices("http://api:8000", os.environ["LAB_API_TOKEN"])) == load_inventory("inventory.yml")
hint

Put requests.get(...) and r.raise_for_status() together in one try:. Then except requests.HTTPError: first, because it’s the more specific one, and except requests.RequestException after it, for everything else.

hint

In the HTTPError branch, r exists: use r.status_code, and the API’s own reason from r.json()["detail"]. In the other branch there’s no response, only the exception: type(e).__name__ names what went wrong.

hint

For the reshape, the docstring has the map: primary_ip.address without its prefix (split("/")[0]), platform.slug, role.slug, device_type.model. Each a.b is d["a"]["b"].

show the solutionnetlab/inventory.py

or pull it into your lab: make solve T=7.2, then git diff --staged shows what changed

def get_devices(base_url, token):
    try:
        r = requests.get(f"{base_url}/api/devices/", headers={"Authorization": f"Token {token}"}, timeout=5)
        r.raise_for_status()
    except requests.HTTPError:
        try:
            detail = r.json().get("detail", r.reason)
        except ValueError:          # an error page that isn't JSON, from a proxy or a web server
            detail = r.reason
        raise ApiError(f"the API answered {r.status_code}: {detail}") from None
    except requests.RequestException as e:
        raise ApiError(f"no answer from the API at {base_url}: {type(e).__name__}") from None
    return r.json()["results"]


def api_to_inventory(results):
    devices = []
    for d in results:
        devices.append({
            "name": d["name"],
            "host": d["primary_ip"]["address"].split("/")[0],
            "device_type": d["platform"]["slug"],
            "role": d["role"]["slug"],
            "model": d["device_type"]["model"],
        })
    return devices
>>> get_devices("http://api:8000", "nope")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/workspaces/python-101/netlab/inventory.py", line 59, in get_devices
    raise ApiError(f"the API answered {r.status_code}: {detail}") from None
netlab.inventory.ApiError: the API answered 403: Invalid token

The order of the excepts matters: HTTPError is itself a RequestException, so with the general one first, a 403 would be reported as no answer. .get("detail", r.reason) falls back to the status’s own name (Forbidden) for an answer without a detail, and the inner try does the same for an error page that isn’t JSON at all, the kind a proxy in front of an API sends (the check serves one). Note the two meanings of device_type: in the API it’s the hardware model, as in NetBox; in the inventory it’s Netmiko’s name for the CLI, which NetBox calls the platform.

5 check: run it, then prove it

$ make check T=7.2

when it's right, you'll see

task 7.2  checks/test_7_2.py
checks/test_7_2.py::test_reshape_unseen PASSED                           [ 25%]
checks/test_7_2.py::test_live PASSED                                     [ 50%]
checks/test_7_2.py::test_bad_token PASSED                                [ 75%]
checks/test_7_2.py::test_no_answer PASSED                                [100%]
============================== 4 passed in 0.23s ===============================
✓ task 7.2 done