task 7.2 · chapter 7 of 8: yaml, json and rest · 35 min
The same inventory from an API
objective
Write get_devices(), which asks the lab’s API for its devices with the token and turns a 403 or no answer into an ApiError with one line, and api_to_inventory(), which reshapes the API’s nested JSON into the same list of dicts as inventory.yml.
netlab/inventory.pyopen a file in VS Code: Ctrl+P (Cmd+P on a Mac), type its name, Enter
| The API | http://api:8000/api/devices/: read-only, NetBox-style JSON (LAB_API_URL in .env) |
|---|---|
| The token | LAB_API_TOKEN in .env, sent as the header Authorization: Token <token> |
| The answer | a dict with count, next, previous and results, the list of devices |
| Your error | ApiError, already defined at the top of netlab/inventory.py |
new to this? step by stepbeginner
curl -i http://api:8000/api/devices/: a 403, because there’s no token.- In the REPL,
requests.get()the same URL with theAuthorizationheader andtimeout=5; look at.status_codeand.json(). - Challenge:
get_devices()(TODO 7.2a) withraise_for_status()and anApiError, andapi_to_inventory()(TODO 7.2b). make check T=7.2.
more in meet the lab
Asking a server instead of a file
A GET request
A REST API holds data at URLs, and you ask for it over HTTP. GET means read. The answer has a
status code (200 for OK, 403 for Forbidden, 404 for Not Found), headers, and a body, here in JSON.
In Python, requests.get(url, headers={...}, timeout=5) sends the request and returns a response object:
r.status_code is the code, r.json() parses the body.
in one sentence requests.get() asks; r.status_code says how it went; r.json() is the data.
The token
The lab’s API, like NetBox, wants a token in a header: Authorization: Token <token>. A token is a
password for a program, so it lives in .env, not in the code. Your function gets it as an argument.
An error status isn’t an exception
A wrong token gets an answer: 403, and a JSON body that says why. requests doesn’t raise for that. Your code
has to look, and r.raise_for_status() does it for you: it raises requests.HTTPError for any 4xx or 5xx
status. A server that doesn’t answer at all raises requests.ConnectionError (or Timeout); all of them are
requests.RequestException.
in one sentence Call raise_for_status() before you trust the body.
Nested JSON
The API answers the way NetBox does: a device’s role is {"name": "Router", "slug": "router"}, not
"router", and its address is inside primary_ip. To compare with inventory.yml, you reshape each device
into the same five flat keys.
200, 403, and r1 as the API sees it
With the token:
>>> r = requests.get("http://api:8000/api/devices/", headers={"Authorization": "Token 0123456789abcdef0123456789abcdef01234567"}, timeout=5)
>>> r
<Response [200]>
>>> data = r.json()
>>> data.keys()
dict_keys(['count', 'next', 'previous', 'results'])
>>> data["count"]
4
>>> data["results"][0]
{'id': 1, 'url': 'http://api:8000/api/devices/1/', 'name': 'r1', 'device_type': {'model': 'CapeOS-Bolt', 'manufacturer': {'name': 'Network Doodles', 'slug': 'network-doodles'}}, 'role': {'name': 'Router', 'slug': 'router'}, 'platform': {'name': 'cisco_ios', 'slug': 'cisco_ios'}, 'serial': '', 'site': {'name': 'LAB', 'slug': 'lab'}, 'status': {'value': 'active', 'label': 'Active'}, 'primary_ip': {'address': '172.80.111.11/24', 'family': {'value': 4, 'label': 'IPv4'}}}
With a wrong one:
>>> bad = requests.get("http://api:8000/api/devices/", headers={"Authorization": "Token nope"}, timeout=5)
>>> bad.status_code, bad.json()
(403, {'detail': 'Invalid token'})
>>> bad.raise_for_status()
Traceback (most recent call last):
...
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: http://api:8000/api/devices/
And what the check reports for a get_devices() that skips the status and goes straight for results:
________________________________ test_bad_token ________________________________
checks/test_7_2.py:29: in test_bad_token
call(get_devices, env["LAB_API_URL"], "not-the-token")
checks/conftest.py:20: in call
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
netlab/inventory.py:47: in get_devices
return r.json()["results"]
^^^^^^^^^^^^^^^^^^^
E KeyError: 'results'
A KeyError about results, when the real problem was the token.
Your first request
Ask the API with
curl, without a token, and look at the status line:run in the terminalcurl -i http://api:8000/api/devices/1/HTTP/1.0 403 Forbidden, and a body saying Authentication credentials were not provided.Now from Python, with the token read from
.env, the way your code will get it:>>> import os, requests >>> from netlab.env import load_env >>> load_env() >>> h = {"Authorization": f"Token {os.environ['LAB_API_TOKEN']}"} >>> page = requests.get("http://api:8000/api/devices/", headers=h, params={"limit": 2}, timeout=5).json() >>> page["count"], page["next"], page["previous"] (4, 'http://api:8000/api/devices/?limit=2&offset=2', None) >>> [d["name"] for d in page["results"]] ['r1', 'r2']params=adds?limit=2to the URL. Two devices to a page,countsays there are four, andnextis the URL of the second page. Withoutlimit, a page holds 50, so the lab’s four devices arrive in one.Try a device that doesn’t exist, and a method the API doesn’t allow:
>>> requests.get("http://api:8000/api/devices/99/", headers=h, timeout=5).status_code 404 >>> requests.post("http://api:8000/api/devices/", headers=h, timeout=5).status_code 405
in one sentence Every answer has a status code; only 2xx means the body is the data you asked for.
get_devices() and api_to_inventory()
Write the two functions in netlab/inventory.py:
get_devices(base_url, token)(TODO 7.2a):GET <base_url>/api/devices/with the token header and a 5-second timeout, and return the JSON’sresultslist as it is. For an error status, raiseApiErrorwith one line that includes the status code (the check looks for403). For no answer at all, raiseApiErrortoo.api_to_inventory(results)(TODO 7.2b): each API device as the five keys ofinventory.yml.
When they work, this is True:
>>> api_to_inventory(get_devices("http://api:8000", os.environ["LAB_API_TOKEN"])) == load_inventory("inventory.yml")
hint
Put requests.get(...) and r.raise_for_status() together in one try:. Then except requests.HTTPError:
first, because it’s the more specific one, and except requests.RequestException after it, for everything
else.
hint
In the HTTPError branch, r exists: use r.status_code, and the API’s own reason from r.json()["detail"].
In the other branch there’s no response, only the exception: type(e).__name__ names what went wrong.
hint
For the reshape, the docstring has the map: primary_ip.address without its prefix (split("/")[0]),
platform.slug, role.slug, device_type.model. Each a.b is d["a"]["b"].
show the solutionnetlab/inventory.py
or pull it into your lab: make solve T=7.2, then git diff --staged shows what changed
def get_devices(base_url, token):
try:
r = requests.get(f"{base_url}/api/devices/", headers={"Authorization": f"Token {token}"}, timeout=5)
r.raise_for_status()
except requests.HTTPError:
try:
detail = r.json().get("detail", r.reason)
except ValueError: # an error page that isn't JSON, from a proxy or a web server
detail = r.reason
raise ApiError(f"the API answered {r.status_code}: {detail}") from None
except requests.RequestException as e:
raise ApiError(f"no answer from the API at {base_url}: {type(e).__name__}") from None
return r.json()["results"]
def api_to_inventory(results):
devices = []
for d in results:
devices.append({
"name": d["name"],
"host": d["primary_ip"]["address"].split("/")[0],
"device_type": d["platform"]["slug"],
"role": d["role"]["slug"],
"model": d["device_type"]["model"],
})
return devices
>>> get_devices("http://api:8000", "nope")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/workspaces/python-101/netlab/inventory.py", line 59, in get_devices
raise ApiError(f"the API answered {r.status_code}: {detail}") from None
netlab.inventory.ApiError: the API answered 403: Invalid token
The order of the excepts matters: HTTPError is itself a RequestException, so with the general one first,
a 403 would be reported as no answer. .get("detail", r.reason) falls back to the status’s own name
(Forbidden) for an answer without a detail, and the inner try does the same for an error page that
isn’t JSON at all, the kind a proxy in front of an API sends (the check serves one). Note the two meanings of device_type: in the API it’s the
hardware model, as in NetBox; in the inventory it’s Netmiko’s name for the CLI, which NetBox calls the
platform.
5 check: run it, then prove it
$ make check T=7.2when it's right, you'll see
task 7.2 checks/test_7_2.py
checks/test_7_2.py::test_reshape_unseen PASSED [ 25%]
checks/test_7_2.py::test_live PASSED [ 50%]
checks/test_7_2.py::test_bad_token PASSED [ 75%]
checks/test_7_2.py::test_no_answer PASSED [100%]
============================== 4 passed in 0.23s ===============================
✓ task 7.2 done