online posts 16
python 101 for network engineers

task 4.2 · chapter 4 of 8: files, modules and errors · 30 min

Survive a bad file

objective

Two configs in the folder can’t be read: one is locked, one isn’t UTF-8. Make the report skip them, say why, write every other device, and exit 1. Catch those two problems and nothing else.

you edit, in the labnetlab/configs.py

open a file in VS Code: Ctrl+P (Cmd+P on a Mac), type its name, Enter

given
The problemmake break T=4.2 adds configs/locked.cfg (chmod 000) and configs/old-r9.cfg (saved in Latin-1)
Your functionload_configs_safe() in netlab/configs.py (TODO 4.2): returns (configs, problems)
The scriptscripts/ntp_report.py, unchanged: prints each problem to stderr, exits 1 if there were any
Afterwardsmake reset takes the two files out again
new to this? step by stepbeginner
  1. make break T=4.2, then python scripts/ntp_report.py: a traceback.
  2. Read it from the bottom: which exception, which file, which line of yours?
  3. Challenge: write load_configs_safe() with try/except inside the loop, for PermissionError and UnicodeDecodeError only.
  4. Run the script again (it lists two skipped files and exits 1), make check T=4.2, then make reset.

more in meet the lab

1 explain

Catch it, report it, carry on

Exceptions

When Python can’t do something, it raises an exception: an object whose type says what went wrong. Reading files, you meet these:

ExceptionMeaning
PermissionErrorthe file’s there, and you’re not allowed to read it
UnicodeDecodeErrorthe bytes aren’t valid UTF-8 (or whatever encoding you asked for)
FileNotFoundErrorthere’s no such file
IsADirectoryErrorthat name is a folder

If nothing catches it, the program stops with a traceback. Read it from the bottom: the last line is the exception and its message, and the lowest frame in your own files is where to look.

try and except

try:
    text = path.read_text(encoding="utf-8")
except PermissionError:
    ...             # runs only if the try block raised a PermissionError

Python runs the try block. If it raises an exception that an except names, Python runs that block and carries on after the whole try. except UnicodeDecodeError as e: also keeps the exception in e; a UnicodeDecodeError knows where the bad byte was, in e.start.

in one sentence Put the try around the one line that can fail, inside the loop, so one bad file costs one file.

Only what you can handle

A locked file or a Latin-1 file is bad data: skipping it and saying so is the right answer. A folder called something.cfg, or a typo in your code, isn’t bad data; it’s a mistake, and it should still stop the program with a traceback, so someone fixes it. except Exception: would hide both kinds. The check tests that: it puts a folder named not-a-file.cfg in the folder, and expects your function to raise IsADirectoryError.

in one sentence Name the exceptions you expect. Everything else is a bug, and a bug should be loud.

2 show

Two bad files

$ make break T=4.2

  setting up task 4.2's problem

  wrote     configs/old-r9.cfg: saved in Latin-1, so the é isn't valid UTF-8
  wrote     configs/locked.cfg: and took away every permission on it (chmod 000)

  make reset puts everything back.

$ ls -l configs
total 24
---------- 1 vscode vscode  37 Nov 11 05:52 locked.cfg
-rw-r--r-- 1 vscode vscode 102 Nov 11 05:52 old-r9.cfg
-rw-r--r-- 1 vscode vscode 802 Nov 11 05:52 r1.cfg
-rw-r--r-- 1 vscode vscode 748 Nov 11 05:52 r2.cfg
-rw-r--r-- 1 vscode vscode 641 Nov 11 05:52 r3.cfg
-rw-r--r-- 1 vscode vscode 854 Nov 11 05:52 sw1.cfg

---------- is chmod 000: no one may read it, you included. With 4.1 done and 4.2 not, the report stops at the first bad file in sorted order, locked.cfg:

$ python scripts/ntp_report.py
Traceback (most recent call last):
  ...
  File "/workspaces/python-101/netlab/configs.py", line 27, in load_configs
    configs[path.stem] = path.read_text(encoding="utf-8")
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/pathlib.py", line 1058, in read_text
    with self.open(mode='r', encoding=encoding, errors=errors) as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/pathlib.py", line 1044, in open
    return io.open(self, mode, buffering, encoding, errors, newline)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
PermissionError: [Errno 13] Permission denied: '/workspaces/python-101/configs/locked.cfg'

When it’s done, the same run finishes:

$ python scripts/ntp_report.py configs reports/ntp.csv
wrote reports/ntp.csv (4 devices)
skipped locked.cfg: permission denied
skipped old-r9.cfg: not UTF-8 text (byte 64)
$ echo $?
1
3 follow along

Meet the two files

  1. Set up the problem, and run the report:

    run in the terminal
    make break T=4.2
    python scripts/ntp_report.py
    

    Read the traceback from the bottom. The exception is PermissionError, the file is locked.cfg, and the lowest frame in your code is the read_text() line in load_configs(). Your line number may differ.

  2. Meet the other one in the REPL. It’s readable, but not as UTF-8:

    >>> from pathlib import Path
    >>> Path("configs/old-r9.cfg").read_text()
    Traceback (most recent call last):
      ...
    UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 64: invalid continuation byte
    >>> Path("configs/old-r9.cfg").read_bytes()[60:80]
    b' Caf\xe9 network ^C\nntp'
    

    \xe9 is é in Latin-1, a single byte. In UTF-8, é takes two bytes, so this one on its own is invalid. The banner was saved by an older tool.

  3. Try catching it, once, at the prompt:

    >>> try:
    ...     Path("configs/old-r9.cfg").read_text(encoding="utf-8")
    ... except UnicodeDecodeError as e:
    ...     print("skipped:", e.reason, "at byte", e.start)
    ...
    skipped: invalid continuation byte at byte 64
    

in one sentence Each bad file raises its own exception type, and that type is what an except matches on.

4 challenge

load_configs_safe()

Write load_configs_safe(folder). It reads the same *.cfg files as load_configs(), and returns two dicts:

  • configs: device name to text, for every file that read cleanly;
  • problems: file name ("old-r9.cfg") to a short reason, for every file that raised PermissionError or UnicodeDecodeError.

Catch only those two. Then run the script: two skipped lines, a CSV of the other four, and exit code 1.

hint

Start from load_configs(): the same loop, with a try around the line that reads the file. A try outside the loop would stop at the first bad file, which is what you have now.

hint

Two except blocks, one per exception, each putting path.name and a reason in problems. f"not UTF-8 text (byte {e.start})" says where.

hint

If the check says DID NOT RAISE IsADirectoryError, your except is too broad. This is except Exception:

checks/test_4_2.py::test_only_what_you_can_handle FAILED                 [ 66%]
...
E   Failed: DID NOT RAISE IsADirectoryError
show the solutionnetlab/configs.py

or pull it into your lab: make solve T=4.2, then git diff --staged shows what changed

def load_configs_safe(folder):
    """Like load_configs, but a file that can't be read doesn't stop the others.

    Returns two dicts: (configs, problems). configs is device name -> text for every file that read
    cleanly. problems is file name (like "r9.cfg") -> a short reason, for every *.cfg file that
    couldn't be read: one you're not allowed to read (PermissionError), or one that isn't UTF-8 text
    (UnicodeDecodeError). Catch only those two: anything else is a bug, and should still stop you.
    """
    configs, problems = {}, {}
    for path in sorted(pathlib.Path(folder).glob("*.cfg")):
        try:
            configs[path.stem] = path.read_text(encoding="utf-8")
        except PermissionError:
            problems[path.name] = "permission denied"
        except UnicodeDecodeError as e:
            problems[path.name] = f"not UTF-8 text (byte {e.start})"
    return configs, problems

The function only collects the problems; the script decides what to do with them: print each one to stderr, so they don’t mix with the normal output, and exit 1, so a cron job or a pipeline can tell. The check builds a folder of its own with a Latin-1 file and a locked one under made-up names, and a third folder with a directory called not-a-file.cfg. When you’re done, make reset removes the two files from configs/.

5 check: run it, then prove it

$ make check T=4.2

when it's right, you'll see

task 4.2  checks/test_4_2.py
checks/test_4_2.py::test_function PASSED                                 [ 33%]
checks/test_4_2.py::test_only_what_you_can_handle PASSED                 [ 66%]
checks/test_4_2.py::test_the_script PASSED                               [100%]
============================== 3 passed in 0.02s ===============================
✓ task 4.2 done