task 4.2 · chapter 4 of 8: files, modules and errors · 30 min
Survive a bad file
objective
Two configs in the folder can’t be read: one is locked, one isn’t UTF-8. Make the report skip them, say why, write every other device, and exit 1. Catch those two problems and nothing else.
netlab/configs.pyopen a file in VS Code: Ctrl+P (Cmd+P on a Mac), type its name, Enter
| The problem | make break T=4.2 adds configs/locked.cfg (chmod 000) and configs/old-r9.cfg (saved in Latin-1) |
|---|---|
| Your function | load_configs_safe() in netlab/configs.py (TODO 4.2): returns (configs, problems) |
| The script | scripts/ntp_report.py, unchanged: prints each problem to stderr, exits 1 if there were any |
| Afterwards | make reset takes the two files out again |
new to this? step by stepbeginner
make break T=4.2, thenpython scripts/ntp_report.py: a traceback.- Read it from the bottom: which exception, which file, which line of yours?
- Challenge: write
load_configs_safe()withtry/exceptinside the loop, forPermissionErrorandUnicodeDecodeErroronly. - Run the script again (it lists two skipped files and exits 1),
make check T=4.2, thenmake reset.
more in meet the lab
Catch it, report it, carry on
Exceptions
When Python can’t do something, it raises an exception: an object whose type says what went wrong. Reading files, you meet these:
| Exception | Meaning |
|---|---|
PermissionError | the file’s there, and you’re not allowed to read it |
UnicodeDecodeError | the bytes aren’t valid UTF-8 (or whatever encoding you asked for) |
FileNotFoundError | there’s no such file |
IsADirectoryError | that name is a folder |
If nothing catches it, the program stops with a traceback. Read it from the bottom: the last line is the exception and its message, and the lowest frame in your own files is where to look.
try and except
try:
text = path.read_text(encoding="utf-8")
except PermissionError:
... # runs only if the try block raised a PermissionError
Python runs the try block. If it raises an exception that an except names, Python runs that block and
carries on after the whole try. except UnicodeDecodeError as e: also keeps the exception in e; a
UnicodeDecodeError knows where the bad byte was, in e.start.
in one sentence Put the try around the one line that can fail, inside the loop, so one bad file costs one file.
Only what you can handle
A locked file or a Latin-1 file is bad data: skipping it and saying so is the right answer. A folder
called something.cfg, or a typo in your code, isn’t bad data; it’s a mistake, and it should still stop the
program with a traceback, so someone fixes it. except Exception: would hide both kinds. The check tests
that: it puts a folder named not-a-file.cfg in the folder, and expects your function to raise
IsADirectoryError.
in one sentence Name the exceptions you expect. Everything else is a bug, and a bug should be loud.
Two bad files
$ make break T=4.2
setting up task 4.2's problem
wrote configs/old-r9.cfg: saved in Latin-1, so the é isn't valid UTF-8
wrote configs/locked.cfg: and took away every permission on it (chmod 000)
make reset puts everything back.
$ ls -l configs
total 24
---------- 1 vscode vscode 37 Nov 11 05:52 locked.cfg
-rw-r--r-- 1 vscode vscode 102 Nov 11 05:52 old-r9.cfg
-rw-r--r-- 1 vscode vscode 802 Nov 11 05:52 r1.cfg
-rw-r--r-- 1 vscode vscode 748 Nov 11 05:52 r2.cfg
-rw-r--r-- 1 vscode vscode 641 Nov 11 05:52 r3.cfg
-rw-r--r-- 1 vscode vscode 854 Nov 11 05:52 sw1.cfg
---------- is chmod 000: no one may read it, you included. With 4.1 done and 4.2 not, the report stops at
the first bad file in sorted order, locked.cfg:
$ python scripts/ntp_report.py
Traceback (most recent call last):
...
File "/workspaces/python-101/netlab/configs.py", line 27, in load_configs
configs[path.stem] = path.read_text(encoding="utf-8")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/pathlib.py", line 1058, in read_text
with self.open(mode='r', encoding=encoding, errors=errors) as f:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/pathlib.py", line 1044, in open
return io.open(self, mode, buffering, encoding, errors, newline)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
PermissionError: [Errno 13] Permission denied: '/workspaces/python-101/configs/locked.cfg'
When it’s done, the same run finishes:
$ python scripts/ntp_report.py configs reports/ntp.csv
wrote reports/ntp.csv (4 devices)
skipped locked.cfg: permission denied
skipped old-r9.cfg: not UTF-8 text (byte 64)
$ echo $?
1
Meet the two files
Set up the problem, and run the report:
run in the terminalmake break T=4.2 python scripts/ntp_report.pyRead the traceback from the bottom. The exception is
PermissionError, the file islocked.cfg, and the lowest frame in your code is theread_text()line inload_configs(). Your line number may differ.Meet the other one in the REPL. It’s readable, but not as UTF-8:
>>> from pathlib import Path >>> Path("configs/old-r9.cfg").read_text() Traceback (most recent call last): ... UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 64: invalid continuation byte >>> Path("configs/old-r9.cfg").read_bytes()[60:80] b' Caf\xe9 network ^C\nntp'\xe9iséin Latin-1, a single byte. In UTF-8,étakes two bytes, so this one on its own is invalid. The banner was saved by an older tool.Try catching it, once, at the prompt:
>>> try: ... Path("configs/old-r9.cfg").read_text(encoding="utf-8") ... except UnicodeDecodeError as e: ... print("skipped:", e.reason, "at byte", e.start) ... skipped: invalid continuation byte at byte 64
in one sentence Each bad file raises its own exception type, and that type is what an except matches on.
load_configs_safe()
Write load_configs_safe(folder). It reads the same *.cfg files as load_configs(), and returns two dicts:
configs: device name to text, for every file that read cleanly;problems: file name ("old-r9.cfg") to a short reason, for every file that raisedPermissionErrororUnicodeDecodeError.
Catch only those two. Then run the script: two skipped lines, a CSV of the other four, and exit code 1.
hint
Start from load_configs(): the same loop, with a try around the line that reads the file. A try outside
the loop would stop at the first bad file, which is what you have now.
hint
Two except blocks, one per exception, each putting path.name and a reason in problems.
f"not UTF-8 text (byte {e.start})" says where.
hint
If the check says DID NOT RAISE IsADirectoryError, your except is too broad. This is except Exception:
checks/test_4_2.py::test_only_what_you_can_handle FAILED [ 66%]
...
E Failed: DID NOT RAISE IsADirectoryError
show the solutionnetlab/configs.py
or pull it into your lab: make solve T=4.2, then git diff --staged shows what changed
def load_configs_safe(folder):
"""Like load_configs, but a file that can't be read doesn't stop the others.
Returns two dicts: (configs, problems). configs is device name -> text for every file that read
cleanly. problems is file name (like "r9.cfg") -> a short reason, for every *.cfg file that
couldn't be read: one you're not allowed to read (PermissionError), or one that isn't UTF-8 text
(UnicodeDecodeError). Catch only those two: anything else is a bug, and should still stop you.
"""
configs, problems = {}, {}
for path in sorted(pathlib.Path(folder).glob("*.cfg")):
try:
configs[path.stem] = path.read_text(encoding="utf-8")
except PermissionError:
problems[path.name] = "permission denied"
except UnicodeDecodeError as e:
problems[path.name] = f"not UTF-8 text (byte {e.start})"
return configs, problems
The function only collects the problems; the script decides what to do with them: print each one to
stderr, so they don’t mix with the normal output, and exit 1, so a cron job or a pipeline can tell. The check
builds a folder of its own with a Latin-1 file and a locked one under made-up names, and a third folder with
a directory called not-a-file.cfg. When you’re done, make reset removes the two files from configs/.
5 check: run it, then prove it
$ make check T=4.2when it's right, you'll see
task 4.2 checks/test_4_2.py
checks/test_4_2.py::test_function PASSED [ 33%]
checks/test_4_2.py::test_only_what_you_can_handle PASSED [ 66%]
checks/test_4_2.py::test_the_script PASSED [100%]
============================== 3 passed in 0.02s ===============================
✓ task 4.2 done