task 3.1 · chapter 3 of 8: pipes, redirection and grep · 30 min
Pipelines
objective
Answer questions about r1’s log with one line each: count with grep -c, filter with grep and grep -v, count per interface with cut | sort | uniq -c | sort -rn, and send output to a file with >.
answers/3.1.ymlopen a file in VS Code: Ctrl+P (Cmd+P on a Mac), type its name, Enter
| Lesson | Pipes, redirection and grep |
|---|---|
| The log | files/logs/r1.log. Your commands say "$LOG" instead of its name |
| Try one | LOG=files/logs/r1.log once in your terminal, then paste a command |
| Your answers | answers/3.1.yml: one command per line |
new to this? step by stepbeginner
LOG=files/logs/r1.log, then try each command with"$LOG"where the file name goes.grep -c GigabitEthernet2 "$LOG",grep LOGIN_FAILED "$LOG",grep -vc CONFIG_DONE "$LOG".- Challenge:
grep LINK_DOWN "$LOG" | cut -d' ' -f6 | sort | uniq -c | sort -rn, and agrepwith> reports/errors.txt. - Put each command on its line in answers/3.1.yml, then
make check T=3.1.
more in meet the lab
Small tools, joined up
Every program has three streams
When a program runs, it gets three streams, numbered:
| # | Name | Normally |
|---|---|---|
| 0 | stdin, standard input | what you type |
| 1 | stdout, standard output | the screen |
| 2 | stderr, standard error | the screen too, kept apart for error messages |
The shell can point each one somewhere else:
| You type | Means |
|---|---|
cmd > file | stdout into file, replacing what was in it |
cmd >> file | stdout onto the end of file |
cmd 2> file | stderr into file (2>/dev/null throws errors away) |
cmd1 | cmd2 | a pipe: cmd1’s stdout becomes cmd2’s stdin |
in one sentence > sends output to a file, | sends it into the next command, and errors travel separately on stderr.
The tools that go in a pipeline
| Tool | Does |
|---|---|
grep WORD | keeps the lines that contain WORD. -c counts them, -v keeps the ones that don’t, -i ignores case |
cut -d' ' -f6 | keeps the 6th field of each line, splitting on spaces (-d is the delimiter) |
sort | sorts lines. -n by number, -r reversed |
uniq -c | squashes repeated neighboring lines into one, with a count in front. That’s why it comes after sort |
wc -l | counts lines |
Why the commands say “$LOG”
LOG=files/logs/r1.log sets a shell variable called LOG, and "$LOG" is replaced with its value. The
checker sets LOG to other log files and runs your command again on each, so a command that only works on
r1.log, or that types the answer in, won’t pass. Chapter 5 explains variables properly.
One question, one line
The LINK_DOWN lines, then the same lines cut down to field 6, the interface:
$ grep LINK_DOWN "$LOG"
*Nov 11 05:42:33.264: %CAPEOS-3-LINK_DOWN: Interface GigabitEthernet2 is down (administratively)
*Nov 11 05:42:55.209: %CAPEOS-3-LINK_DOWN: Interface GigabitEthernet1 is down (administratively)
*Nov 11 05:43:19.370: %CAPEOS-3-LINK_DOWN: Interface GigabitEthernet2 is down (administratively)
*Nov 11 05:43:49.571: %CAPEOS-3-LINK_DOWN: Interface GigabitEthernet2 is down (administratively)
$ grep LINK_DOWN "$LOG" | cut -d' ' -f6
GigabitEthernet2
GigabitEthernet1
GigabitEthernet2
GigabitEthernet2
Counting fields: *Nov is 1, 11 is 2, the time is 3, %CAPEOS-3-LINK_DOWN: is 4, Interface is 5, and the
interface is 6. And an error, on stderr, then the same with stderr thrown away:
$ cat nope.txt
cat: nope.txt: No such file or directory
$ cat nope.txt 2>/dev/null
$
Three questions
Point
LOGat the practice log. Type it exactly, with no spaces around the=:run in the terminalLOG=files/logs/r1.logHow many lines mention GigabitEthernet2?
run in the terminalgrep -c GigabitEthernet2 "$LOG"Only the failed logins:
run in the terminalgrep LOGIN_FAILED "$LOG"How many lines are not CONFIG_DONE?
-vturns grep inside out, and-ccounts what’s left:run in the terminalgrep -vc CONFIG_DONE "$LOG"Copy each command into answers/3.1.yml, on the line after its question. Leave
"$LOG"as it is:editanswers/3.1.ymlCtrl+P → 3.1.ymlgi2_lines: grep -c GigabitEthernet2 "$LOG" failed_lines: grep LOGIN_FAILED "$LOG" not_config: grep -vc CONFIG_DONE "$LOG"
in one sentence grep keeps matching lines, -c counts them, -v keeps the others.
Per interface, and into a file
Two more lines in answers/3.1.yml:
downs_per_interface: for each interface, how many times it went down (LINK_DOWN), biggest count first, one line each, the wayuniq -cprints it:3 GigabitEthernet2 1 GigabitEthernet1save_errors: every severity 3 line (they have-3-in them) intoreports/errors.txt, with nothing on the screen.
Build each one a step at a time in the terminal, adding one | and one tool, and look at the output before
you add the next. Then run make check T=3.1.
hint
The first starts from the show stage: grep LINK_DOWN "$LOG" | cut -d' ' -f6. What’s left is counting the
same lines, and putting the biggest first.
hint
uniq -c only counts lines that are next to each other, so sort first. Then sort -rn sorts the counts as
numbers, biggest first.
hint
To grep for something that starts with -, tell grep the options are over with --: grep -- -3- "$LOG".
Otherwise it thinks -3- is an option.
show the solution
downs_per_interface: grep LINK_DOWN "$LOG" | cut -d' ' -f6 | sort | uniq -c | sort -rn
save_errors: grep -- -3- "$LOG" > reports/errors.txt
cat reports/errors.txt shows the eight severity 3 lines: every LINK_DOWN and LINK_UP. Without the first
sort, uniq -c counts runs instead of totals; without sort -rn at the end, the interfaces come out in
alphabetical order. Both pass on r1.log by luck sometimes, which is why the checker tries other logs.
5 check: run it, then prove it
$ make check T=3.1when it's right, you'll see
╭─ task 3.1 · pipelines
│ ✓ how many lines mention GigabitEthernet2 (follow along): right on r1.log and on three logs it hadn't seen
│ ✓ only the LOGIN_FAILED lines (follow along): right on r1.log and on three logs it hadn't seen
│ ✓ how many lines are not CONFIG_DONE (follow along): right on r1.log and on three logs it hadn't seen
│ ✓ LINK_DOWN events per interface, most first (challenge): right on r1.log and on three logs it hadn't seen
│ ✓ every severity 3 line into reports/errors.txt (challenge): right on r1.log and on three logs it hadn't seen
╰─ 5/5 checks passed verified