online posts 16
linux and the command line 101

task 3.1 · chapter 3 of 8: pipes, redirection and grep · 30 min

Pipelines

objective

Answer questions about r1’s log with one line each: count with grep -c, filter with grep and grep -v, count per interface with cut | sort | uniq -c | sort -rn, and send output to a file with >.

you edit, in the labanswers/3.1.yml

open a file in VS Code: Ctrl+P (Cmd+P on a Mac), type its name, Enter

given
LessonPipes, redirection and grep
The logfiles/logs/r1.log. Your commands say "$LOG" instead of its name
Try oneLOG=files/logs/r1.log once in your terminal, then paste a command
Your answersanswers/3.1.yml: one command per line
new to this? step by stepbeginner
  1. LOG=files/logs/r1.log, then try each command with "$LOG" where the file name goes.
  2. grep -c GigabitEthernet2 "$LOG", grep LOGIN_FAILED "$LOG", grep -vc CONFIG_DONE "$LOG".
  3. Challenge: grep LINK_DOWN "$LOG" | cut -d' ' -f6 | sort | uniq -c | sort -rn, and a grep with > reports/errors.txt.
  4. Put each command on its line in answers/3.1.yml, then make check T=3.1.

more in meet the lab

1 explain

Small tools, joined up

Every program has three streams

When a program runs, it gets three streams, numbered:

#NameNormally
0stdin, standard inputwhat you type
1stdout, standard outputthe screen
2stderr, standard errorthe screen too, kept apart for error messages

The shell can point each one somewhere else:

You typeMeans
cmd > filestdout into file, replacing what was in it
cmd >> filestdout onto the end of file
cmd 2> filestderr into file (2>/dev/null throws errors away)
cmd1 | cmd2a pipe: cmd1’s stdout becomes cmd2’s stdin

in one sentence > sends output to a file, | sends it into the next command, and errors travel separately on stderr.

The tools that go in a pipeline

ToolDoes
grep WORDkeeps the lines that contain WORD. -c counts them, -v keeps the ones that don’t, -i ignores case
cut -d' ' -f6keeps the 6th field of each line, splitting on spaces (-d is the delimiter)
sortsorts lines. -n by number, -r reversed
uniq -csquashes repeated neighboring lines into one, with a count in front. That’s why it comes after sort
wc -lcounts lines

Why the commands say “$LOG”

LOG=files/logs/r1.log sets a shell variable called LOG, and "$LOG" is replaced with its value. The checker sets LOG to other log files and runs your command again on each, so a command that only works on r1.log, or that types the answer in, won’t pass. Chapter 5 explains variables properly.

2 show

One question, one line

The LINK_DOWN lines, then the same lines cut down to field 6, the interface:

$ grep LINK_DOWN "$LOG"
*Nov 11 05:42:33.264: %CAPEOS-3-LINK_DOWN: Interface GigabitEthernet2 is down (administratively)
*Nov 11 05:42:55.209: %CAPEOS-3-LINK_DOWN: Interface GigabitEthernet1 is down (administratively)
*Nov 11 05:43:19.370: %CAPEOS-3-LINK_DOWN: Interface GigabitEthernet2 is down (administratively)
*Nov 11 05:43:49.571: %CAPEOS-3-LINK_DOWN: Interface GigabitEthernet2 is down (administratively)
$ grep LINK_DOWN "$LOG" | cut -d' ' -f6
GigabitEthernet2
GigabitEthernet1
GigabitEthernet2
GigabitEthernet2

Counting fields: *Nov is 1, 11 is 2, the time is 3, %CAPEOS-3-LINK_DOWN: is 4, Interface is 5, and the interface is 6. And an error, on stderr, then the same with stderr thrown away:

$ cat nope.txt
cat: nope.txt: No such file or directory
$ cat nope.txt 2>/dev/null
$
3 follow along

Three questions

  1. Point LOG at the practice log. Type it exactly, with no spaces around the =:

    run in the terminal
    LOG=files/logs/r1.log
    
  2. How many lines mention GigabitEthernet2?

    run in the terminal
    grep -c GigabitEthernet2 "$LOG"
    
  3. Only the failed logins:

    run in the terminal
    grep LOGIN_FAILED "$LOG"
    
  4. How many lines are not CONFIG_DONE? -v turns grep inside out, and -c counts what’s left:

    run in the terminal
    grep -vc CONFIG_DONE "$LOG"
    
  5. Copy each command into answers/3.1.yml, on the line after its question. Leave "$LOG" as it is:

    editanswers/3.1.ymlCtrl+P → 3.1.yml
    gi2_lines: grep -c GigabitEthernet2 "$LOG"
    failed_lines: grep LOGIN_FAILED "$LOG"
    not_config: grep -vc CONFIG_DONE "$LOG"
    

in one sentence grep keeps matching lines, -c counts them, -v keeps the others.

4 challenge

Per interface, and into a file

Two more lines in answers/3.1.yml:

  1. downs_per_interface: for each interface, how many times it went down (LINK_DOWN), biggest count first, one line each, the way uniq -c prints it:

          3 GigabitEthernet2
          1 GigabitEthernet1
    
  2. save_errors: every severity 3 line (they have -3- in them) into reports/errors.txt, with nothing on the screen.

Build each one a step at a time in the terminal, adding one | and one tool, and look at the output before you add the next. Then run make check T=3.1.

hint

The first starts from the show stage: grep LINK_DOWN "$LOG" | cut -d' ' -f6. What’s left is counting the same lines, and putting the biggest first.

hint

uniq -c only counts lines that are next to each other, so sort first. Then sort -rn sorts the counts as numbers, biggest first.

hint

To grep for something that starts with -, tell grep the options are over with --: grep -- -3- "$LOG". Otherwise it thinks -3- is an option.

show the solution
downs_per_interface: grep LINK_DOWN "$LOG" | cut -d' ' -f6 | sort | uniq -c | sort -rn
save_errors: grep -- -3- "$LOG" > reports/errors.txt

cat reports/errors.txt shows the eight severity 3 lines: every LINK_DOWN and LINK_UP. Without the first sort, uniq -c counts runs instead of totals; without sort -rn at the end, the interfaces come out in alphabetical order. Both pass on r1.log by luck sometimes, which is why the checker tries other logs.

5 check: run it, then prove it

$ make check T=3.1

when it's right, you'll see

╭─ task 3.1 · pipelines
│  ✓ how many lines mention GigabitEthernet2 (follow along): right on r1.log and on three logs it hadn't seen
│  ✓ only the LOGIN_FAILED lines (follow along): right on r1.log and on three logs it hadn't seen
│  ✓ how many lines are not CONFIG_DONE (follow along): right on r1.log and on three logs it hadn't seen
│  ✓ LINK_DOWN events per interface, most first (challenge): right on r1.log and on three logs it hadn't seen
│  ✓ every severity 3 line into reports/errors.txt (challenge): right on r1.log and on three logs it hadn't seen
╰─ 5/5 checks passed  verified