online posts 16
linux and the command line 101

Pipes, Redirection and grep: One Question, One Pipeline (Linux CLI 101, Lesson 3)

Every Linux command reads from stdin and writes to stdout and stderr. Redirect those streams to files, pipe one command into the next, and grep, sort and uniq answer real questions about a router’s log in one line.

Linux CLI 101 lesson 3 cover: grep, cut, sort and uniq -c piped together, counting LINK_DOWN events per router interface
lesson 3 of 8 · course Linux and the command line 101
  1. 1The shell and the filesystem
  2. 2Working with files
  3. 3Pipes, redirection and grep
  4. 4Users, permissions and sudo
  5. 5Processes, environment and PATH
  6. 6SSH and keys
  7. 7A little Bash scripting
  8. 8The network from the command line
course overview →

Every command you run in a Linux shell has three streams: one coming in (stdin), one for its results (stdout) and one for its complaints (stderr). Redirection, with >, >> and 2>, sends a stream to a file. A pipe, |, connects one command’s stdout to the next command’s stdin. Put small tools in a row like that, grep to pick lines, sort to order them, uniq -c to count them, and one line answers a question that would take you ten minutes with a mouse.

This is the lesson where the shell stops being a place to look at files and starts being a tool for questions about them. The examples all run on r1.log, a real syslog captured from the lab’s CapeOS router.


What are stdin, stdout and stderr?

A stream is a flow of text into or out of a program. Every program starts with three of them already open:

StreamNumberDefaultUsed for
stdin (standard input)0your keyboardwhat the program reads
stdout (standard output)1your terminalthe results
stderr (standard error)2your terminalerrors and warnings

stdout and stderr both land on your screen, so they look like one thing. They aren’t. Ask ls for one file that exists and one that doesn’t:

$ ls nosuch r1.log
ls: cannot access 'nosuch': No such file or directory
r1.log

The first line went to stderr, the second to stdout. The split is deliberate: it means you can save or process the results without the error messages mixed in, and still see the errors on screen.

The numbers matter because redirection uses them. 2> means “stream 2”, which is stderr.


How do I send output to a file?

With redirection: you tell the shell, before the command even starts, to point one of its streams somewhere else. The program doesn’t know or care. It writes to stdout as usual, and the shell has already connected stdout to a file.

You writeWhat happens
cmd > filestdout goes to file. The file is created, or emptied first if it exists
cmd >> filestdout is added to the end of file
cmd 2> filestderr goes to file
cmd > file 2>&1stdout goes to file, then stderr goes wherever stdout goes: both in one file
cmd 2> /dev/nullstderr is thrown away. /dev/null is a special file that swallows everything
cmd < filestdin comes from file instead of the keyboard

Here’s the same ls with stdout sent to a file. The error still reaches the screen, because only stream 1 moved:

$ ls nosuch r1.log > out.txt
ls: cannot access 'nosuch': No such file or directory
$ cat out.txt
r1.log

And with stderr sent to a file instead, the screen stays quiet and the complaint is in err.txt:

$ ls nosuch 2> err.txt
$ cat err.txt
ls: cannot access 'nosuch': No such file or directory

2>&1 reads as “send stream 2 to wherever stream 1 is going right now”. Order matters: the shell processes redirections left to right, so > file 2>&1 puts both in the file, and 2>&1 > file doesn’t (stderr was pointed at the terminal before stdout moved). The Bash manual’s redirection section has the full list.

A real use for 2> /dev/null: apt, Debian’s package tool, warns on stderr whenever its output goes anywhere but a terminal:

$ apt list --installed 2>&1 | head -3
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Listing...

Add 2>/dev/null and the warning is gone, while the list itself is untouched.


What does a pipe do?

A pipe, written |, connects the stdout of the command on its left to the stdin of the command on its right. No file in between: the text flows straight from one program into the next. A row of commands joined by pipes is a pipeline.

$ wc -l < r1.log
37

wc -l counts lines. The log has 37. Now count only the successful logins: grep picks the matching lines and hands them on.

$ grep LOGIN_OK r1.log
*Nov 11 05:42:32.023: %CAPEOS-5-LOGIN_OK: Login by admin from 172.80.102.1 port 60330
*Nov 11 05:43:18.153: %CAPEOS-5-LOGIN_OK: Login by admin from 172.80.102.1 port 41092
*Nov 11 05:44:03.607: %CAPEOS-5-LOGIN_OK: Login by admin from 172.80.102.1 port 35784
$ grep LOGIN_OK r1.log | wc -l
3

This is the Unix idea in one line: small tools that each do one job, joined together. grep doesn’t count, wc doesn’t search, and neither needs to.

Only stdout goes through a pipe. stderr still goes to the screen, which is usually what you want:

$ ls nosuch r1.log | wc -l
ls: cannot access 'nosuch': No such file or directory
1
$ ls nosuch r1.log 2>&1 | wc -l
2

The first wc counted one line, because the error never entered the pipe. With 2>&1 it counted both.


Which small tools are worth knowing first?

Five do most of the work in log pipelines. All of them read stdin when you don’t give them a file name, which is what makes them pipe-friendly. They’re part of GNU coreutils (version 9.1 in the codespace).

ToolJobOptions you’ll use
wccounts lines, words, bytes-l lines only
sortputs lines in order-n by number, -r reverse, -u drop repeats
uniqcollapses adjacent repeated lines-c prefix each with its count
cutkeeps some columns of each line-d the separator, -f which fields
teecopies stdin to a file and to stdout-a append

cut treats each line as fields split by one character. Split the log’s lines on : and the fourth field is the message type:

$ cut -d: -f4 r1.log | head -3
 %CAPEOS-5-RESTART
 %CAPEOS-5-LOGIN_OK
 %CAPEOS-3-LINK_DOWN

(The time stamp holds two colons of its own, which is why the type is field 4, not 2. Counting fields is the fiddly part of cut.)

tee is for when you want to keep a copy of what’s flowing through a pipe and keep going. Here it saves the list of message types to a file while wc counts them:

$ grep -o "%CAPEOS-[0-9]-[A-Z_]*" r1.log | sort -u | tee types.txt | wc -l
8

How does sort | uniq -c count things?

This is the pattern you’ll use more than any other: pull out the thing you want to count, sort it so that equal lines sit together, count each run with uniq -c, then sort by the count, biggest first. Here’s every message type in the router’s log, counted:

$ grep -o "%CAPEOS-[0-9]-[A-Z_]*" r1.log | sort | uniq -c | sort -rn
      9 %CAPEOS-5-CONFIG_DONE
      8 %CAPEOS-4-LOGIN_FAILED
      4 %CAPEOS-5-PROTO_UP
      4 %CAPEOS-5-PROTO_DOWN
      4 %CAPEOS-3-LINK_UP
      4 %CAPEOS-3-LINK_DOWN
      3 %CAPEOS-5-LOGIN_OK
      1 %CAPEOS-5-RESTART

Read it left to right:

  1. grep -o prints only the part of each line that matched, not the whole line: one message type per line.
  2. sort puts identical types next to each other.
  3. uniq -c turns each run of identical lines into one line with a count in front.
  4. sort -rn sorts by that number (-n), largest first (-r).

In one line you’ve learned something useful: eight failed logins in a log of 37 lines, more than the successful ones. That’s the kind of thing worth a second look on a real router. (The four ties come out Z to A because -r reverses the whole comparison, including the alphabetical tie-break.)

Why the first sort? Because uniq only compares each line with the one before it. Skip the sort and it counts runs, not totals:

$ grep -o "%CAPEOS-[0-9]-[A-Z_]*" r1.log | uniq -c | head -5
      1 %CAPEOS-5-RESTART
      1 %CAPEOS-5-LOGIN_OK
      1 %CAPEOS-3-LINK_DOWN
      1 %CAPEOS-5-PROTO_DOWN
      2 %CAPEOS-5-CONFIG_DONE

No error, just wrong numbers. That’s the worst kind of mistake, so make sort | uniq -c a single habit in your head.


What can grep do besides find a word?

grep prints the lines that match a pattern. The codespace has GNU grep 3.8, and the GNU grep manual covers everything. These options cover most days:

OptionMeansExample on the lab’s files
-iignore casegrep -i "login by" matches Login by
-vinvert: lines that don’t matchgrep -v CAPEOS-5 drops the severity 5 noise
-cprint a count of matching linesgrep -c LOGIN_OK r1.log prints 3
-rsearch every file under a foldergrep -r hostname files/configs
-oprint only the matching partused in the counting pipeline above
-Eextended regular expressionsgrep -E "LINK_(UP|DOWN)"
-nadd line numbershandy when you’ll open the file next

Case matters by default. The log says Login by, so this finds nothing:

$ grep "login by" r1.log | wc -l
0
$ grep -i "login by" r1.log | wc -l
3

-r turns grep into a search across a whole tree, and it prefixes each match with the file it came from. On the lab’s saved configs:

$ grep -r hostname files/configs
files/configs/nyc-leaf1.cfg:hostname nyc-leaf1
files/configs/nyc-leaf2.cfg:hostname nyc-leaf2
files/configs/chi leaf2.cfg:hostname chi-leaf2
files/configs/chi-leaf1.cfg:hostname chi-leaf1

That’s “which file has the config for chi-leaf2?”, answered without opening any of them. Notice chi leaf2.cfg, with a space in its name, and a hostname that doesn’t match it. Real backup folders look like this.

-c counts lines, not matches. A line that matches twice counts once. Usually that’s what you want with logs, where one event is one line.


What is a regular expression?

A regular expression, or regex, is a pattern that describes text instead of spelling it out. Every grep pattern is one: LOGIN_OK is a regex that happens to match only itself. The power is in the characters that mean something else.

PatternMatches
.any one character
[0-9]one digit; [A-Z_] one capital letter or underscore
*the thing before it, any number of times, including none
^ and $the start and the end of the line
(UP|DOWN)UP or DOWN (with -E)
+, ?one or more; zero or one (with -E)

So %CAPEOS-[0-9]-[A-Z_]* reads: a percent sign, CAPEOS-, one digit, a dash, then any run of capitals and underscores. That’s every message type, without listing them. And ^\*Nov 11 05:43 would match only lines whose time stamp starts in that minute.

Two flavours matter. Plain grep uses basic regular expressions, where +, ?, | and () are ordinary characters unless you put a backslash before them. grep -E uses extended ones, where they’re special straight away. Use -E whenever your pattern has an “or” in it; it’s easier to read:

$ grep -cE "LINK_(UP|DOWN)" r1.log
8

Quote your patterns. Without the quotes, the shell sees *, | and ( first and tries to use them itself, as a glob, a pipe and a syntax error.

This is a first taste. Python uses the same ideas, with more power (named groups, pulling fields out instead of just matching lines), and Python 101’s regex lesson picks up from here. The difference in practice: grep answers “which lines?”, and Python answers “which values, as data?”.


What I see go wrong

> when you meant >>. > empties the file before the command runs. Running a report twice with > keeps only the second run. That’s fine for a report you regenerate, and a disaster for a log you meant to add to.

Redirecting a file onto itself. sort r1.log > r1.log doesn’t sort the file, it destroys it. The shell empties r1.log to set up the redirect before sort starts reading, so sort reads an empty file:

$ sort copy.log > copy.log
$ wc -l copy.log
0 copy.log

Write to a new file, then rename it.

uniq without sort. Wrong counts, no warning. See above.

Errors that vanish into a pipe, or don’t. If a pipeline’s output looks short, check whether the first command wrote errors to the screen that you scrolled past. If you’re saving to a file and the file has errors in it, you used 2>&1 and maybe didn’t mean to.

grep -c on several files. It prints one count per file (r1.log:3), not a total. For a total, pipe grep into wc -l.

Unquoted patterns. grep LINK_(UP|DOWN) r1.log without quotes is a shell syntax error, not a grep one. If an error message mentions bash: and not grep:, the shell got there first.

Case. The log says LOGIN_FAILED and Login failed. A pattern in the wrong case matches nothing, quietly. -i when in doubt.


Try it

Chapter 3 of the lab gives you the same r1.log and a list of questions about it: lines about one interface, lines that aren’t config changes, link-down events per interface, the severe lines saved to a report with nothing on the screen. Then a challenge: who keeps failing to log in to the router, and how often. Each answer is a one-line pipeline, and the checker runs yours on logs it has never seen, so it has to answer the question, not just match the numbers in this file.

Lesson 2 covered reading these files with less, head and tail -f; tail -f r1.log | grep LINK is the live version of everything above. If this was your first time in a terminal, the Meet the lab page explains the checker and make.

What’s next

Lesson 4 is about who may read and change a file: users, groups, the rwx columns in ls -l, and why some files, like a private SSH key, must be readable by you alone.