<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/"><channel><title>Syslog · Network Doodles</title><link>https://networkdoodles.com/tags/syslog/</link><description>Network automation, from first playbook to agentic operations. Ansible, NetBox, MCP, Python and the occasional doodle.</description><language>en-us</language><managingEditor>David Henderson</managingEditor><lastBuildDate>Mon, 16 Nov 2026 09:00:00 -0500</lastBuildDate><atom:link href="https://networkdoodles.com/tags/syslog/index.xml" rel="self" type="application/rss+xml"/><item><title>Pipes, Redirection and grep: One Question, One Pipeline (Linux CLI 101, Lesson 3)</title><link>https://networkdoodles.com/lessons/linux-cli-101/pipes-redirection-grep/</link><pubDate>Mon, 16 Nov 2026 09:00:00 -0500</pubDate><dc:creator>David Henderson</dc:creator><guid>https://networkdoodles.com/lessons/linux-cli-101/pipes-redirection-grep/</guid><category>linux</category><category>networking-basics</category><category>syslog</category><category>troubleshooting</category><category>capeos</category><description>stdin, stdout and stderr, redirecting them to files, and pipes that join grep, cut, sort and uniq to answer real questions about a router's log in one line.</description><content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[<p>Every command you run in a Linux shell has three streams: one coming in (<strong>stdin</strong>), one for its results
(<strong>stdout</strong>) and one for its complaints (<strong>stderr</strong>). Redirection, with <code>&gt;</code>, <code>&gt;&gt;</code> and <code>2&gt;</code>, sends a stream to a
file. A pipe, <code>|</code>, connects one command&rsquo;s stdout to the next command&rsquo;s stdin. Put small tools in a row like that,
<code>grep</code> to pick lines, <code>sort</code> to order them, <code>uniq -c</code> to count them, and one line answers a question that would
take you ten minutes with a mouse.</p>
<p>This is the lesson where the shell stops being a place to look at files and starts being a tool for questions
about them. The examples all run on <code>r1.log</code>, a real syslog captured from the lab&rsquo;s CapeOS router.</p>
<hr>
<h2 id="what-are-stdin-stdout-and-stderr">What are stdin, stdout and stderr?</h2>
<p>A <strong>stream</strong> is a flow of text into or out of a program. Every program starts with three of them already open:</p>
<table>
  <thead>
      <tr>
          <th>Stream</th>
          <th>Number</th>
          <th>Default</th>
          <th>Used for</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><strong>stdin</strong> (standard input)</td>
          <td>0</td>
          <td>your keyboard</td>
          <td>what the program reads</td>
      </tr>
      <tr>
          <td><strong>stdout</strong> (standard output)</td>
          <td>1</td>
          <td>your terminal</td>
          <td>the results</td>
      </tr>
      <tr>
          <td><strong>stderr</strong> (standard error)</td>
          <td>2</td>
          <td>your terminal</td>
          <td>errors and warnings</td>
      </tr>
  </tbody>
</table>
<p>stdout and stderr both land on your screen, so they look like one thing. They aren&rsquo;t. Ask <code>ls</code> for one file
that exists and one that doesn&rsquo;t:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ ls nosuch r1.log
</span></span><span class="line"><span class="cl">ls: cannot access &#39;nosuch&#39;: No such file or directory
</span></span><span class="line"><span class="cl">r1.log
</span></span></code></pre></div><p>The first line went to stderr, the second to stdout. The split is deliberate: it means you can save or process
the results without the error messages mixed in, and still see the errors on screen.</p>
<p>The numbers matter because redirection uses them. <code>2&gt;</code> means &ldquo;stream 2&rdquo;, which is stderr.</p>
<hr>
<h2 id="how-do-i-send-output-to-a-file">How do I send output to a file?</h2>
<p>With <strong>redirection</strong>: you tell the shell, before the command even starts, to point one of its streams somewhere
else. The program doesn&rsquo;t know or care. It writes to stdout as usual, and the shell has already connected stdout
to a file.</p>
<table>
  <thead>
      <tr>
          <th>You write</th>
          <th>What happens</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>cmd &gt; file</code></td>
          <td>stdout goes to <code>file</code>. The file is created, or <strong>emptied first</strong> if it exists</td>
      </tr>
      <tr>
          <td><code>cmd &gt;&gt; file</code></td>
          <td>stdout is added to the end of <code>file</code></td>
      </tr>
      <tr>
          <td><code>cmd 2&gt; file</code></td>
          <td>stderr goes to <code>file</code></td>
      </tr>
      <tr>
          <td><code>cmd &gt; file 2&gt;&amp;1</code></td>
          <td>stdout goes to <code>file</code>, then stderr goes wherever stdout goes: both in one file</td>
      </tr>
      <tr>
          <td><code>cmd 2&gt; /dev/null</code></td>
          <td>stderr is thrown away. <code>/dev/null</code> is a special file that swallows everything</td>
      </tr>
      <tr>
          <td><code>cmd &lt; file</code></td>
          <td>stdin comes from <code>file</code> instead of the keyboard</td>
      </tr>
  </tbody>
</table>
<p>Here&rsquo;s the same <code>ls</code> with stdout sent to a file. The error still reaches the screen, because only stream 1 moved:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ ls nosuch r1.log &gt; out.txt
</span></span><span class="line"><span class="cl">ls: cannot access &#39;nosuch&#39;: No such file or directory
</span></span><span class="line"><span class="cl">$ cat out.txt
</span></span><span class="line"><span class="cl">r1.log
</span></span></code></pre></div><p>And with stderr sent to a file instead, the screen stays quiet and the complaint is in <code>err.txt</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ ls nosuch 2&gt; err.txt
</span></span><span class="line"><span class="cl">$ cat err.txt
</span></span><span class="line"><span class="cl">ls: cannot access &#39;nosuch&#39;: No such file or directory
</span></span></code></pre></div><p><code>2&gt;&amp;1</code> reads as &ldquo;send stream 2 to wherever stream 1 is going right now&rdquo;. Order matters: the shell processes
redirections left to right, so <code>&gt; file 2&gt;&amp;1</code> puts both in the file, and <code>2&gt;&amp;1 &gt; file</code> doesn&rsquo;t (stderr was
pointed at the terminal before stdout moved). The <a href="https://www.gnu.org/software/bash/manual/html_node/Redirections.html">Bash manual&rsquo;s redirection
section</a> has the full list.</p>
<p>A real use for <code>2&gt; /dev/null</code>: <code>apt</code>, Debian&rsquo;s package tool, warns on stderr whenever its output goes anywhere
but a terminal:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ apt list --installed 2&gt;&amp;1 | head -3
</span></span><span class="line"><span class="cl">WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">Listing...
</span></span></code></pre></div><p>Add <code>2&gt;/dev/null</code> and the warning is gone, while the list itself is untouched.</p>
<hr>
<h2 id="what-does-a-pipe-do">What does a pipe do?</h2>
<p>A <strong>pipe</strong>, written <code>|</code>, connects the stdout of the command on its left to the stdin of the command on its right.
No file in between: the text flows straight from one program into the next. A row of commands joined by pipes is
a <strong>pipeline</strong>.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ wc -l &lt; r1.log
</span></span><span class="line"><span class="cl">37
</span></span></code></pre></div><p><code>wc -l</code> counts lines. The log has 37. Now count only the successful logins: <code>grep</code> picks the matching lines and
hands them on.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ grep LOGIN_OK r1.log
</span></span><span class="line"><span class="cl">*Nov 11 05:42:32.023: %CAPEOS-5-LOGIN_OK: Login by admin from 172.80.102.1 port 60330
</span></span><span class="line"><span class="cl">*Nov 11 05:43:18.153: %CAPEOS-5-LOGIN_OK: Login by admin from 172.80.102.1 port 41092
</span></span><span class="line"><span class="cl">*Nov 11 05:44:03.607: %CAPEOS-5-LOGIN_OK: Login by admin from 172.80.102.1 port 35784
</span></span><span class="line"><span class="cl">$ grep LOGIN_OK r1.log | wc -l
</span></span><span class="line"><span class="cl">3
</span></span></code></pre></div><p>This is the Unix idea in one line: <strong>small tools that each do one job</strong>, joined together. <code>grep</code> doesn&rsquo;t count,
<code>wc</code> doesn&rsquo;t search, and neither needs to.</p>
<p>Only stdout goes through a pipe. stderr still goes to the screen, which is usually what you want:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ ls nosuch r1.log | wc -l
</span></span><span class="line"><span class="cl">ls: cannot access &#39;nosuch&#39;: No such file or directory
</span></span><span class="line"><span class="cl">1
</span></span><span class="line"><span class="cl">$ ls nosuch r1.log 2&gt;&amp;1 | wc -l
</span></span><span class="line"><span class="cl">2
</span></span></code></pre></div><p>The first <code>wc</code> counted one line, because the error never entered the pipe. With <code>2&gt;&amp;1</code> it counted both.</p>
<hr>
<h2 id="which-small-tools-are-worth-knowing-first">Which small tools are worth knowing first?</h2>
<p>Five do most of the work in log pipelines. All of them read stdin when you don&rsquo;t give them a file name, which is
what makes them pipe-friendly. They&rsquo;re part of <a href="https://www.gnu.org/software/coreutils/manual/">GNU coreutils</a>
(version 9.1 in the codespace).</p>
<table>
  <thead>
      <tr>
          <th>Tool</th>
          <th>Job</th>
          <th>Options you&rsquo;ll use</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>wc</code></td>
          <td>counts lines, words, bytes</td>
          <td><code>-l</code> lines only</td>
      </tr>
      <tr>
          <td><code>sort</code></td>
          <td>puts lines in order</td>
          <td><code>-n</code> by number, <code>-r</code> reverse, <code>-u</code> drop repeats</td>
      </tr>
      <tr>
          <td><code>uniq</code></td>
          <td>collapses <strong>adjacent</strong> repeated lines</td>
          <td><code>-c</code> prefix each with its count</td>
      </tr>
      <tr>
          <td><code>cut</code></td>
          <td>keeps some columns of each line</td>
          <td><code>-d</code> the separator, <code>-f</code> which fields</td>
      </tr>
      <tr>
          <td><code>tee</code></td>
          <td>copies stdin to a file <strong>and</strong> to stdout</td>
          <td><code>-a</code> append</td>
      </tr>
  </tbody>
</table>
<p><code>cut</code> treats each line as fields split by one character. Split the log&rsquo;s lines on <code>:</code> and the fourth field is the
message type:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ cut -d: -f4 r1.log | head -3
</span></span><span class="line"><span class="cl"> %CAPEOS-5-RESTART
</span></span><span class="line"><span class="cl"> %CAPEOS-5-LOGIN_OK
</span></span><span class="line"><span class="cl"> %CAPEOS-3-LINK_DOWN
</span></span></code></pre></div><p>(The time stamp holds two colons of its own, which is why the type is field 4, not 2. Counting fields is the
fiddly part of <code>cut</code>.)</p>
<p><code>tee</code> is for when you want to keep a copy of what&rsquo;s flowing through a pipe and keep going. Here it saves the list
of message types to a file while <code>wc</code> counts them:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ grep -o &#34;%CAPEOS-[0-9]-[A-Z_]*&#34; r1.log | sort -u | tee types.txt | wc -l
</span></span><span class="line"><span class="cl">8
</span></span></code></pre></div><hr>
<h2 id="how-does-sort--uniq--c-count-things">How does sort | uniq -c count things?</h2>
<p>This is the pattern you&rsquo;ll use more than any other: pull out the thing you want to count, sort it so that equal
lines sit together, count each run with <code>uniq -c</code>, then sort by the count, biggest first. Here&rsquo;s every message
type in the router&rsquo;s log, counted:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ grep -o &#34;%CAPEOS-[0-9]-[A-Z_]*&#34; r1.log | sort | uniq -c | sort -rn
</span></span><span class="line"><span class="cl">      9 %CAPEOS-5-CONFIG_DONE
</span></span><span class="line"><span class="cl">      8 %CAPEOS-4-LOGIN_FAILED
</span></span><span class="line"><span class="cl">      4 %CAPEOS-5-PROTO_UP
</span></span><span class="line"><span class="cl">      4 %CAPEOS-5-PROTO_DOWN
</span></span><span class="line"><span class="cl">      4 %CAPEOS-3-LINK_UP
</span></span><span class="line"><span class="cl">      4 %CAPEOS-3-LINK_DOWN
</span></span><span class="line"><span class="cl">      3 %CAPEOS-5-LOGIN_OK
</span></span><span class="line"><span class="cl">      1 %CAPEOS-5-RESTART
</span></span></code></pre></div><p>Read it left to right:</p>
<ol>
<li><code>grep -o</code> prints only the part of each line that matched, not the whole line: one message type per line.</li>
<li><code>sort</code> puts identical types next to each other.</li>
<li><code>uniq -c</code> turns each run of identical lines into one line with a count in front.</li>
<li><code>sort -rn</code> sorts by that number (<code>-n</code>), largest first (<code>-r</code>).</li>
</ol>
<p>In one line you&rsquo;ve learned something useful: eight failed logins in a log of 37 lines, more than the successful
ones. That&rsquo;s the kind of thing worth a second look on a real router. (The four ties come out Z to A because <code>-r</code>
reverses the whole comparison, including the alphabetical tie-break.)</p>
<p>Why the first <code>sort</code>? Because <code>uniq</code> only compares each line with the one before it. Skip the sort and it counts
runs, not totals:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ grep -o &#34;%CAPEOS-[0-9]-[A-Z_]*&#34; r1.log | uniq -c | head -5
</span></span><span class="line"><span class="cl">      1 %CAPEOS-5-RESTART
</span></span><span class="line"><span class="cl">      1 %CAPEOS-5-LOGIN_OK
</span></span><span class="line"><span class="cl">      1 %CAPEOS-3-LINK_DOWN
</span></span><span class="line"><span class="cl">      1 %CAPEOS-5-PROTO_DOWN
</span></span><span class="line"><span class="cl">      2 %CAPEOS-5-CONFIG_DONE
</span></span></code></pre></div><p>No error, just wrong numbers. That&rsquo;s the worst kind of mistake, so make <code>sort | uniq -c</code> a single habit in your
head.</p>
<hr>
<h2 id="what-can-grep-do-besides-find-a-word">What can grep do besides find a word?</h2>
<p><code>grep</code> prints the lines that match a pattern. The codespace has GNU grep 3.8, and the <a href="https://www.gnu.org/software/grep/manual/grep.html">GNU grep
manual</a> covers everything. These options cover most days:</p>
<table>
  <thead>
      <tr>
          <th>Option</th>
          <th>Means</th>
          <th>Example on the lab&rsquo;s files</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>-i</code></td>
          <td>ignore case</td>
          <td><code>grep -i &quot;login by&quot;</code> matches <code>Login by</code></td>
      </tr>
      <tr>
          <td><code>-v</code></td>
          <td>invert: lines that <strong>don&rsquo;t</strong> match</td>
          <td><code>grep -v CAPEOS-5</code> drops the severity 5 noise</td>
      </tr>
      <tr>
          <td><code>-c</code></td>
          <td>print a count of matching lines</td>
          <td><code>grep -c LOGIN_OK r1.log</code> prints <code>3</code></td>
      </tr>
      <tr>
          <td><code>-r</code></td>
          <td>search every file under a folder</td>
          <td><code>grep -r hostname files/configs</code></td>
      </tr>
      <tr>
          <td><code>-o</code></td>
          <td>print only the matching part</td>
          <td>used in the counting pipeline above</td>
      </tr>
      <tr>
          <td><code>-E</code></td>
          <td>extended regular expressions</td>
          <td><code>grep -E &quot;LINK_(UP|DOWN)&quot;</code></td>
      </tr>
      <tr>
          <td><code>-n</code></td>
          <td>add line numbers</td>
          <td>handy when you&rsquo;ll open the file next</td>
      </tr>
  </tbody>
</table>
<p>Case matters by default. The log says <code>Login by</code>, so this finds nothing:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ grep &#34;login by&#34; r1.log | wc -l
</span></span><span class="line"><span class="cl">0
</span></span><span class="line"><span class="cl">$ grep -i &#34;login by&#34; r1.log | wc -l
</span></span><span class="line"><span class="cl">3
</span></span></code></pre></div><p><code>-r</code> turns <code>grep</code> into a search across a whole tree, and it prefixes each match with the file it came from.
On the lab&rsquo;s saved configs:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ grep -r hostname files/configs
</span></span><span class="line"><span class="cl">files/configs/nyc-leaf1.cfg:hostname nyc-leaf1
</span></span><span class="line"><span class="cl">files/configs/nyc-leaf2.cfg:hostname nyc-leaf2
</span></span><span class="line"><span class="cl">files/configs/chi leaf2.cfg:hostname chi-leaf2
</span></span><span class="line"><span class="cl">files/configs/chi-leaf1.cfg:hostname chi-leaf1
</span></span></code></pre></div><p>That&rsquo;s &ldquo;which file has the config for chi-leaf2?&rdquo;, answered without opening any of them. Notice <code>chi leaf2.cfg</code>,
with a space in its name, and a hostname that doesn&rsquo;t match it. Real backup folders look like this.</p>
<p><code>-c</code> counts <strong>lines</strong>, not matches. A line that matches twice counts once. Usually that&rsquo;s what you want with
logs, where one event is one line.</p>
<hr>
<h2 id="what-is-a-regular-expression">What is a regular expression?</h2>
<p>A <strong>regular expression</strong>, or <strong>regex</strong>, is a pattern that describes text instead of spelling it out. Every
<code>grep</code> pattern is one: <code>LOGIN_OK</code> is a regex that happens to match only itself. The power is in the characters
that mean something else.</p>
<table>
  <thead>
      <tr>
          <th>Pattern</th>
          <th>Matches</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>.</code></td>
          <td>any one character</td>
      </tr>
      <tr>
          <td><code>[0-9]</code></td>
          <td>one digit; <code>[A-Z_]</code> one capital letter or underscore</td>
      </tr>
      <tr>
          <td><code>*</code></td>
          <td>the thing before it, any number of times, including none</td>
      </tr>
      <tr>
          <td><code>^</code> and <code>$</code></td>
          <td>the start and the end of the line</td>
      </tr>
      <tr>
          <td><code>(UP|DOWN)</code></td>
          <td><code>UP</code> or <code>DOWN</code> (with <code>-E</code>)</td>
      </tr>
      <tr>
          <td><code>+</code>, <code>?</code></td>
          <td>one or more; zero or one (with <code>-E</code>)</td>
      </tr>
  </tbody>
</table>
<p>So <code>%CAPEOS-[0-9]-[A-Z_]*</code> reads: a percent sign, <code>CAPEOS-</code>, one digit, a dash, then any run of capitals and
underscores. That&rsquo;s every message type, without listing them. And <code>^\*Nov 11 05:43</code> would match only lines whose
time stamp starts in that minute.</p>
<p>Two flavours matter. Plain <code>grep</code> uses <strong>basic</strong> regular expressions, where <code>+</code>, <code>?</code>, <code>|</code> and <code>()</code> are ordinary
characters unless you put a backslash before them. <code>grep -E</code> uses <strong>extended</strong> ones, where they&rsquo;re special
straight away. Use <code>-E</code> whenever your pattern has an &ldquo;or&rdquo; in it; it&rsquo;s easier to read:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ grep -cE &#34;LINK_(UP|DOWN)&#34; r1.log
</span></span><span class="line"><span class="cl">8
</span></span></code></pre></div><p>Quote your patterns. Without the quotes, the shell sees <code>*</code>, <code>|</code> and <code>(</code> first and tries to use them itself,
as a glob, a pipe and a syntax error.</p>
<p>This is a first taste. Python uses the same ideas, with more power (named groups, pulling fields out instead of
just matching lines), and <a href="/lessons/python-101/regex-and-textfsm/">Python 101&rsquo;s regex lesson</a> picks up from
here. The difference in practice: <code>grep</code> answers &ldquo;which lines?&rdquo;, and Python answers &ldquo;which values, as data?&rdquo;.</p>
<hr>
<h2 id="what-i-see-go-wrong">What I see go wrong</h2>
<p><strong><code>&gt;</code> when you meant <code>&gt;&gt;</code>.</strong> <code>&gt;</code> empties the file before the command runs. Running a report twice with <code>&gt;</code> keeps
only the second run. That&rsquo;s fine for a report you regenerate, and a disaster for a log you meant to add to.</p>
<p><strong>Redirecting a file onto itself.</strong> <code>sort r1.log &gt; r1.log</code> doesn&rsquo;t sort the file, it destroys it. The shell
empties <code>r1.log</code> to set up the redirect before <code>sort</code> starts reading, so <code>sort</code> reads an empty file:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ sort copy.log &gt; copy.log
</span></span><span class="line"><span class="cl">$ wc -l copy.log
</span></span><span class="line"><span class="cl">0 copy.log
</span></span></code></pre></div><p>Write to a new file, then rename it.</p>
<p><strong><code>uniq</code> without <code>sort</code>.</strong> Wrong counts, no warning. See above.</p>
<p><strong>Errors that vanish into a pipe, or don&rsquo;t.</strong> If a pipeline&rsquo;s output looks short, check whether the first
command wrote errors to the screen that you scrolled past. If you&rsquo;re saving to a file and the file has errors
in it, you used <code>2&gt;&amp;1</code> and maybe didn&rsquo;t mean to.</p>
<p><strong><code>grep -c</code> on several files.</strong> It prints one count per file (<code>r1.log:3</code>), not a total. For a total, pipe
<code>grep</code> into <code>wc -l</code>.</p>
<p><strong>Unquoted patterns.</strong> <code>grep LINK_(UP|DOWN) r1.log</code> without quotes is a shell syntax error, not a grep one.
If an error message mentions <code>bash:</code> and not <code>grep:</code>, the shell got there first.</p>
<p><strong>Case.</strong> The log says <code>LOGIN_FAILED</code> and <code>Login failed</code>. A pattern in the wrong case matches nothing, quietly.
<code>-i</code> when in doubt.</p>
<hr>
<h2 id="try-it">Try it</h2>
<p><a href="/labs/linux-cli-101/task-3-1/">Chapter 3 of the lab</a> gives you the same <code>r1.log</code> and a list of questions about
it: lines about one interface, lines that aren&rsquo;t config changes, link-down events per interface, the severe
lines saved to a report with nothing on the screen. Then a challenge: who keeps failing to log in to the router,
and how often. Each answer is a one-line pipeline, and the checker runs yours on logs it has never seen, so it
has to answer the question, not just match the numbers in this file.</p>
<p><a href="/lessons/linux-cli-101/working-with-files/">Lesson 2</a> covered reading these files with <code>less</code>, <code>head</code> and
<code>tail -f</code>; <code>tail -f r1.log | grep LINK</code> is the live version of everything above. If this was your first
time in a terminal, the <a href="/labs/basics/">Meet the lab</a> page explains the checker and <code>make</code>.</p>
<h2 id="whats-next">What&rsquo;s next</h2>
<p><a href="/lessons/linux-cli-101/users-permissions-sudo/">Lesson 4</a> is about who may read and change a file: users,
groups, the <code>rwx</code> columns in <code>ls -l</code>, and why some files, like a private SSH key, must be readable by you alone.</p>
]]></content:encoded></item><item><title>Working with Files: Read, Edit, Move and Find (Linux CLI 101, Lesson 2)</title><link>https://networkdoodles.com/lessons/linux-cli-101/working-with-files/</link><pubDate>Mon, 16 Nov 2026 09:00:00 -0500</pubDate><dc:creator>David Henderson</dc:creator><guid>https://networkdoodles.com/lessons/linux-cli-101/working-with-files/</guid><category>linux</category><category>networking-basics</category><category>network-automation</category><category>syslog</category><description>Read logs and configs with less, head and tail -f, edit in nano, copy, move and delete safely, match names with globs, quote spaces, and find files by name.</description><content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[<p>On Linux, nearly everything you care about is a <strong>text file</strong>: logs, configs, inventories, scripts. You read a
short file with <code>cat</code>, a long one with <code>less</code>, the start with <code>head</code>, the end with <code>tail</code>, and a log that&rsquo;s
still growing with <code>tail -f</code>. You change files in an editor (nano is the friendly one), move them with <code>cp</code>,
<code>mv</code> and <code>rm</code>, pick out groups of them with <strong>globs</strong> like <code>*.cfg</code>, and find them anywhere in the tree with
<code>find</code>.</p>
<p>There&rsquo;s no mouse and no recycle bin, so the habits matter as much as the commands. This lesson covers what
each tool is for, how the shell treats names, and the mistakes that cost people files.</p>
<hr>
<h2 id="what-is-the-difference-between-a-text-file-and-a-binary-file">What is the difference between a text file and a binary file?</h2>
<p>A <strong>text file</strong> is bytes that mean characters: letters, digits, spaces and line breaks. You can read it with
any tool, change it in any editor, and compare two versions line by line. A <strong>binary file</strong> is bytes that mean
something only to the program that reads it: a compiled program, an image, a packet capture.</p>
<p>The <code>file</code> command looks inside a file and tells you which it is. In the lab:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">files/logs/r1.log:           ASCII text
</span></span><span class="line"><span class="cl">files/configs/nyc-leaf1.cfg: ASCII text
</span></span><span class="line"><span class="cl">/usr/bin/ls:                 ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, ...
</span></span></code></pre></div><p>The router&rsquo;s log and a device config are plain text. <code>ls</code> itself is a program in <strong>ELF</strong>, Linux&rsquo;s format for
executables. Its first bytes are a marker, not words:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">0000000 177   E   L   F 002 001 001  \0  \0  \0  \0  \0  \0  \0  \0  \0
</span></span></code></pre></div><p>Linux doesn&rsquo;t care about extensions. <code>.cfg</code>, <code>.log</code> and <code>.txt</code> are for humans: <code>file</code> reads the content.
That&rsquo;s why a file with no extension at all, such as <code>/etc/hostname</code>, can still be a perfectly good text file.</p>
<p>Configs, syslog, YAML inventories and scripts are all text, which is why <code>grep</code>, Git and Ansible can work on
them. Print a binary file with <code>cat</code> and the terminal fills with garbage, sometimes enough
to scramble the display. If that happens, type <code>reset</code> and press <kbd>Enter</kbd>.</p>
<hr>
<h2 id="how-do-you-read-a-file-cat-less-head-or-tail">How do you read a file: cat, less, head or tail?</h2>
<p>They all print a file. They differ in how much and how you move around it.</p>
<table>
  <thead>
      <tr>
          <th>Tool</th>
          <th>Shows</th>
          <th>Use it for</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>cat file</code></td>
          <td>the whole file, all at once</td>
          <td>short files: a config, a hosts list</td>
      </tr>
      <tr>
          <td><code>less file</code></td>
          <td>one screen at a time, and you scroll</td>
          <td>anything longer than a screen</td>
      </tr>
      <tr>
          <td><code>head file</code></td>
          <td>the first 10 lines (<code>-n 5</code> for five)</td>
          <td>the start of a log, a CSV&rsquo;s header row</td>
      </tr>
      <tr>
          <td><code>tail file</code></td>
          <td>the last 10 lines</td>
          <td>the most recent log entries</td>
      </tr>
      <tr>
          <td><code>tail -f file</code></td>
          <td>the end, then new lines as they&rsquo;re written</td>
          <td>watching a log live</td>
      </tr>
  </tbody>
</table>
<p>Here&rsquo;s <code>head -n 6</code> on one of the lab&rsquo;s practice configs, a CapeOS <code>show running-config</code> saved to a file:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">Building configuration...
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">Current configuration : 411 bytes
</span></span><span class="line"><span class="cl">!
</span></span><span class="line"><span class="cl">hostname nyc-leaf1
</span></span><span class="line"><span class="cl">!
</span></span></code></pre></div><p>And <code>tail -n 3</code> on ws1&rsquo;s package log, the newest lines last:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">2026-11-11 05:33:18 trigproc ca-certificates:all 20250419+deb13u1 &lt;none&gt;
</span></span><span class="line"><span class="cl">2026-11-11 05:33:18 status half-configured ca-certificates:all 20250419+deb13u1
</span></span><span class="line"><span class="cl">2026-11-11 05:33:19 status installed ca-certificates:all 20250419+deb13u1
</span></span></code></pre></div><p>Log files are written in time order and only ever added to at the end, which is why <code>tail</code> is the first thing
you run on one. <code>wc -l</code> counts lines, so you know whether <code>cat</code> will fit on the screen: the lab&rsquo;s router log
is <code>37 files/logs/r1.log</code>, short enough for either tool.</p>
<h3 id="how-do-you-move-around-in-less">How do you move around in less?</h3>
<p><code>less</code> is a <strong>pager</strong>: a program that shows text one page at a time. <code>man</code> uses one too, so the keys are worth
learning once.</p>
<table>
  <thead>
      <tr>
          <th>Key</th>
          <th>Does</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><kbd>Space</kbd> / <kbd>b</kbd></td>
          <td>a page forward / back</td>
      </tr>
      <tr>
          <td><kbd>↓</kbd> <kbd>↑</kbd></td>
          <td>a line at a time</td>
      </tr>
      <tr>
          <td><kbd>g</kbd> / <kbd>G</kbd></td>
          <td>the start / the end of the file</td>
      </tr>
      <tr>
          <td><code>/text</code> then <kbd>Enter</kbd></td>
          <td>search forward for <em>text</em></td>
      </tr>
      <tr>
          <td><kbd>n</kbd> / <kbd>N</kbd></td>
          <td>the next / previous match</td>
      </tr>
      <tr>
          <td><kbd>F</kbd></td>
          <td>follow the end, like <code>tail -f</code> (<kbd>Ctrl</kbd>+<kbd>C</kbd> to stop)</td>
      </tr>
      <tr>
          <td><kbd>q</kbd></td>
          <td>quit</td>
      </tr>
  </tbody>
</table>
<p>Searching is the reason to use <code>less</code> on a log rather than scrolling: <code>/LINK_DOWN</code> jumps straight to the first
interface that went down. The lab has <code>less</code> 590.</p>
<h3 id="what-does-tail--f-do">What does tail -f do?</h3>
<p><code>tail -f</code> (<strong>follow</strong>) prints the last lines and then waits. Each time something adds a line to the file, it
appears on your screen. It doesn&rsquo;t end on its own: press <kbd>Ctrl</kbd>+<kbd>C</kbd> to stop watching. The
file is untouched.</p>
<p>It&rsquo;s how you watch a problem happen: run <code>tail -f</code> on a log in one terminal, bounce the interface or rerun the
job in another, and see the lines arrive. When a log file gets replaced rather than added to (log rotation
does this), <code>tail -F</code>, with a capital F, keeps following the new file under the same name.</p>
<hr>
<h2 id="which-editor-should-you-use-and-how-do-you-get-out-of-vim">Which editor should you use, and how do you get out of vim?</h2>
<p>You&rsquo;ll edit text files on machines that have no VS Code. There are three editors worth knowing, and you only
need to be good at one.</p>
<table>
  <thead>
      <tr>
          <th>Editor</th>
          <th>Where you&rsquo;ll find it</th>
          <th>Why</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><strong>VS Code</strong></td>
          <td>the codespace; <code>code file</code> opens a file in it</td>
          <td>the easiest: mouse, tabs, colors, undo</td>
      </tr>
      <tr>
          <td><strong>nano</strong></td>
          <td>almost every Linux machine, including ws1 and srv1</td>
          <td>works in any terminal, and shows its keys on screen</td>
      </tr>
      <tr>
          <td><strong>vim</strong> (or <code>vi</code>)</td>
          <td>most servers, often as the default editor</td>
          <td>powerful, but it has <em>modes</em>, which surprise everyone once</td>
      </tr>
  </tbody>
</table>
<p><strong>nano</strong> is the one to learn for remote machines. The two bottom lines list its commands, where <code>^</code> means
<kbd>Ctrl</kbd>: <code>^O</code> writes the file (<kbd>Ctrl</kbd>+<kbd>S</kbd> also saves), <code>^X</code> exits, <code>^W</code> searches,
<code>^K</code> cuts a line and <code>^U</code> pastes it. It asks before throwing away changes. The codespace has GNU nano 7.2.</p>
<p><strong>vim</strong> starts in <em>normal mode</em>, where letters are commands, not text. That&rsquo;s why typing into it seems to do
random things. If you land in vim by accident (a Git commit message, <code>crontab -e</code>, a <code>visudo</code>), this gets you
out:</p>
<table>
  <thead>
      <tr>
          <th>You want to</th>
          <th>Type</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>leave and throw away changes</td>
          <td><kbd>Esc</kbd>, then <code>:q!</code> and <kbd>Enter</kbd></td>
      </tr>
      <tr>
          <td>save and leave</td>
          <td><kbd>Esc</kbd>, then <code>:wq</code> and <kbd>Enter</kbd></td>
      </tr>
  </tbody>
</table>
<p><kbd>Esc</kbd> first, always: it gets you back to normal mode from wherever you are. In the codespace, <code>vi</code> is
<code>vim.tiny</code>, a small build of vim. The slim ws1 container has no <code>vi</code> at all, only nano.</p>
<hr>
<h2 id="how-do-you-copy-move-rename-and-delete-files">How do you copy, move, rename and delete files?</h2>
<p>Four commands, from GNU coreutils (9.1 in the codespace):</p>
<table>
  <thead>
      <tr>
          <th>Command</th>
          <th>Does</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>cp source target</code></td>
          <td>copies; <code>-r</code> to copy a folder and everything in it</td>
      </tr>
      <tr>
          <td><code>mv source target</code></td>
          <td>moves, and renames: a rename is a move within the same folder</td>
      </tr>
      <tr>
          <td><code>rm file</code></td>
          <td>deletes; <code>-r</code> for a folder and everything in it</td>
      </tr>
      <tr>
          <td><code>mkdir folder</code></td>
          <td>makes a folder; <code>-p</code> makes the parents too, and doesn&rsquo;t complain if it exists</td>
      </tr>
  </tbody>
</table>
<p>There&rsquo;s no separate rename command. <code>mv r1.bak r1-before.cfg</code> renames, and <code>mv r2.cfg old/</code> moves the file
into the folder <code>old</code>. The trailing <code>/</code> is a good habit: if <code>old</code> doesn&rsquo;t exist, <code>mv r2.cfg old</code> quietly
renames the file to <code>old</code> instead, and the slash makes <code>mv</code> fail loudly.</p>
<h3 id="why-is-rm-dangerous">Why is rm dangerous?</h3>
<p><strong>There is no recycle bin on the command line.</strong> <code>rm</code> removes the name, and the space is free for reuse
straight away. No undo, no Trash folder: in the codespace, <code>~/.Trash</code> doesn&rsquo;t exist. <code>rm -r</code> takes a whole
tree with it, without asking.</p>
<p><code>cp</code> and <code>mv</code> have a quieter version of the same problem: if the target already exists, they <strong>overwrite it
without asking</strong>. The <code>-i</code> (<strong>interactive</strong>) option makes them check first:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">mv: overwrite &#39;r10.cfg&#39;?
</span></span></code></pre></div><p>Answer <code>n</code> and nothing changes. Habits that help:</p>
<ul>
<li><code>ls</code> a pattern first, and only then <code>rm</code> the same pattern: you see exactly what will go.</li>
<li>Use <code>rm -i</code> when you&rsquo;re unsure, and <code>cp -i</code> and <code>mv -i</code> when the target might exist.</li>
<li>Keep anything you&rsquo;d miss in Git (<span class="link-soon">the GitOps course</span>): then a mistake is one
<code>git restore</code> away.</li>
</ul>
<hr>
<h2 id="what-are-globs-and-who-expands-them">What are globs, and who expands them?</h2>
<p>A <strong>glob</strong> is a pattern for file names. The two you&rsquo;ll use most:</p>
<table>
  <thead>
      <tr>
          <th>Pattern</th>
          <th>Matches</th>
          <th>Example</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>*</code></td>
          <td>any number of characters, including none</td>
          <td><code>*.cfg</code>: every name ending <code>.cfg</code></td>
      </tr>
      <tr>
          <td><code>?</code></td>
          <td>exactly one character</td>
          <td><code>r?.cfg</code>: <code>r1.cfg</code> and <code>r2.cfg</code>, but not <code>r10.cfg</code></td>
      </tr>
      <tr>
          <td><code>[12]</code></td>
          <td>one of the listed characters</td>
          <td><code>r[12].cfg</code>: <code>r1.cfg</code> and <code>r2.cfg</code></td>
      </tr>
  </tbody>
</table>
<p>The important part: <strong>the shell expands the glob, not the command</strong>. Before <code>ls</code> runs, Bash looks in the folder,
swaps the pattern for the matching names, and passes those. You can watch it happen with <code>set -x</code>, which makes
Bash print each command after expansion. In a scratch folder:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">+ ls &#39;nyc spine1.cfg&#39; r1.cfg r10.cfg r2.cfg sw1.cfg
</span></span><span class="line"><span class="cl">+ ls r1.cfg r2.cfg
</span></span></code></pre></div><p>That&rsquo;s <code>ls *.cfg</code> and <code>ls r?.cfg</code> after Bash had finished with them. <code>ls</code> never saw a <code>*</code>. It&rsquo;s why globs work
the same with every command: <code>cp *.cfg backup/</code>, <code>rm *.tmp</code>, <code>mv r*.cfg r1/</code>.</p>
<p>Two consequences:</p>
<ul>
<li><strong>A glob that matches nothing is left as it is.</strong> <code>echo *.yml</code> in a folder with no YAML files prints <code>*.yml</code>,
and the command then complains that the file <code>*.yml</code> doesn&rsquo;t exist.</li>
<li>Globs don&rsquo;t match hidden files: <code>*</code> skips names that start with a dot, the same way <code>ls</code> does.</li>
</ul>
<hr>
<h2 id="how-do-you-handle-file-names-with-spaces">How do you handle file names with spaces?</h2>
<p>The shell splits a command line at spaces, so a name with a space in it turns into two arguments. With a file
called <code>nyc spine1.cfg</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">ls: cannot access &#39;nyc&#39;: No such file or directory
</span></span><span class="line"><span class="cl">ls: cannot access &#39;spine1.cfg&#39;: No such file or directory
</span></span></code></pre></div><p><code>ls</code> went looking for two files that don&rsquo;t exist. Either of these keeps the name in one piece:</p>
<table>
  <thead>
      <tr>
          <th>Write</th>
          <th>How it works</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>&quot;nyc spine1.cfg&quot;</code></td>
          <td>quotes: everything inside is one argument</td>
      </tr>
      <tr>
          <td><code>nyc\ spine1.cfg</code></td>
          <td>a backslash: the next character is taken literally</td>
      </tr>
  </tbody>
</table>
<p>Tab completion writes the backslash for you, which is one more reason to let it finish names. Single quotes
work too, and are stricter: inside <code>'...'</code>, nothing is special at all. Better still, don&rsquo;t create names with
spaces: use <code>-</code> or <code>_</code>. Other people&rsquo;s files will still arrive with them.</p>
<hr>
<h2 id="how-do-you-find-a-file-when-you-dont-know-where-it-is">How do you find a file when you don&rsquo;t know where it is?</h2>
<p><code>find</code> walks down a folder tree and prints every path that matches the tests you give it. The shape is always
<code>find where tests</code>. The lab has GNU findutils 4.9.0.</p>
<table>
  <thead>
      <tr>
          <th>Test</th>
          <th>Matches</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>-name '*.cfg'</code></td>
          <td>the name matches the glob (<code>-iname</code> ignores case)</td>
      </tr>
      <tr>
          <td><code>-type f</code> / <code>-type d</code></td>
          <td>files only / folders only</td>
      </tr>
      <tr>
          <td><code>-mmin -60</code></td>
          <td>changed in the last 60 minutes</td>
      </tr>
      <tr>
          <td><code>-mtime -1</code></td>
          <td>changed in the last day</td>
      </tr>
      <tr>
          <td><code>-mtime +1</code></td>
          <td>changed longer ago (find counts whole days, so <code>+1</code> means at least two days)</td>
      </tr>
      <tr>
          <td><code>-empty</code></td>
          <td>empty files and folders</td>
      </tr>
      <tr>
          <td><code>-size +1M</code></td>
          <td>bigger than a mebibyte</td>
      </tr>
  </tbody>
</table>
<p>Several tests together must all match. In the scratch folder, after a few files were moved around:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ find . -name &#39;*.cfg&#39;
</span></span><span class="line"><span class="cl">./r1.cfg
</span></span><span class="line"><span class="cl">./r10.cfg
</span></span><span class="line"><span class="cl">./sw1.cfg
</span></span><span class="line"><span class="cl">./nyc spine1.cfg
</span></span><span class="line"><span class="cl">./old/r2.cfg
</span></span><span class="line"><span class="cl">./r1-before.cfg
</span></span><span class="line"><span class="cl">$ find . -type d
</span></span><span class="line"><span class="cl">.
</span></span><span class="line"><span class="cl">./old
</span></span><span class="line"><span class="cl">$ find . -mtime +1
</span></span><span class="line"><span class="cl">./old/r2.cfg
</span></span></code></pre></div><p><code>find</code> looks inside subfolders, which <code>ls *.cfg</code> doesn&rsquo;t: it found <code>old/r2.cfg</code>, the one file old enough to
match <code>-mtime +1</code>. Unlike <code>ls</code>, its output isn&rsquo;t sorted. It prints paths in the order it walks the tree.</p>
<p>Notice the quotes around <code>'*.cfg'</code>. Here the glob is for <code>find</code>, not the shell, so it must reach <code>find</code>
unexpanded. Leave them off in a folder that has <code>.cfg</code> files, and the shell expands the pattern first:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">find: paths must precede expression: `r1-before.cfg&#39;
</span></span><span class="line"><span class="cl">find: possible unquoted pattern after predicate `-name&#39;?
</span></span></code></pre></div><p>Searching by age is the one network engineers reach for most: <em>which configs changed since yesterday?</em> is
<code>find</code> with <code>-name</code> and <code>-mtime -1</code>. Searching <em>inside</em> files, for a line rather than a name, is <code>grep</code>, in
<a href="/lessons/linux-cli-101/pipes-redirection-grep/">lesson 3</a>.</p>
<hr>
<h2 id="what-i-see-go-wrong">What I see go wrong</h2>
<p><strong><code>rm</code> with a typo in the pattern.</strong> <code>rm * .cfg</code>, with a stray space, is two arguments: <code>*</code>, everything, and
<code>.cfg</code>. <code>ls</code> the pattern first.</p>
<p><strong>A copy that overwrote the original.</strong> <code>cp new.cfg old.cfg</code> replaces <code>old.cfg</code> silently. <code>-i</code>, or Git.</p>
<p><strong><code>mv</code> into a folder that wasn&rsquo;t there.</strong> The file is now called the folder&rsquo;s name. Write the trailing <code>/</code>.</p>
<p><strong>Stuck in vim.</strong> <kbd>Esc</kbd>, <code>:q!</code>, <kbd>Enter</kbd>.</p>
<p><strong><code>tail -f</code> that never ends.</strong> It&rsquo;s not broken; it&rsquo;s waiting for new lines. <kbd>Ctrl</kbd>+<kbd>C</kbd>.</p>
<p><strong>Unquoted names and patterns.</strong> A name with a space splits in two; a <code>find -name</code> pattern without quotes is
expanded by the shell. Quote both.</p>
<p><strong>Editing a config in Windows and copying it over.</strong> Windows ends lines with two characters, Linux with one.
<code>file</code> reports <em>with CRLF line terminators</em>, and some tools choke on the extra character. VS Code&rsquo;s status bar
shows <code>CRLF</code> or <code>LF</code>, and one click changes it.</p>
<hr>
<h2 id="try-it">Try it</h2>
<p><a href="/labs/linux-cli-101/task-2-1/">Chapter 2 of the lab</a> gives you a real CapeOS syslog from the lab&rsquo;s router.
You read it with <code>less</code> and its search, pick out the first and last events with <code>head</code> and <code>tail</code>, watch new
lines arrive with <code>tail -f</code>, and write a note in nano. The challenge is a messy folder of config backups,
saved by hand: you sort it into folders, fix a name with a space in it, clear out the empty files, and use
<code>find</code> to count what&rsquo;s there. The same skills come back in <a href="/lessons/python-101/files-and-errors/">Python 101&rsquo;s lesson on
files</a>, where a script does the reading.</p>
<h2 id="whats-next">What&rsquo;s next</h2>
<p><a href="/lessons/linux-cli-101/pipes-redirection-grep/">Lesson 3</a> connects commands together: send one command&rsquo;s
output into the next, save it to a file, and use <code>grep</code> to answer a question about a log in one line.</p>
]]></content:encoded></item></channel></rss>