<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/"><channel><title>Networking-Basics · Network Doodles</title><link>https://networkdoodles.com/tags/networking-basics/</link><description>Network automation, from first playbook to agentic operations. Ansible, NetBox, MCP, Python and the occasional doodle.</description><language>en-us</language><managingEditor>David Henderson</managingEditor><lastBuildDate>Mon, 16 Nov 2026 09:00:00 -0500</lastBuildDate><atom:link href="https://networkdoodles.com/tags/networking-basics/index.xml" rel="self" type="application/rss+xml"/><item><title>Troubleshooting a Broken Path with Wireshark: Loss, Silent Drops and MTU (Wireshark 101, Lesson 8)</title><link>https://networkdoodles.com/lessons/wireshark-101/broken-path/</link><pubDate>Mon, 16 Nov 2026 09:00:00 -0500</pubDate><dc:creator>David Henderson</dc:creator><guid>https://networkdoodles.com/lessons/wireshark-101/broken-path/</guid><category>wireshark</category><category>packet-capture</category><category>networking-basics</category><category>troubleshooting</category><category>tcp</category><description>Find where a path breaks: capture at several points, follow one packet, read loss and retransmissions, silent ACL drops, missing routes and MTU holes.</description><content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[<p>When a connection fails, the capture on the client tells you <em>that</em> something went wrong, rarely <em>where</em>. The
method that finds where is simple: capture at two or more points on the path at the same time, and follow one
packet from point to point. The fault is between the last place you see it and the first place you don&rsquo;t. Then
the shape of what you see tells you what kind of fault it is: a burst of duplicate ACKs is loss, a SYN that
vanishes is a filter, a SYN-ACK that never comes back is a route, and a connection that opens and then stalls
is very often the MTU.</p>
<p>This lesson pulls the whole course together on four faults from the
<a href="/labs/wireshark-101/">Wireshark 101 lab</a>. Each one looks different on the wire, and two of them look the same
from the laptop.</p>
<hr>
<h2 id="silence-versus-refusal">Silence versus refusal</h2>
<p><a href="/lessons/wireshark-101/tcp/">Lesson 6</a> drew the line that matters most here. A SYN answered by a RST is a
<strong>refusal</strong>: the network works and the far end said no. A SYN answered by nothing is <strong>silence</strong>: something on
the way dropped it, or dropped the answer, and nobody said a word.</p>
<p>A refusal is easy to troubleshoot, because the answer came from somewhere. Silence is hard, because the client
sees exactly the same thing whatever the cause: the same SYN, sent again and again. In the lab&rsquo;s mystery fault,
the laptop&rsquo;s capture is seven copies of one SYN over seven seconds, then nothing. Three different faults
produce that capture. You can&rsquo;t tell them apart from the laptop, however long you stare.</p>
<p>So you stop staring and add capture points.</p>
<hr>
<h2 id="capture-at-two-points-and-follow-one-packet">Capture at two points and follow one packet</h2>
<p>The lab&rsquo;s path from the laptop to the web server crosses a switch and two routers:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">laptop ──[laptop:eth1]── sw1 ── r1 ──[r1:eth2]── r2 ──[r2:eth2]── web
</span></span></code></pre></div><p>Capture all three marked points into one file, at the same time, and filter on one connection. A SYN that
appears at <code>laptop:eth1</code> and <code>r1:eth2</code>, but never at <code>r2:eth2</code>, got through r1 and died at r2. That&rsquo;s the
whole method: divide the path, and look for the boundary between seen and not seen. Wireshark&rsquo;s
<code>frame.interface_name</code> field records which point saw each frame.</p>
<p>Two rules make it work:</p>
<ul>
<li><strong>Capture at the same time.</strong> Faults come and go. Two captures taken ten minutes apart don&rsquo;t describe the
same packet.</li>
<li><strong>Take away what you&rsquo;re not testing.</strong> The lab&rsquo;s mystery traffic opens the web server by its IP address, not
its name, so a DNS problem can&rsquo;t hide the fault you&rsquo;re looking for.</li>
</ul>
<h3 id="the-multi-interface-trap">The multi-interface trap</h3>
<p>Put one packet in a file three times and Wireshark&rsquo;s TCP analysis gets confused. It doesn&rsquo;t know the copies are
the same packet seen in three places. It thinks the second and third copies were sent again:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">1  0.000000000  laptop:eth1  51606 → 80 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 ...
</span></span><span class="line"><span class="cl">2  0.000217167  r1:eth2      [TCP Retransmission] 51606 → 80 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 ...
</span></span></code></pre></div><p>Frame 2 isn&rsquo;t a retransmission. It&rsquo;s frame 1, 0.2 ms later and one router further on. Every <code>tcp.analysis</code>
flag in a multi-point capture is suspect until you deal with this.</p>
<p>The obvious fix, a display filter like <code>frame.interface_name == &quot;laptop:eth1&quot;</code>, isn&rsquo;t enough. A display filter
hides frames; it doesn&rsquo;t redo the analysis. In the lab&rsquo;s MTU capture, the SYN-ACK reached the switch&rsquo;s capture
point first, so the laptop&rsquo;s own copy is still flagged <em>TCP Retransmission</em> with everything else filtered out.
Two fixes that do work, both checked against the lab&rsquo;s captures with Wireshark 4.6:</p>
<ul>
<li><strong>Export one interface to its own file</strong> (File › Export Specified Packets, with the interface filter applied)
and open that. The analysis starts over, and the false flags are gone.</li>
<li><strong>Tell Wireshark the copies are different conversations.</strong> The preference
<code>protocols.conversation_deinterlacing_key</code> (find it under Edit › Preferences › Advanced) set to <code>.MI</code> keeps
frames from different interfaces and MACs in separate TCP streams. Each capture point becomes its own stream,
and only real retransmissions are flagged.</li>
</ul>
<hr>
<h2 id="packet-loss-duplicate-acks-and-fast-retransmission">Packet loss: duplicate ACKs and fast retransmission</h2>
<p>A link that loses a few percent of its packets doesn&rsquo;t look broken. Pages load, downloads finish, ping mostly
answers. Everything is just slower. A capture shows it plainly, because TCP has to do something about every lost
packet, and it does it on the wire.</p>
<p>In the lab, r2 drops 3% of what it sends towards r1, and the laptop downloads a 2 MB file. Here&rsquo;s one lost
segment and its repair:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">35 0.578603209 192.168.2.10 → 192.168.1.175 [TCP Previous segment not captured] 80 → 56336 [ACK] Seq=17623 ... Len=1448
</span></span><span class="line"><span class="cl">36 0.578605084 192.168.1.175 → 192.168.2.10 [TCP Dup ACK 34#1] 56336 → 80 [ACK] Seq=79 Ack=16175 ... SLE=17623 SRE=19071
</span></span><span class="line"><span class="cl">38 0.578606375 192.168.1.175 → 192.168.2.10 [TCP Dup ACK 34#2] 56336 → 80 [ACK] Seq=79 Ack=16175 ... SLE=17623 SRE=20519
</span></span><span class="line"><span class="cl">40 0.578607334 192.168.1.175 → 192.168.2.10 [TCP Dup ACK 34#3] 56336 → 80 [ACK] Seq=79 Ack=16175 ... SLE=17623 SRE=21967
</span></span><span class="line"><span class="cl">...
</span></span><span class="line"><span class="cl">58 0.578705625 192.168.2.10 → 192.168.1.175 [TCP Fast Retransmission] 80 → 56336 [ACK] Seq=16175 ... Len=1448
</span></span></code></pre></div><p>Read it with the sequence numbers from lesson 6:</p>
<ol>
<li><strong>The gap.</strong> Frame 35 starts at byte 17,623, but the laptop was expecting 16,175. One 1,448-byte segment is
missing. Wireshark marks it <em>Previous segment not captured</em>.</li>
<li><strong>Duplicate ACKs.</strong> The laptop can only acknowledge bytes up to the gap, so every segment that arrives after it
triggers the same ACK again: <code>Ack=16175</code>, &ldquo;I still need byte 16,175&rdquo;.</li>
<li><strong>SACK blocks.</strong> Each duplicate ACK also carries a <strong>SACK</strong> block (<a href="https://www.rfc-editor.org/rfc/rfc2018">RFC 2018</a>):
<code>SLE</code> and <code>SRE</code> (the left and right edges) list the bytes the laptop <em>did</em> get past the gap. The right edge
grows with every segment. The sender learns exactly what&rsquo;s missing and resends only that.</li>
<li><strong>Fast retransmission.</strong> The sender doesn&rsquo;t wait for a timeout. Duplicate ACKs are a strong hint that one
segment was lost and the ones after it arrived, so it resends the missing one straight away.
<a href="https://www.rfc-editor.org/rfc/rfc5681">RFC 5681</a> sets the trigger at three duplicate ACKs.</li>
</ol>
<p>In the lab&rsquo;s reference capture, 3% loss on one direction of one link produced 26 retransmissions (25 of them
fast) and 142 duplicate ACKs, plus 39 gaps and 18 segments marked out of order. The receiver is noisy about loss;
the sender is quiet and efficient.</p>
<h3 id="wiresharks-labels-are-heuristics">Wireshark&rsquo;s labels are heuristics</h3>
<p>Those <code>tcp.analysis</code> labels are Wireshark&rsquo;s best guess from the sequence numbers, not something TCP wrote in
the packet. <a href="https://www.wireshark.org/docs/wsug_html_chunked/ChAdvTCPAnalysis.html">Wireshark&rsquo;s TCP analysis documentation</a>
gives the rules, and they don&rsquo;t always match the RFCs. It calls a segment a <em>fast retransmission</em> after two
duplicate ACKs, not three. And it calls a filled gap <em>out-of-order</em> instead of a retransmission when it arrives
very soon after the previous segment, within the initial RTT or 3 ms. The lab&rsquo;s link loses packets but never reorders them, so
those 18 &ldquo;out-of-order&rdquo; segments are really repairs too.</p>
<table>
  <thead>
      <tr>
          <th>Flag</th>
          <th>Wireshark&rsquo;s meaning</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>tcp.analysis.lost_segment</code></td>
          <td>a gap: the sequence number jumped past what was expected</td>
      </tr>
      <tr>
          <td><code>tcp.analysis.duplicate_ack</code></td>
          <td>an ACK that repeats the previous one, with no data and the same window</td>
      </tr>
      <tr>
          <td><code>tcp.analysis.retransmission</code></td>
          <td>data the sender had already sent (fast retransmissions included)</td>
      </tr>
      <tr>
          <td><code>tcp.analysis.fast_retransmission</code></td>
          <td>a retransmission that followed duplicate ACKs</td>
      </tr>
      <tr>
          <td><code>tcp.analysis.out_of_order</code></td>
          <td>a filled gap that arrived quickly after the segment before it</td>
      </tr>
  </tbody>
</table>
<p><code>tcp.analysis.flags</code> matches them all, and <strong>Analyze › Expert Information</strong> counts them by type. That&rsquo;s the
fastest summary of a capture&rsquo;s health. Just remember the multi-interface trap above before trusting the counts.</p>
<p>To find <em>where</em> the loss happens, use the method: a segment on <code>r2:eth2</code> that never appears on <code>r1:eth2</code> was lost
between them.</p>
<hr>
<h2 id="acl-drops-silent-or-icmp-administratively-prohibited">ACL drops: silent, or ICMP administratively prohibited</h2>
<p>An <strong>access list</strong> (ACL) is a router&rsquo;s list of permit and deny rules. When it denies a packet, the router has two
choices: drop it silently, or tell the sender with an ICMP <strong>Destination Unreachable</strong> message, type 3,
code 13, <em>communication administratively prohibited</em> (<a href="https://www.rfc-editor.org/rfc/rfc1812">RFC 1812</a>).</p>
<p>The two make very different captures:</p>
<table>
  <thead>
      <tr>
          <th>Router behavior</th>
          <th>What the client sees</th>
          <th>What you can learn</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>sends ICMP type 3 code 13</td>
          <td>the SYN, then an ICMP message from the router that denied it</td>
          <td>which router, straight away: it&rsquo;s the source of the ICMP</td>
      </tr>
      <tr>
          <td>drops silently</td>
          <td>the SYN, again and again, then a timeout</td>
          <td>nothing, until you capture further along</td>
      </tr>
  </tbody>
</table>
<p>The routers in the lab run CapeOS 0.11.2, which answers an inbound ACL deny with ICMP administratively
prohibited unless the interface has <code>no ip unreachables</code>. The mystery fault sets it, so the drop is silent: the
harder case, and the one you&rsquo;ll meet most often, because many networks turn these messages off.</p>
<p>In the lab&rsquo;s reference capture the fault was on r2. The SYN shows up at <code>laptop:eth1</code> and at <code>r1:eth2</code>, seven
times each, and never once at <code>r2:eth2</code>. It crossed r1, arrived at r2, and went no further. No ICMP anywhere.</p>
<hr>
<h2 id="a-missing-return-route">A missing return route</h2>
<p>The third mystery fault looks the same from the laptop, and it&rsquo;s completely different underneath. r2 loses its
route back to the home network, 192.168.1.0/24.</p>
<p>Follow the SYN and it gets all the way to web. web answers with a SYN-ACK. The SYN-ACK reaches r2, which has no
route for 192.168.1.175, and dies there. The captures show it:</p>
<ul>
<li>the SYN at all three points;</li>
<li>a SYN-ACK at <code>r2:eth2</code> only, never at <code>r1:eth2</code> or the laptop;</li>
<li>an ICMP <em>Network unreachable</em> (type 3, code 0) from r2&rsquo;s address on that link, 192.168.2.1, <strong>to web</strong>.</li>
</ul>
<p>That last one catches people out. A router that can&rsquo;t forward a packet tells the packet&rsquo;s <em>source</em>, and the
packet it couldn&rsquo;t forward was web&rsquo;s SYN-ACK. So the ICMP message goes to web, not to the laptop. The laptop never
hears a thing. You only see that message if you capture on web&rsquo;s side of r2.</p>
<p>The general lesson: a path has two directions, and a fault in the return direction looks like a fault in the
forward direction from the client. If the SYN arrives and the answer doesn&rsquo;t come back, look at routes <em>back</em>.</p>
<hr>
<h2 id="mtu-when-small-packets-work-and-big-ones-dont">MTU: when small packets work and big ones don&rsquo;t</h2>
<p>The <strong>MTU</strong> (maximum transmission unit) is the biggest IP packet a link carries in one frame. On ordinary
Ethernet it&rsquo;s 1,500 bytes, a 1,514-byte frame with the Ethernet header. Some networks use <strong>jumbo frames</strong>, often
9,000 bytes, to move data in fewer, bigger packets.</p>
<p>At the start of a connection, each side announces its <strong>MSS</strong> (maximum segment size), worked out from its own
MTU: 1,460 for 1,500, 8,960 for 9,000. The other side then sends segments up to that size. Neither side knows
anything about the links in between.</p>
<p>In the lab, someone turned on jumbo frames on the laptop and on web, but not on the switch between the laptop
and r1. Both hosts announced <code>MSS=8960</code>. The handshake is small, so it crossed fine. Then:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">11  laptop:eth1    74  57982 → 443 [SYN] Seq=0 Win=62720 Len=0 MSS=8960 ...
</span></span><span class="line"><span class="cl">13  sw1:eth1       74  443 → 57982 [SYN, ACK] Seq=0 Ack=1 ... MSS=8960 ...
</span></span><span class="line"><span class="cl">15  laptop:eth1    66  57982 → 443 [ACK] Seq=1 Ack=1 ...
</span></span><span class="line"><span class="cl">17  laptop:eth1  1624  Client Hello (SNI=web.lab)
</span></span><span class="line"><span class="cl">18  laptop:eth1  1624  [TCP Retransmission] 57982 → 443 [PSH, ACK] Seq=1 Ack=1 ... Len=1558
</span></span><span class="line"><span class="cl">19  laptop:eth1  1624  [TCP Retransmission] 57982 → 443 [PSH, ACK] Seq=1 Ack=1 ... Len=1558
</span></span><span class="line"><span class="cl">...
</span></span></code></pre></div><p>The third column is the frame length. The TLS 1.3 Client Hello from <a href="/lessons/wireshark-101/tls-without-decryption/">lesson 7</a>,
1,558 bytes, now went as <strong>one</strong> 1,624-byte frame, because the laptop believed it had 8,960 bytes to play with.
It left the laptop 16 times (the original and 15 retransmissions), about every half second, and never once appeared on <code>sw1:eth1</code>, the switch&rsquo;s port
towards r1. Filter <code>frame.len &gt; 1514</code> and the picture is complete in two lines: big frames on one side of the
switch, none on the other.</p>
<h3 id="why-a-switch-makes-a-black-hole">Why a switch makes a black hole</h3>
<p>A router that receives a packet too big for the next link, with the IP header&rsquo;s <strong>DF</strong> (don&rsquo;t fragment) bit set,
drops it and sends back ICMP type 3, code 4, <em>fragmentation needed</em>, with the next link&rsquo;s MTU in the message.
That&rsquo;s <strong>Path MTU Discovery</strong> (<a href="https://www.rfc-editor.org/rfc/rfc1191">RFC 1191</a>): the sender learns the
smaller size and sends smaller segments. TCP on Linux sets DF on its packets for exactly this reason. The lab&rsquo;s
1,624-byte frame had it set.</p>
<p>A <strong>switch</strong> can&rsquo;t do that. It works at layer 2, forwarding frames by MAC address, and has no IP address on the
path to send ICMP from. A frame bigger than its port&rsquo;s MTU is simply dropped. Nobody is told, so the sender never
learns, and it keeps retransmitting into the hole until the application gives up. That&rsquo;s a <strong>black hole</strong>:
packets go in and nothing comes out.</p>
<p>Routers can make black holes too. If the <em>fragmentation needed</em> message is filtered on the way back, often by a
firewall that blocks &ldquo;all ICMP&rdquo;, PMTUD breaks in the same silent way.
<a href="https://www.rfc-editor.org/rfc/rfc2923">RFC 2923</a> describes the problem.</p>
<h3 id="why-tls-13-breaks-first">Why TLS 1.3 breaks first</h3>
<p>An MTU problem only hurts packets bigger than the smallest link. Ping is 98 bytes. A DNS lookup is under 100.
A TCP handshake is 74. A plain HTTP request is often under 200. On the lab&rsquo;s broken path, all of those work. The
first big packet in most HTTPS connections is now the client&rsquo;s TLS 1.3 Client Hello, and with a post-quantum key
share it&rsquo;s over 1,500 bytes on its own. So the classic symptom is: <strong>ping works, HTTP works, HTTPS hangs right
after the handshake.</strong> It used to take a large download to trip over an MTU mismatch. Now a single Client Hello
does it.</p>
<hr>
<h2 id="what-i-see-go-wrong">What I see go wrong</h2>
<p><strong>Trusting the analysis in a multi-point capture.</strong> Half the <em>Retransmission</em> and <em>Dup ACK</em> labels in a
three-point capture are the same packet seen three times. Export one interface, or set the deinterlacing
preference, before you count anything.</p>
<p><strong>Capturing at one point and guessing.</strong> The client&rsquo;s capture says &ldquo;no answer&rdquo;. It can&rsquo;t say where. Two capture
points, at the same time, usually settle it in minutes.</p>
<p><strong>Forgetting the return path.</strong> A SYN that arrives and an answer that doesn&rsquo;t come back is a return-route or
return-filter problem. The ICMP message that would explain it goes to the server, not to you.</p>
<p><strong>&ldquo;Ping works, so the network&rsquo;s fine.&rdquo;</strong> Ping is small. Test with the size that fails. When the application
hangs after the handshake, filter <code>frame.len &gt; 1514</code> and look for large frames sent over and over and never
acknowledged.</p>
<p><strong>Mixed MTUs.</strong> Jumbo frames on the hosts and not on the switches, or the other way around, is a black hole
waiting for its first big packet. Make every device on a layer 2 network agree on the MTU.</p>
<hr>
<h2 id="try-it">Try it</h2>
<p><a href="/labs/wireshark-101/#chapter-8">Chapter 8 of the lab</a> breaks the path three ways: packet loss you measure, a
fault the lab picks at random that you find with three capture points, and the jumbo-frame black hole.</p>
<h2 id="where-to-go-next">Where to go next</h2>
<p>That&rsquo;s the end of Wireshark 101. Captures are for one conversation, on demand. For watching a whole network all
the time, logs and metrics do the job, and the <span class="link-soon">observability course</span> builds them. When
those say something is wrong and can&rsquo;t say why, come back to the packets. Or start again at the
<a href="/lessons/wireshark-101/dns/">DNS lesson</a> with a capture from your own network.</p>
]]></content:encoded></item><item><title>The Network from the Command Line: ip route get, ss, curl -v, dig and tcpdump (Linux CLI 101, Lesson 8)</title><link>https://networkdoodles.com/lessons/linux-cli-101/network-commands/</link><pubDate>Mon, 16 Nov 2026 09:00:00 -0500</pubDate><dc:creator>David Henderson</dc:creator><guid>https://networkdoodles.com/lessons/linux-cli-101/network-commands/</guid><category>linux</category><category>networking-basics</category><category>troubleshooting</category><category>packet-capture</category><category>tcp</category><description>Troubleshoot from a Linux host: ip route get, dig, traceroute, what ss says about listening addresses, curl -v from DNS to HTTP, and packets with tcpdump.</description><content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[<p>When a web service works on the server but not from another host, work through the path in order. Does the
name resolve to the right address (<code>dig</code>)? Which way will the packet go (<code>ip route get</code>)? Is anything on the
way answering (<code>ping</code>, <code>traceroute</code>)? Is the service listening <strong>on an address the other host can reach</strong>
(<code>ss -tlnp</code>)? Then let <code>curl -v</code> show where the conversation stops, and <code>tcpdump</code> show what&rsquo;s on the wire.</p>
<p>A common answer to &ldquo;works locally, not remotely&rdquo; is the listening address. A program listening on
<code>127.0.0.1</code> only accepts connections from its own machine. <code>0.0.0.0</code> means every address the machine has.</p>
<p>You met most of these commands in the <a href="/labs/networking-101-lab/task-3-3/">Networking 101 lab</a>. This lesson is
about the options and output that matter when you&rsquo;re troubleshooting. Everything below ran on the lab&rsquo;s <code>ws1</code>
(Debian 13: iproute2 6.15.0, iputils 20240905, traceroute 2.1.6, curl 8.14.1, DiG 9.20.30, tcpdump 4.99.5),
except the TLS example, which ran in the codespace.</p>
<hr>
<h2 id="how-do-i-see-a-hosts-addresses-at-a-glance">How do I see a host&rsquo;s addresses at a glance?</h2>
<p><code>ip addr</code> prints a screenful per interface. <code>-br</code> (<em>brief</em>) prints one line each, and <code>-4</code> drops IPv6:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ ip -br -4 addr
</span></span><span class="line"><span class="cl">lo               UNKNOWN        127.0.0.1/8
</span></span><span class="line"><span class="cl">eth0@if76        UP             172.80.102.11/24
</span></span><span class="line"><span class="cl">eth1@if80        UP             192.168.10.10/24
</span></span></code></pre></div><p>Name, state, addresses. On ws1, <code>eth0</code> is the lab&rsquo;s management network (the way the codespace reaches it), and
<code>eth1</code> is its cable to the router, r1. <code>lo</code> is the <strong>loopback</strong> interface: a virtual interface that only ever
talks to the machine itself, with <code>127.0.0.1</code> on it. The <code>@if76</code> part says the interface is one end of a
virtual cable whose other end is interface 76 somewhere else, which is how containers are wired. Without
<code>-4</code>, <code>ip -br addr</code> also lists half a dozen tunnel interfaces that are <code>DOWN</code> and can be ignored.</p>
<hr>
<h2 id="which-way-will-a-packet-go">Which way will a packet go?</h2>
<p><code>ip route</code> shows the routing table. <code>ip route get</code> answers the question you actually have: <em>for this one
address, which route wins?</em> It asks the kernel&rsquo;s routing decision directly and sends nothing
(<a href="https://man7.org/linux/man-pages/man8/ip-route.8.html">ip-route(8)</a>).</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ ip route get 172.80.102.12
</span></span><span class="line"><span class="cl">172.80.102.12 dev eth0 src 172.80.102.11 uid 1000
</span></span><span class="line"><span class="cl">    cache
</span></span><span class="line"><span class="cl">$ ip route get 8.8.8.8
</span></span><span class="line"><span class="cl">8.8.8.8 via 192.168.10.1 dev eth1 src 192.168.10.10 uid 1000
</span></span><span class="line"><span class="cl">    cache
</span></span></code></pre></div><table>
  <thead>
      <tr>
          <th>Word</th>
          <th>Means</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>via 192.168.10.1</code></td>
          <td>send it to this router. No <code>via</code> means the address is on a directly connected network</td>
      </tr>
      <tr>
          <td><code>dev eth1</code></td>
          <td>out of this interface</td>
      </tr>
      <tr>
          <td><code>src 192.168.10.10</code></td>
          <td>with this as the source address</td>
      </tr>
      <tr>
          <td><code>uid 1000</code></td>
          <td>the user who asked. Ignore it</td>
      </tr>
  </tbody>
</table>
<p>The first address is on the management network, so it goes straight out of <code>eth0</code>. The second matches only the
default route, so it goes to r1. On a host with several interfaces, this is the quickest way to find out
whether replies leave by the interface you expect. The <code>src</code> field matters too: it&rsquo;s the address the far end
sees, and the address its firewall rules have to allow.</p>
<hr>
<h2 id="which-ping-and-traceroute-options-matter">Which ping and traceroute options matter?</h2>
<p>Linux <code>ping</code> runs until you press Ctrl-C. In anything you&rsquo;ll script or paste, give it a count:</p>
<table>
  <thead>
      <tr>
          <th>Option</th>
          <th>Does</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>ping -c 3</code></td>
          <td>send 3 and stop, with a summary</td>
      </tr>
      <tr>
          <td><code>ping -W 1</code></td>
          <td>wait at most 1 second for each reply</td>
      </tr>
      <tr>
          <td><code>ping -n</code></td>
          <td>numbers only, no reverse lookups</td>
      </tr>
      <tr>
          <td><code>traceroute -n</code></td>
          <td>numbers only, no names</td>
      </tr>
      <tr>
          <td><code>traceroute -q 1</code></td>
          <td>one probe per hop instead of three</td>
      </tr>
      <tr>
          <td><code>traceroute -w 1</code></td>
          <td>wait at most 1 second per probe</td>
      </tr>
      <tr>
          <td><code>traceroute -m 5</code></td>
          <td>give up after 5 hops instead of 30</td>
      </tr>
  </tbody>
</table>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ ping -c 3 -n srv1.lab
</span></span><span class="line"><span class="cl">PING srv1.lab (192.168.20.10) 56(84) bytes of data.
</span></span><span class="line"><span class="cl">64 bytes from 192.168.20.10: icmp_seq=1 ttl=63 time=0.074 ms
</span></span><span class="line"><span class="cl">64 bytes from 192.168.20.10: icmp_seq=2 ttl=63 time=0.252 ms
</span></span><span class="line"><span class="cl">64 bytes from 192.168.20.10: icmp_seq=3 ttl=63 time=0.075 ms
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">--- srv1.lab ping statistics ---
</span></span><span class="line"><span class="cl">3 packets transmitted, 3 received, 0% packet loss, time 2041ms
</span></span><span class="line"><span class="cl">rtt min/avg/max/mdev = 0.074/0.133/0.252/0.083 ms
</span></span></code></pre></div><p><code>ttl=63</code> is a free hop count. Linux sends with a TTL of 64, and each router on the way takes one off, so 63
means one router in between. The exit code also matters here: <code>ping</code> exits 0 if replies came back and 1 if none
did, which is what <a href="/lessons/linux-cli-101/bash-scripting-basics/">lesson 7</a>&rsquo;s scripts read.</p>
<p>Without <code>-n</code>, traceroute looks up a name for every hop. In the lab that&rsquo;s quick, because srv1&rsquo;s DNS server
knows the names. On a real network, a reverse lookup that nobody answers can add seconds per hop:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ traceroute 192.168.20.10
</span></span><span class="line"><span class="cl">traceroute to 192.168.20.10 (192.168.20.10), 30 hops max, 60 byte packets
</span></span><span class="line"><span class="cl"> 1  r1.lab (192.168.10.1)  1.073 ms  0.124 ms  0.057 ms
</span></span><span class="line"><span class="cl"> 2  srv1.lab (192.168.20.10)  4.811 ms  0.082 ms  0.045 ms
</span></span></code></pre></div><p>Stars don&rsquo;t always mean a fault. Running <code>traceroute -n</code> straight after that one gave this:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">traceroute to 192.168.20.10 (192.168.20.10), 30 hops max, 60 byte packets
</span></span><span class="line"><span class="cl"> 1  192.168.10.1  0.799 ms  0.035 ms  0.031 ms
</span></span><span class="line"><span class="cl"> 2  * * *
</span></span><span class="line"><span class="cl"> 3  * * *
</span></span><span class="line"><span class="cl"> 4  * * *
</span></span><span class="line"><span class="cl"> 5  * * *
</span></span><span class="line"><span class="cl"> 6  * * *
</span></span><span class="line"><span class="cl"> 7  * 192.168.20.10  0.461 ms  0.107 ms
</span></span></code></pre></div><p>Same two-hop path, a few seconds later, run again: back to two lines. Linux traceroute sends many probes at
once, and Linux limits how fast it sends the ICMP errors traceroute depends on (<code>net.ipv4.icmp_ratelimit</code>,
1000 ms on these hosts), so a burst of probes can get only some answers. If a count looks wrong, wait a moment
and run it again with <code>-q 1</code>.</p>
<hr>
<h2 id="what-does-a-listening-address-mean">What does a listening address mean?</h2>
<p>A server program <strong>listens</strong> on a port: it asks the operating system for every new connection to that port. It
also picks <strong>which of the machine&rsquo;s addresses</strong> to listen on. <code>ss -tlnp</code> shows both: TCP (<code>t</code>), listening
(<code>l</code>), numbers (<code>n</code>), and the process (<code>p</code>) that owns each socket
(<a href="https://man7.org/linux/man-pages/man8/ss.8.html">ss(8)</a>).</p>
<p>To see it, I started Python&rsquo;s built-in web server on ws1, listening only on loopback:
<code>python3 -m http.server --bind 127.0.0.1 9999</code>.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ ss -tln
</span></span><span class="line"><span class="cl">State  Recv-Q Send-Q Local Address:Port  Peer Address:Port
</span></span><span class="line"><span class="cl">LISTEN 0      5          127.0.0.1:9999       0.0.0.0:*
</span></span><span class="line"><span class="cl">LISTEN 0      128          0.0.0.0:22         0.0.0.0:*
</span></span><span class="line"><span class="cl">LISTEN 0      4096      127.0.0.11:35819      0.0.0.0:*
</span></span><span class="line"><span class="cl">LISTEN 0      128             [::]:22            [::]:*
</span></span></code></pre></div><p>Two programs, two very different promises. SSH on <code>0.0.0.0:22</code> takes connections on any IPv4 address ws1 has
(<code>[::]:22</code> is the same for IPv6). The web server on <code>127.0.0.1:9999</code> takes them only on loopback. The
<code>127.0.0.11</code> line is Docker&rsquo;s own DNS helper inside the container.</p>
<p><code>-p</code> adds the process, but only for sockets you own, unless you use <code>sudo</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ sudo ss -tlnp &#34;sport = :9999&#34;
</span></span><span class="line"><span class="cl">State  Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
</span></span><span class="line"><span class="cl">LISTEN 0      5          127.0.0.1:9999      0.0.0.0:*    users:((&#34;python3&#34;,pid=467,fd=3))
</span></span></code></pre></div><p>The <code>&quot;sport = :9999&quot;</code> part is a filter: only sockets whose source port is 9999. Then I tried the same server
listening three ways, and connected to it by both addresses:</p>
<table>
  <thead>
      <tr>
          <th>Listening on</th>
          <th><code>curl 127.0.0.1:9999</code></th>
          <th><code>curl 192.168.10.10:9999</code></th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>127.0.0.1</code></td>
          <td>the page</td>
          <td>refused</td>
      </tr>
      <tr>
          <td><code>192.168.10.10</code></td>
          <td>refused</td>
          <td>the page</td>
      </tr>
      <tr>
          <td><code>0.0.0.0</code></td>
          <td>the page</td>
          <td>the page</td>
      </tr>
  </tbody>
</table>
<p><code>127.0.0.1</code> is the &ldquo;works on the server, nowhere else&rdquo; case. Loopback traffic never leaves the machine, so no
other host can ever reach it, whatever the firewall says. Plenty of software listens there by default, on
purpose, until you tell it otherwise. A specific address, such as <code>192.168.10.10</code>, accepts only connections sent
to that address. <code>0.0.0.0</code> (or <code>*</code> in some tools) means all of them. When <code>ss</code> says the right port but the wrong
address, the fix is in the program&rsquo;s configuration, not the network.</p>
<hr>
<h2 id="what-does-curl--v-show">What does curl -v show?</h2>
<p><code>curl -v</code> (<em>verbose</em>) narrates the whole request: lines starting <code>*</code> are curl&rsquo;s notes, <code>&gt;</code> is what it sent,
<code>&lt;</code> is what came back. Here it is against the same test server on <code>0.0.0.0</code>, by name:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ curl -sv http://ws1.lab:9999/
</span></span><span class="line"><span class="cl">* Host ws1.lab:9999 was resolved.
</span></span><span class="line"><span class="cl">* IPv6: (none)
</span></span><span class="line"><span class="cl">* IPv4: 192.168.10.10
</span></span><span class="line"><span class="cl">*   Trying 192.168.10.10:9999...
</span></span><span class="line"><span class="cl">* Connected to ws1.lab (192.168.10.10) port 9999
</span></span><span class="line"><span class="cl">* using HTTP/1.x
</span></span><span class="line"><span class="cl">&gt; GET / HTTP/1.1
</span></span><span class="line"><span class="cl">&gt; Host: ws1.lab:9999
</span></span><span class="line"><span class="cl">&gt; User-Agent: curl/8.14.1
</span></span><span class="line"><span class="cl">&gt; Accept: */*
</span></span><span class="line"><span class="cl">&gt;
</span></span><span class="line"><span class="cl">* Request completely sent off
</span></span><span class="line"><span class="cl">* HTTP 1.0, assume close after body
</span></span><span class="line"><span class="cl">&lt; HTTP/1.0 200 OK
</span></span><span class="line"><span class="cl">&lt; Server: SimpleHTTP/0.6 Python/3.13.5
</span></span><span class="line"><span class="cl">&lt; Date: Wed, 11 Nov 2026 05:57:33 GMT
</span></span><span class="line"><span class="cl">&lt; Content-type: text/html
</span></span><span class="line"><span class="cl">&lt; Content-Length: 26
</span></span><span class="line"><span class="cl">&lt; Last-Modified: Wed, 11 Nov 2026 05:57:12 GMT
</span></span><span class="line"><span class="cl">&lt;
</span></span><span class="line"><span class="cl">{ [26 bytes data]
</span></span><span class="line"><span class="cl">* shutting down connection #0
</span></span><span class="line"><span class="cl">Hello from ws1, port 9999
</span></span></code></pre></div><p>Read it as layers, top down. The first block is <strong>DNS</strong>: the name became <code>192.168.10.10</code>. <code>Trying</code> then
<code>Connected</code> is <strong>TCP</strong>: the handshake worked. The <code>&gt;</code> lines are the <strong>HTTP</strong> request, and <code>&lt; HTTP/1.0 200 OK</code>
is the answer. (<code>-s</code> only hides the progress meter.) Wherever it stops tells you which layer failed:</p>
<table>
  <thead>
      <tr>
          <th>curl says</th>
          <th>Layer</th>
          <th>Usual cause</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>Could not resolve host: nosuch.lab</code> (exit 6)</td>
          <td>DNS</td>
          <td>wrong name, or the DNS server doesn&rsquo;t know it</td>
      </tr>
      <tr>
          <td><code>connect to ... failed: Connection refused</code> (exit 7)</td>
          <td>TCP</td>
          <td>the host answered, but nothing listens on that address and port</td>
      </tr>
      <tr>
          <td><code>connect to ... failed: No route to host</code> (exit 7)</td>
          <td>IP</td>
          <td>nobody has that address, or no route to it</td>
      </tr>
      <tr>
          <td><code>Connection timed out</code> or <code>Operation timed out</code> (exit 28)</td>
          <td>IP or firewall</td>
          <td>something dropped the packets silently</td>
      </tr>
      <tr>
          <td><code>&lt; HTTP/1.1 404 Not Found</code></td>
          <td>HTTP</td>
          <td>the server works, the path doesn&rsquo;t</td>
      </tr>
  </tbody>
</table>
<p>The two <code>connect</code> failures, from the lab: a port nobody listened on, and an address nobody had.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">* connect to 192.168.10.10 port 9999 from 192.168.10.10 port 38612 failed: Connection refused
</span></span><span class="line"><span class="cl">* connect to 192.168.20.99 port 80 from 192.168.10.10 port 39970 failed: No route to host
</span></span></code></pre></div><p>Refused came back in 5 ms. No route to host took about 3 seconds: most likely the time r1 spent asking for <code>192.168.20.99</code> with ARP
before it gave up and told ws1. <code>-m 5</code> caps the whole request at 5 seconds, which you want in a script.</p>
<p>For HTTPS, <code>-v</code> adds the <strong>TLS</strong> layer between TCP and HTTP. Trimmed, from the codespace&rsquo;s curl 7.88.1 against
this site:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">* Connected to networkdoodles.com (35.185.44.232) port 443 (#0)
</span></span><span class="line"><span class="cl">* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
</span></span><span class="line"><span class="cl">* ALPN: server accepted h2
</span></span><span class="line"><span class="cl">* Server certificate:
</span></span><span class="line"><span class="cl">*  subject: CN=networkdoodles.com
</span></span><span class="line"><span class="cl">*  expire date: Dec 11 12:42:43 2026 GMT
</span></span><span class="line"><span class="cl">*  issuer: C=US; O=Let&#39;s Encrypt; CN=YR1
</span></span><span class="line"><span class="cl">*  SSL certificate verify ok.
</span></span><span class="line"><span class="cl">&gt; GET / HTTP/2
</span></span><span class="line"><span class="cl">&lt; HTTP/2 200
</span></span></code></pre></div><p>That&rsquo;s the certificate&rsquo;s name, expiry and issuer without opening a browser. The
<a href="https://curl.se/docs/manpage.html">curl man page</a> lists every exit code.</p>
<hr>
<h2 id="when-do-i-want-digs-full-answer-and-when-short">When do I want dig&rsquo;s full answer, and when +short?</h2>
<p><code>dig +short</code> prints only the answer, which is what a script wants:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ dig +short srv1.lab
</span></span><span class="line"><span class="cl">192.168.20.10
</span></span></code></pre></div><p>The full answer is what you want when something&rsquo;s wrong:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ dig srv1.lab
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">; &lt;&lt;&gt;&gt; DiG 9.20.30-1~deb13u1-Debian &lt;&lt;&gt;&gt; srv1.lab
</span></span><span class="line"><span class="cl">;; global options: +cmd
</span></span><span class="line"><span class="cl">;; Got answer:
</span></span><span class="line"><span class="cl">;; -&gt;&gt;HEADER&lt;&lt;- opcode: QUERY, status: NOERROR, id: 51819
</span></span><span class="line"><span class="cl">;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">;; OPT PSEUDOSECTION:
</span></span><span class="line"><span class="cl">; EDNS: version: 0, flags:; udp: 1232
</span></span><span class="line"><span class="cl">;; QUESTION SECTION:
</span></span><span class="line"><span class="cl">;srv1.lab.			IN	A
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">;; ANSWER SECTION:
</span></span><span class="line"><span class="cl">srv1.lab.		60	IN	A	192.168.20.10
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">;; Query time: 2 msec
</span></span><span class="line"><span class="cl">;; SERVER: 192.168.20.10#53(192.168.20.10) (UDP)
</span></span><span class="line"><span class="cl">;; WHEN: Wed Nov 11 05:57:44 UTC 2026
</span></span><span class="line"><span class="cl">;; MSG SIZE  rcvd: 53
</span></span></code></pre></div><table>
  <thead>
      <tr>
          <th>Part</th>
          <th>Tells you</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>status: NOERROR</code></td>
          <td>the name exists. <code>NXDOMAIN</code> means it doesn&rsquo;t, <code>REFUSED</code> means this server won&rsquo;t answer</td>
      </tr>
      <tr>
          <td><code>flags: ... aa</code></td>
          <td><em>authoritative answer</em>: this server owns the name, it didn&rsquo;t ask anyone else</td>
      </tr>
      <tr>
          <td><code>60</code> in the answer</td>
          <td>the TTL: how many seconds clients may cache it</td>
      </tr>
      <tr>
          <td><code>SERVER:</code></td>
          <td>which DNS server actually answered</td>
      </tr>
  </tbody>
</table>
<p><code>+short</code> hides the status, and that matters. A name that doesn&rsquo;t exist prints <strong>nothing</strong>, and <code>dig</code> still
exits 0, because it did get an answer: <em>no such name</em>. The full output shows <code>status: NXDOMAIN</code>.</p>
<p>One more trap: <code>dig</code> ignores the <code>search lab</code> line in <code>/etc/resolv.conf</code>. <code>curl http://srv1</code> works, because it
uses the system&rsquo;s resolver, which adds <code>.lab</code>. <code>dig srv1</code> asks for exactly <code>srv1</code>, and the lab&rsquo;s DNS server
answers <code>status: REFUSED</code>. Use the full name, or <code>dig +search srv1</code>.</p>
<hr>
<h2 id="how-do-i-capture-packets-with-tcpdump">How do I capture packets with tcpdump?</h2>
<p><code>tcpdump</code> prints, or saves, the packets crossing an interface. It needs root, because seeing other programs'
traffic is a privilege:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ tcpdump -n -i eth1 -c 1
</span></span><span class="line"><span class="cl">tcpdump: eth1: You don&#39;t have permission to perform this capture on that device
</span></span><span class="line"><span class="cl">(socket: Operation not permitted)
</span></span></code></pre></div><p>With <code>sudo</code>, and a <strong>capture filter</strong> that keeps only DNS, while <code>dig +short srv1.lab</code> ran in another terminal:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ sudo tcpdump -n -i eth1 udp port 53
</span></span><span class="line"><span class="cl">tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
</span></span><span class="line"><span class="cl">listening on eth1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
</span></span><span class="line"><span class="cl">05:59:01.773866 IP 192.168.10.10.33468 &gt; 192.168.20.10.53: 35944+ [1au] A? srv1.lab. (49)
</span></span><span class="line"><span class="cl">05:59:01.782984 IP 192.168.20.10.53 &gt; 192.168.10.10.33468: 35944* 1/0/1 A 192.168.20.10 (53)
</span></span></code></pre></div><p>The question (<code>A? srv1.lab.</code>) and the answer (<code>A 192.168.20.10</code>), with each address written as
<code>address.port</code>.</p>
<table>
  <thead>
      <tr>
          <th>Option or filter</th>
          <th>Does</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>-i eth1</code></td>
          <td>capture on this interface. Pick the one <code>ip route get</code> named</td>
      </tr>
      <tr>
          <td><code>-n</code></td>
          <td>don&rsquo;t turn addresses and ports into names (and don&rsquo;t send DNS queries of its own)</td>
      </tr>
      <tr>
          <td><code>-c 10</code></td>
          <td>stop after 10 packets</td>
      </tr>
      <tr>
          <td><code>-w file.pcap</code></td>
          <td>save raw packets to a file instead of printing them</td>
      </tr>
      <tr>
          <td><code>-r file.pcap</code></td>
          <td>read a saved file back</td>
      </tr>
      <tr>
          <td><code>host 192.168.20.10</code></td>
          <td>only packets to or from this address</td>
      </tr>
      <tr>
          <td><code>port 8080</code>, <code>tcp port 22</code></td>
          <td>only this port</td>
      </tr>
      <tr>
          <td><code>and</code>, <code>or</code>, <code>not</code></td>
          <td>combine them: <code>host 192.168.20.10 and tcp port 22</code></td>
      </tr>
  </tbody>
</table>
<p>The filter language is libpcap&rsquo;s, documented in <a href="https://www.tcpdump.org/manpages/pcap-filter.7.html">pcap-filter(7)</a>,
and the options in <a href="https://www.tcpdump.org/manpages/tcpdump.1.html">tcpdump(1)</a>. A capture filter decides what&rsquo;s
kept at all: anything it drops is gone. Wireshark 101&rsquo;s lesson on
<a href="/lessons/wireshark-101/capture-vs-display-filters/">capture filters vs display filters</a> covers the difference.</p>
<p>Saving with <code>-w</code> and reading back with <code>-r</code>: a test connection from ws1 to srv1&rsquo;s SSH port, written to a file
and then read back:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ sudo tcpdump -n -i eth1 -w /tmp/lesson8-ssh.pcap host 192.168.20.10 and tcp port 22
</span></span><span class="line"><span class="cl">tcpdump: listening on eth1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
</span></span><span class="line"><span class="cl">6 packets captured
</span></span><span class="line"><span class="cl">$ tcpdump -n -r /tmp/lesson8-ssh.pcap
</span></span><span class="line"><span class="cl">reading from file /tmp/lesson8-ssh.pcap, link-type EN10MB (Ethernet), snapshot length 262144
</span></span><span class="line"><span class="cl">05:59:05.387230 IP 192.168.10.10.60292 &gt; 192.168.20.10.22: Flags [S], seq 3914208904, win 64240, options [mss 1460,sackOK,TS val 1308834635 ecr 0,nop,wscale 7], length 0
</span></span><span class="line"><span class="cl">05:59:05.387361 IP 192.168.20.10.22 &gt; 192.168.10.10.60292: Flags [S.], seq 1686286914, ack 3914208905, win 65160, options [mss 1460,sackOK,TS val 590197084 ecr 1308834635,nop,wscale 7], length 0
</span></span><span class="line"><span class="cl">05:59:05.387369 IP 192.168.10.10.60292 &gt; 192.168.20.10.22: Flags [.], ack 1, win 502, options [nop,nop,TS val 1308834635 ecr 590197084], length 0
</span></span><span class="line"><span class="cl">05:59:05.420529 IP 192.168.20.10.22 &gt; 192.168.10.10.60292: Flags [P.], seq 1:42, ack 1, win 510, options [nop,nop,TS val 590197117 ecr 1308834635], length 41: SSH: SSH-2.0-OpenSSH_10.0p2 Debian-7+deb13u4
</span></span><span class="line"><span class="cl">05:59:05.420549 IP 192.168.10.10.60292 &gt; 192.168.20.10.22: Flags [.], ack 42, win 502, options [nop,nop,TS val 1308834668 ecr 590197117], length 0
</span></span><span class="line"><span class="cl">05:59:05.420966 IP 192.168.10.10.60292 &gt; 192.168.20.10.22: Flags [R.], seq 1, ack 42, win 502, options [nop,nop,TS val 1308834668 ecr 590197117], length 0
</span></span></code></pre></div><p>Reading the file back needs no <code>sudo</code>. The first three lines are the TCP handshake: <code>[S]</code> (SYN), <code>[S.]</code>
(SYN-ACK) and <code>[.]</code> (ACK). Then srv1 sends its SSH banner, and my test hung up abruptly, so the last packet is a
reset, <code>[R.]</code>. <code>60292</code> is ws1&rsquo;s temporary port for this one connection. A <strong>pcap</strong> file like this opens in
Wireshark, which is the better place to read anything longer than a handful of lines:
<a href="/lessons/wireshark-101/tcp/">TCP in Wireshark</a> reads the same flags field by field.</p>
<hr>
<h2 id="what-i-see-go-wrong">What I see go wrong</h2>
<p><strong>&ldquo;It works on the server.&rdquo;</strong> <code>curl localhost</code> on the server proves the program runs. It doesn&rsquo;t prove anyone
else can reach it. Check the address in <code>ss -tlnp</code> before you look at the network.</p>
<p><strong>Reading <code>0.0.0.0</code> in the Peer column as a problem.</strong> On a <code>LISTEN</code> line, <code>0.0.0.0:*</code> under Peer Address just
means nobody&rsquo;s connected yet. The address that matters is the Local one.</p>
<p><strong><code>dig</code> and the application disagree.</strong> <code>dig</code> asks the DNS server directly and skips <code>/etc/hosts</code> and the
search list. The application uses the system resolver, which reads both. <code>getent hosts srv1</code> asks the way the
application does.</p>
<p><strong><code>dig +short</code> printing nothing.</strong> That&rsquo;s an answer: the name doesn&rsquo;t exist (or has no record of that type).
Run it without <code>+short</code> and read the status.</p>
<p><strong>Capturing on the wrong interface.</strong> ws1 has two. Traffic to srv1 leaves by <code>eth1</code>; a capture on <code>eth0</code>
shows nothing and looks like &ldquo;no traffic&rdquo;. <code>ip route get</code> says which one.</p>
<p><strong>Forgetting <code>-n</code> in tcpdump.</strong> Name lookups slow the output down, and the DNS queries tcpdump sends for them
can end up in your own capture.</p>
<p><strong>A <code>-w</code> file that&rsquo;s short or empty.</strong> tcpdump buffers what it writes. Stop it with Ctrl-C, or let <code>-c</code> stop it,
so it finishes writing the file. Killed any harder, it can leave the last packets out.</p>
<hr>
<h2 id="try-it">Try it</h2>
<p><a href="/labs/linux-cli-101/task-8-1/">Chapter 8 of the lab</a> works from ws1 towards srv1, through r1. You find the
gateway and the outgoing interface, count the hops and resolve srv1 with <code>dig</code>. Then a web service on srv1
answers when you <code>curl</code> it on srv1 itself but not from ws1, and it&rsquo;s yours to find out why and fix.
Last, you capture the conversation with <code>tcpdump -w</code>, copy the file back with <code>scp</code>, and read the handshake out
of it.</p>
<h2 id="where-to-go-next">Where to go next</h2>
<p>That&rsquo;s the end of Linux and the command line 101. You can now move around the filesystem, read and search files,
chain tools with pipes, fix permissions, manage processes, log in with keys, script a check, and troubleshoot a
service from the command line. Every lab on this site assumes those.</p>
<p>Two courses pick up from here:</p>
<ul>
<li><a href="/lessons/wireshark-101/">Wireshark 101</a> takes the pcap files <code>tcpdump</code> saves and reads them properly:
ARP, DHCP, DNS, the TCP handshake and TLS, and a broken path to troubleshoot.</li>
<li><a href="/lessons/python-101/">Python 101</a> takes over where Bash gets awkward: real data structures, parsing <code>show</code>
output, and logging in to every device from one script.</li>
</ul>
<p>If you want a refresher first, the course starts at <a href="/lessons/linux-cli-101/shell-and-filesystem/">the shell and the filesystem</a>.</p>
]]></content:encoded></item><item><title>What TLS Shows Without Decrypting It: SNI, Versions and Ciphers (Wireshark 101, Lesson 7)</title><link>https://networkdoodles.com/lessons/wireshark-101/tls-without-decryption/</link><pubDate>Mon, 16 Nov 2026 09:00:00 -0500</pubDate><dc:creator>David Henderson</dc:creator><guid>https://networkdoodles.com/lessons/wireshark-101/tls-without-decryption/</guid><category>wireshark</category><category>packet-capture</category><category>networking-basics</category><category>tls</category><description>What a TLS handshake shows without decrypting: SNI, versions, cipher, ALPN and the certificate, TLS 1.2 vs 1.3, and key log files for your own traffic.</description><content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[<p>TLS encrypts the content of a connection, but not the start of it. Before the two ends can encrypt anything,
they have to agree how, and that negotiation, the <strong>handshake</strong>, is partly in the clear. From a capture,
without any keys, you can read the server name the client asked for, the TLS versions and cipher suites it
offered, the ones the server chose, and the application protocols on offer. In TLS 1.2 you can read the
server&rsquo;s certificate too. In TLS 1.3 you can&rsquo;t. And with a key log file from your own client, you can decrypt
the rest.</p>
<p>That&rsquo;s usually all a network engineer needs. When an HTTPS connection fails, the question is rarely &ldquo;what was in
the page&rdquo;. It&rsquo;s &ldquo;which name, which version, which certificate, and did the handshake finish&rdquo;. This lesson uses
two connections from the <a href="/labs/wireshark-101/">Wireshark 101 lab</a>: curl opening <code>https://web.lab/</code> with TLS
1.3, and <code>https://intranet.lab/secret.txt</code> with TLS 1.2.</p>
<hr>
<h2 id="what-is-tls-and-what-does-it-hide">What is TLS, and what does it hide?</h2>
<p><strong>TLS</strong> (Transport Layer Security) is the S in HTTPS. It runs on top of an ordinary TCP connection
(<a href="/lessons/wireshark-101/tcp/">lesson 6</a>) and encrypts everything the application sends: the page, the path
after the name, headers, cookies, passwords. In a capture, all of that is a series of records labeled
<em>Application Data</em>, with a length and nothing else.</p>
<p>What it doesn&rsquo;t hide:</p>
<table>
  <thead>
      <tr>
          <th>Visible</th>
          <th>Where</th>
          <th>Why it&rsquo;s there</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>IP addresses and ports</td>
          <td>IP and TCP headers</td>
          <td>TLS rides inside TCP; routers still need them</td>
      </tr>
      <tr>
          <td><strong>SNI</strong>, the server name</td>
          <td>Client Hello</td>
          <td>one server can host many names, and it needs to know which certificate to send</td>
      </tr>
      <tr>
          <td>versions and cipher suites offered</td>
          <td>Client Hello</td>
          <td>the client lists what it can do</td>
      </tr>
      <tr>
          <td>version and cipher suite chosen</td>
          <td>Server Hello</td>
          <td>the server picks one of each</td>
      </tr>
      <tr>
          <td><strong>ALPN</strong> protocols offered</td>
          <td>Client Hello</td>
          <td>which application protocol will run inside, such as <code>h2</code> or <code>http/1.1</code></td>
      </tr>
      <tr>
          <td>the server&rsquo;s certificate</td>
          <td>TLS 1.2 only</td>
          <td>in TLS 1.3 it&rsquo;s encrypted</td>
      </tr>
      <tr>
          <td>record sizes and timing</td>
          <td>every record</td>
          <td>encryption hides content, not length</td>
      </tr>
  </tbody>
</table>
<p>The <strong>Client Hello</strong> is the client&rsquo;s first message: everything it supports, in one go. The <strong>Server Hello</strong> is
the server&rsquo;s reply with its choices. Everything else follows from those two.</p>
<hr>
<h2 id="sni-the-name-you-asked-for">SNI: the name you asked for</h2>
<p><strong>SNI</strong> (Server Name Indication) is an extension in the Client Hello that carries the host name. Wireshark puts
it straight into the Info column:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl"> 9 0.321656708 192.168.1.175 → 192.168.2.10 SSL 1514
</span></span><span class="line"><span class="cl">10 0.321665750 192.168.1.175 → 192.168.2.10 TLSv1.2 176 Client Hello (SNI=web.lab)
</span></span><span class="line"><span class="cl">...
</span></span><span class="line"><span class="cl">34 0.389741458 192.168.1.175 → 192.168.2.10 SSL 1514
</span></span><span class="line"><span class="cl">35 0.389743208 192.168.1.175 → 192.168.2.10 TLSv1.2 181 Client Hello (SNI=intranet.lab)
</span></span></code></pre></div><p>Both connections go to the same address, 192.168.2.10. The IP header can&rsquo;t tell them apart. The SNI can. It
exists because one address commonly serves many names, and the server has to choose the right certificate
before any encryption starts.</p>
<p>That makes SNI the most useful field in an HTTPS capture. It&rsquo;s how you find which connection is the one to
the site that&rsquo;s broken, and it&rsquo;s what firewalls, proxies and monitoring tools that don&rsquo;t decrypt use to decide
what a connection is. The filter <code>tls.handshake.extensions_server_name == &quot;web.lab&quot;</code> finds every connection to
one name.</p>
<hr>
<h2 id="why-does-the-client-hello-say-tls-12-when-its-tls-13">Why does the Client Hello say TLS 1.2 when it&rsquo;s TLS 1.3?</h2>
<p>Look at the Protocol column above: the Client Hello says <em>TLSv1.2</em>. Open it and it gets stranger:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">TLSv1.3 Record Layer: Handshake Protocol: Client Hello
</span></span><span class="line"><span class="cl">    Version: TLS 1.0 (0x0301)
</span></span><span class="line"><span class="cl">    Handshake Protocol: Client Hello
</span></span><span class="line"><span class="cl">        Version: TLS 1.2 (0x0303)
</span></span><span class="line"><span class="cl">        ...
</span></span><span class="line"><span class="cl">        Extension: supported_versions (len=5) TLS 1.3, TLS 1.2
</span></span></code></pre></div><p>Three version numbers in one message, and only the last one is honest. This is deliberate.
<a href="https://www.rfc-editor.org/rfc/rfc8446">RFC 8446</a>, the TLS 1.3 standard, freezes the old version fields at
TLS 1.2 (and allows 1.0 on the record of the first Client Hello), because some middleboxes broke connections
that showed a version they didn&rsquo;t recognize. The real negotiation moved into an extension,
<code>supported_versions</code>, where the client lists every version it can do and the server answers with the one it
picked.</p>
<p>The Server Hello shows the result:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">TLSv1.3 Record Layer: Handshake Protocol: Server Hello
</span></span><span class="line"><span class="cl">    Version: TLS 1.2 (0x0303)
</span></span><span class="line"><span class="cl">    Handshake Protocol: Server Hello
</span></span><span class="line"><span class="cl">        Version: TLS 1.2 (0x0303)
</span></span><span class="line"><span class="cl">        Cipher Suite: TLS_AES_256_GCM_SHA384 (0x1302)
</span></span><span class="line"><span class="cl">        Extension: supported_versions (len=2) TLS 1.3
</span></span><span class="line"><span class="cl">        Extension: key_share (len=1124) X25519MLKEM768
</span></span></code></pre></div><p>Twice &ldquo;TLS 1.2&rdquo;, once &ldquo;TLS 1.3&rdquo;, and the extension wins. Wireshark knows this and labels the record TLSv1.3. The
rule for reading a version: <strong>trust <code>supported_versions</code> in the Server Hello.</strong> If there isn&rsquo;t one, the
connection is TLS 1.2 or older, and the Server Hello&rsquo;s own version field is the real one. The filter
<code>tls.handshake.extensions.supported_version == 0x0304</code> matches TLS 1.3 handshakes.</p>
<hr>
<h2 id="cipher-suites-offered-and-chosen">Cipher suites: offered and chosen</h2>
<p>A <strong>cipher suite</strong> names the algorithms a connection will use. The laptop&rsquo;s curl offered 30 of them, most
preferred first:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">Cipher Suites (30 suites)
</span></span><span class="line"><span class="cl">    Cipher Suite: TLS_AES_256_GCM_SHA384 (0x1302)
</span></span><span class="line"><span class="cl">    Cipher Suite: TLS_CHACHA20_POLY1305_SHA256 (0x1303)
</span></span><span class="line"><span class="cl">    Cipher Suite: TLS_AES_128_GCM_SHA256 (0x1301)
</span></span><span class="line"><span class="cl">    Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 (0xc02c)
</span></span><span class="line"><span class="cl">    Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (0xc030)
</span></span><span class="line"><span class="cl">    ...
</span></span></code></pre></div><p>The first three are TLS 1.3 suites. In 1.3 a suite names only the encryption (AES-256 in GCM mode) and the hash
(SHA-384); the key exchange is negotiated separately. The rest are TLS 1.2 suites, which name everything in one
string: <code>ECDHE_RSA_WITH_AES_256_GCM_SHA384</code> is an ECDHE key exchange, an RSA certificate, AES-256-GCM and SHA-384.</p>
<p>The two lab connections chose differently:</p>
<table>
  <thead>
      <tr>
          <th></th>
          <th>TLS 1.3 connection</th>
          <th>TLS 1.2 connection</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>Version</td>
          <td>TLS 1.3 (from <code>supported_versions</code>)</td>
          <td>TLS 1.2 (no <code>supported_versions</code>)</td>
      </tr>
      <tr>
          <td>Cipher suite</td>
          <td><code>TLS_AES_256_GCM_SHA384</code> (0x1302)</td>
          <td><code>TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384</code> (0xc030)</td>
      </tr>
      <tr>
          <td>ALPN chosen</td>
          <td>encrypted</td>
          <td><code>http/1.1</code>, in the clear</td>
      </tr>
      <tr>
          <td>Certificate</td>
          <td>encrypted</td>
          <td>in the clear</td>
      </tr>
  </tbody>
</table>
<p>When a handshake fails because the two sides have nothing in common, this is where you see it: compare the
client&rsquo;s list with what the server supports. A server that answers a Client Hello with an alert instead of a
Server Hello usually didn&rsquo;t like something in that list.</p>
<hr>
<h2 id="why-is-the-labs-client-hello-1558-bytes">Why is the lab&rsquo;s Client Hello 1,558 bytes?</h2>
<p>Notice frame 9 in the listing above: <em>SSL, 1514 bytes</em>, with no description. The Client Hello didn&rsquo;t fit in one
packet. It&rsquo;s 1,558 bytes of TCP payload, more than the 1,460 a segment can carry on a 1,500-byte network, so TCP
split it, and Wireshark can only decode it once frame 10 completes it.</p>
<p>The reason is its <strong>key share</strong>. In TLS 1.3 the client guesses which key exchange the server will pick and sends
its half of it straight away, to save a round trip. curl on the lab&rsquo;s laptop offers <code>X25519MLKEM768</code> first: a
hybrid that combines the classic X25519 exchange with ML-KEM, a key exchange designed to resist future quantum
computers.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">Extension: key_share (len=1258) X25519MLKEM768, x25519
</span></span><span class="line"><span class="cl">    Key Share Entry: Group: X25519MLKEM768, Key Exchange length: 1216
</span></span><span class="line"><span class="cl">    Key Share Entry: Group: x25519, Key Exchange length: 32
</span></span></code></pre></div><p>The hybrid share is 1,216 bytes; the plain X25519 one is 32. The server took the hybrid, and its own share in the
Server Hello is 1,120 bytes. A whole Client Hello from a few years ago fitted in a few hundred bytes. The
lab&rsquo;s TLS 1.2 Client Hello, which leaves out the TLS 1.3 key shares, is a 292-byte frame.</p>
<p>That size has consequences. A Client Hello that needs two segments is the first thing to break on a path with an
MTU problem, while ping, DNS and small HTTP requests keep working.
<a href="/lessons/wireshark-101/broken-path/">Lesson 8</a> breaks exactly that.</p>
<hr>
<h2 id="alpn-which-protocol-runs-inside">ALPN: which protocol runs inside</h2>
<p><strong>ALPN</strong> (Application-Layer Protocol Negotiation) lets the client say which protocols it can speak inside the
encrypted connection. curl offered two:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">Extension: application_layer_protocol_negotiation (len=14)
</span></span><span class="line"><span class="cl">    ALPN Next Protocol: h2
</span></span><span class="line"><span class="cl">    ALPN Next Protocol: http/1.1
</span></span></code></pre></div><p><code>h2</code> is HTTP/2. The server&rsquo;s choice is where TLS 1.2 and 1.3 differ. In TLS 1.2 it&rsquo;s in the Server Hello, in the
clear: the lab&rsquo;s server answered <code>http/1.1</code>. In TLS 1.3 it moved into the <em>Encrypted Extensions</em> message, sent
after encryption starts, so a capture only shows what was offered.</p>
<hr>
<h2 id="the-certificate-in-the-clear-in-12-encrypted-in-13">The certificate: in the clear in 1.2, encrypted in 1.3</h2>
<p>A <strong>certificate</strong> is a signed document that says &ldquo;this public key belongs to these names&rdquo;, signed by a
<strong>certificate authority</strong> (CA) that the client already trusts. In TLS 1.2 the server sends it in the clear, right
after its Server Hello:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">11 0.252885000 192.168.2.10 → 192.168.1.175 TLSv1.2 1402 Server Hello, Certificate, Server Key Exchange, Server Hello Done
</span></span></code></pre></div><p>Open it and the details are all there:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">issuer: rdnSequence (0)
</span></span><span class="line"><span class="cl">    rdnSequence: 2 items (id-at-commonName=Network Doodles Lab CA,id-at-organizationName=Network Doodles Lab)
</span></span><span class="line"><span class="cl">subject: rdnSequence (0)
</span></span><span class="line"><span class="cl">    rdnSequence: 2 items (id-at-commonName=web.lab,id-at-organizationName=Network Doodles Lab)
</span></span><span class="line"><span class="cl">...
</span></span><span class="line"><span class="cl">    GeneralName: dNSName (2)
</span></span><span class="line"><span class="cl">        dNSName: web.lab
</span></span><span class="line"><span class="cl">    GeneralName: dNSName (2)
</span></span><span class="line"><span class="cl">        dNSName: intranet.lab
</span></span></code></pre></div><p>The client asked for <code>intranet.lab</code>, and got a certificate whose common name is <code>web.lab</code>. That isn&rsquo;t an error:
the <strong>subjectAltName</strong> (SAN) list covers both names, and the SAN list is what clients check. Reading this off the
wire is how you settle &ldquo;certificate name mismatch&rdquo; and &ldquo;unknown issuer&rdquo; arguments without logging in to the
server.</p>
<p>In TLS 1.3, the Server Hello is followed by <em>Change Cipher Spec, Application Data</em>. The Change Cipher Spec is a
leftover that RFC 8446 keeps for middlebox compatibility. The certificate is inside the Application Data,
encrypted with keys from the key exchange. A capture of a TLS 1.3 connection doesn&rsquo;t show which certificate the
server sent.</p>
<hr>
<h2 id="decrypting-your-own-traffic-with-a-key-log-file">Decrypting your own traffic with a key log file</h2>
<p>You can&rsquo;t decrypt someone else&rsquo;s TLS from a capture. With an ephemeral key exchange, which both lab connections
used (ECDHE in 1.2, the X25519MLKEM768 hybrid in 1.3), even the server&rsquo;s private key isn&rsquo;t enough: each connection makes fresh keys and throws them away.
That property is called forward secrecy.</p>
<p>But you can decrypt your <strong>own</strong> traffic, because your client knows the keys. Many TLS libraries write their
session secrets to a file when the environment variable <code>SSLKEYLOGFILE</code> names one. curl does, and so do Firefox
and Chrome. The file is plain text, one line per secret:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">CLIENT_RANDOM &lt;64 hex digits: the Client Hello&#39;s random&gt; &lt;96 hex digits: the session&#39;s secret&gt;
</span></span></code></pre></div><p>The random value tells Wireshark which connection a line belongs to. Point Wireshark at the file (the TLS
protocol preferences, <em>(Pre)-Master-Secret log filename</em>) and the Application Data becomes what it was:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">15 0.259969042 192.168.1.175 → 192.168.2.10 HTTP 181 GET /secret.txt HTTP/1.1
</span></span><span class="line"><span class="cl">16 0.260409750 192.168.2.10 → 192.168.1.175 HTTP 442 HTTP/1.1 200 OK  (text/plain)
</span></span></code></pre></div><p>The capture didn&rsquo;t change. The keys are the only thing that makes it readable. Wireshark&rsquo;s
<a href="https://wiki.wireshark.org/TLS">TLS page</a> covers the other ways to supply keys.</p>
<h3 id="why-a-key-log-file-is-as-sensitive-as-a-password">Why a key log file is as sensitive as a password</h3>
<p>Anyone who has the key log file and a capture of the same traffic can read everything in it: session cookies,
passwords typed into forms, API tokens. It defeats forward secrecy completely, after the fact, for every
connection in the file. So:</p>
<ul>
<li>set <code>SSLKEYLOGFILE</code> for one debugging session, in one shell, never in a profile that every program inherits;</li>
<li>keep the file next to the capture only as long as you need it, then delete both;</li>
<li>never attach a key log file to a ticket, a chat or a shared drive with the capture.</li>
</ul>
<hr>
<h2 id="what-about-encrypted-client-hello">What about Encrypted Client Hello?</h2>
<p><strong>ECH</strong> (Encrypted Client Hello) closes the biggest gap left in TLS 1.3. It was published in March 2026 as
<a href="https://www.rfc-editor.org/rfc/rfc9849">RFC 9849</a>. The client sends an outer Client Hello in the clear and the
real one encrypted inside it, so the real server name and the ALPN list are hidden. The outer Client Hello still
carries a server name, but it&rsquo;s a public name chosen by the server&rsquo;s operator, often shared by many sites. The
lab doesn&rsquo;t use ECH. Where it is used, the SNI in a capture tells you less than this lesson suggests.</p>
<hr>
<h2 id="what-i-see-go-wrong">What I see go wrong</h2>
<p><strong>Reading the wrong version.</strong> The Protocol column and the record header both say 1.2 for a TLS 1.3 Client
Hello. Read <code>supported_versions</code> in the Server Hello, every time.</p>
<p><strong>Hunting for the certificate in TLS 1.3.</strong> It isn&rsquo;t there to find. If you need to see which certificate a
server sends, connect with a client that shows it, or decrypt your own session.</p>
<p><strong>Missing a split Client Hello.</strong> A frame marked <em>SSL</em> with no description, followed by the Client Hello, is one
message in two segments. That&rsquo;s normal, and it&rsquo;s also the first thing to vanish on a path with an MTU problem.</p>
<p><strong>Leaving <code>SSLKEYLOGFILE</code> set.</strong> It quietly keeps writing secrets for every connection every program makes. Set
it for one command, and unset it.</p>
<hr>
<h2 id="try-it">Try it</h2>
<p><a href="/labs/wireshark-101/#chapter-7">Chapter 7 of the lab</a> has you read the SNI, the versions, the cipher and ALPN
from two connections, then read a TLS 1.2 certificate and decrypt a page with the key log file curl saved.</p>
<h2 id="whats-next">What&rsquo;s next</h2>
<p><a href="/lessons/wireshark-101/broken-path/">Lesson 8</a> puts the whole course to work on three broken paths: packet
loss, a fault the lab picks at random, and a black hole that lets ping through but not that 1,558-byte Client
Hello.</p>
]]></content:encoded></item><item><title>TCP in Wireshark: Handshake, Sequence Numbers, FIN and RST (Wireshark 101, Lesson 6)</title><link>https://networkdoodles.com/lessons/wireshark-101/tcp/</link><pubDate>Mon, 16 Nov 2026 09:00:00 -0500</pubDate><dc:creator>David Henderson</dc:creator><guid>https://networkdoodles.com/lessons/wireshark-101/tcp/</guid><category>wireshark</category><category>packet-capture</category><category>networking-basics</category><category>tcp</category><description>A TCP connection in Wireshark from SYN to FIN: handshake, sequence and ACK numbers, window, MSS, streams, and a refused connection versus silence.</description><content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[<p>TCP turns a network that loses and reorders packets into a reliable stream of bytes, and it does all of that
in the open, where a capture can watch. A connection opens with three packets (SYN, SYN-ACK, ACK), numbers
every byte it sends, acknowledges what arrived, and closes each direction with a FIN. A refused connection is
two packets: a SYN and a RST. Once you can read those, you can read most of what goes wrong on a network.</p>
<p>Every protocol in this course so far (ARP, DHCP, DNS) has been one question and one answer. A web page can&rsquo;t
work like that, so this lesson follows one: <code>curl http://web.lab/</code> in the
<a href="/labs/wireshark-101/">Wireshark 101 lab</a>, captured on the laptop&rsquo;s cable.</p>
<hr>
<h2 id="what-does-a-whole-tcp-connection-look-like">What does a whole TCP connection look like?</h2>
<p>Here&rsquo;s the entire connection, from the reference capture. DNS for <code>web.lab</code> came first and isn&rsquo;t shown.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl"> 6 0.021836458 192.168.1.175 → 192.168.2.10 TCP 74 47996 → 80 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM TSval=2403693004 TSecr=0 WS=128
</span></span><span class="line"><span class="cl"> 7 0.022324792 192.168.2.10 → 192.168.1.175 TCP 74 80 → 47996 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1460 SACK_PERM TSval=4000512039 TSecr=2403693004 WS=128
</span></span><span class="line"><span class="cl"> 8 0.022400375 192.168.1.175 → 192.168.2.10 TCP 66 47996 → 80 [ACK] Seq=1 Ack=1 Win=64256 Len=0 ...
</span></span><span class="line"><span class="cl"> 9 0.022699583 192.168.1.175 → 192.168.2.10 HTTP 137 GET / HTTP/1.1
</span></span><span class="line"><span class="cl">10 0.022722250 192.168.2.10 → 192.168.1.175 TCP 66 80 → 47996 [ACK] Seq=1 Ack=72 Win=65152 Len=0 ...
</span></span><span class="line"><span class="cl">11 0.025193000 192.168.2.10 → 192.168.1.175 TCP 302 HTTP/1.1 200 OK
</span></span><span class="line"><span class="cl">12 0.025222875 192.168.1.175 → 192.168.2.10 TCP 66 47996 → 80 [ACK] Seq=72 Ack=237 Win=64128 Len=0 ...
</span></span><span class="line"><span class="cl">13 0.025324750 192.168.2.10 → 192.168.1.175 HTTP 99 HTTP/1.1 200 OK  (text/html)
</span></span><span class="line"><span class="cl">14 0.025327875 192.168.1.175 → 192.168.2.10 TCP 66 47996 → 80 [ACK] Seq=72 Ack=270 Win=64128 Len=0 ...
</span></span><span class="line"><span class="cl">15 0.025727375 192.168.1.175 → 192.168.2.10 TCP 66 47996 → 80 [FIN, ACK] Seq=72 Ack=270 Win=64128 Len=0 ...
</span></span><span class="line"><span class="cl">16 0.026296000 192.168.2.10 → 192.168.1.175 TCP 66 80 → 47996 [FIN, ACK] Seq=270 Ack=73 Win=65152 Len=0 ...
</span></span><span class="line"><span class="cl">17 0.026305292 192.168.1.175 → 192.168.2.10 TCP 66 47996 → 80 [ACK] Seq=73 Ack=271 Win=64128 Len=0 ...
</span></span></code></pre></div><p>Twelve frames, four and a half milliseconds, for a 33-byte page. Three open the connection, three carry data
(the request, and the answer in two pieces), three acknowledge that data, and three close it. Most of the packets are TCP
keeping its promises, not carrying the page.</p>
<p><code>47996 → 80</code> is the two <strong>ports</strong>. A port says which program at each end owns the connection. Port 80 is the
web server. 47996 is an <strong>ephemeral port</strong> the laptop picked for this one connection; the next connection gets
a different one. The four numbers together (two addresses, two ports) identify a connection uniquely.</p>
<hr>
<h2 id="the-three-way-handshake">The three-way handshake</h2>
<p>Before any data moves, the two ends agree to talk and tell each other where their numbering starts.</p>
<ol>
<li><strong>SYN</strong> (synchronise), from the laptop: &ldquo;I&rsquo;d like to connect, and my bytes start here.&rdquo;</li>
<li><strong>SYN, ACK</strong>, from web: &ldquo;Agreed. Mine start here, and I&rsquo;ve got your SYN.&rdquo;</li>
<li><strong>ACK</strong>, from the laptop: &ldquo;Got yours.&rdquo;</li>
</ol>
<p>After frame 8, both sides know each other&rsquo;s starting number, and either can send. The time from the SYN to the
final ACK is the <strong>initial round-trip time</strong>. Wireshark calculates it as <code>tcp.analysis.initial_rtt</code> and shows
it on every packet of the stream: here it&rsquo;s 0.56 ms. It&rsquo;s a useful number to have, because every later delay in
the connection can be measured against it.</p>
<hr>
<h2 id="sequence-numbers-raw-and-relative">Sequence numbers: raw and relative</h2>
<p>TCP numbers <strong>bytes</strong>, not packets. Every segment carries two numbers:</p>
<ul>
<li><strong>Seq</strong>: the number of the first byte of data in this segment;</li>
<li><strong>Ack</strong>: the next byte this side expects from the other one. It means &ldquo;I have everything before this&rdquo;.</li>
</ul>
<p>Each side starts at a random <strong>initial sequence number</strong> (ISN), chosen fresh for every connection so that
stray packets from an old connection, or a guess from an attacker, don&rsquo;t land inside a new one. In this capture
the laptop&rsquo;s ISN was 500774591 and web&rsquo;s was 702278227. Those are hard to read, so Wireshark subtracts the ISN
and shows <strong>relative</strong> numbers that start at 0. The raw values are still in the packet as <code>tcp.seq_raw</code> and
<code>tcp.ack_raw</code>. You need them when you compare captures from two points and something in between rewrites
sequence numbers, which some firewalls do.</p>
<p>Now follow the numbers through the connection:</p>
<table>
  <thead>
      <tr>
          <th>Frame</th>
          <th>Who</th>
          <th>Seq</th>
          <th>Ack</th>
          <th>Len</th>
          <th>What it says</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>6</td>
          <td>laptop</td>
          <td>0</td>
          <td></td>
          <td>0</td>
          <td>SYN. A SYN counts as one byte</td>
      </tr>
      <tr>
          <td>7</td>
          <td>web</td>
          <td>0</td>
          <td>1</td>
          <td>0</td>
          <td>&ldquo;your SYN arrived; next I expect byte 1&rdquo;</td>
      </tr>
      <tr>
          <td>9</td>
          <td>laptop</td>
          <td>1</td>
          <td>1</td>
          <td>71</td>
          <td>the HTTP request, bytes 1 to 71</td>
      </tr>
      <tr>
          <td>10</td>
          <td>web</td>
          <td>1</td>
          <td>72</td>
          <td>0</td>
          <td>&ldquo;got all 71; next I expect byte 72&rdquo;</td>
      </tr>
      <tr>
          <td>11</td>
          <td>web</td>
          <td>1</td>
          <td>72</td>
          <td>236</td>
          <td>HTTP headers</td>
      </tr>
      <tr>
          <td>13</td>
          <td>web</td>
          <td>237</td>
          <td>72</td>
          <td>33</td>
          <td>the page itself</td>
      </tr>
      <tr>
          <td>14</td>
          <td>laptop</td>
          <td>72</td>
          <td>270</td>
          <td>0</td>
          <td>&ldquo;got everything up to 269&rdquo;</td>
      </tr>
  </tbody>
</table>
<p><code>Ack=72</code> after a 71-byte request starting at 1 is the whole idea. You can check, packet by packet, exactly
what each side has received. When something goes missing, it shows up as an Ack that stops moving, which is
what <a href="/lessons/wireshark-101/broken-path/">lesson 8</a> is about.</p>
<p>Frames 11 and 13 also show something you&rsquo;ll see a lot: nginx sent its headers and the body in <strong>separate
segments</strong>, 236 bytes and then 33. TCP doesn&rsquo;t care where one HTTP message ends; it carries a stream. Wireshark
reassembles the pieces and shows the complete HTTP response on the last one, frame 13, which is why frame 11&rsquo;s
Protocol column says TCP and not HTTP.</p>
<hr>
<h2 id="the-window-and-the-options-in-the-syn">The window, and the options in the SYN</h2>
<p><strong>Win</strong> is the receive <strong>window</strong>: how many more bytes this side can accept right now, before it needs the
sender to wait. It&rsquo;s flow control. A receiver that&rsquo;s falling behind advertises a smaller window, and a window
of zero means &ldquo;stop&rdquo;.</p>
<p>The window field is only 16 bits, so it can&rsquo;t say more than 65,535 bytes. That&rsquo;s where the SYN&rsquo;s options come
in. A SYN is the only time each side can announce what it supports, and the lab&rsquo;s SYN has four:</p>
<table>
  <thead>
      <tr>
          <th>Option</th>
          <th>In the SYN</th>
          <th>Means</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><strong>MSS</strong></td>
          <td><code>MSS=1460</code></td>
          <td>the biggest chunk of data this side can receive in one segment: a 1,500-byte MTU minus 20 bytes of IP header and 20 of TCP</td>
      </tr>
      <tr>
          <td><strong>SACK_PERM</strong></td>
          <td><code>SACK_PERM</code></td>
          <td>&ldquo;I understand selective acknowledgements&rdquo; (<a href="https://www.rfc-editor.org/rfc/rfc2018">RFC 2018</a>). Used when packets are lost, in lesson 8</td>
      </tr>
      <tr>
          <td><strong>Timestamps</strong></td>
          <td><code>TSval=… TSecr=…</code></td>
          <td>a clock value in every segment, echoed back, for measuring round-trip time (<a href="https://www.rfc-editor.org/rfc/rfc7323">RFC 7323</a>)</td>
      </tr>
      <tr>
          <td><strong>Window scale</strong></td>
          <td><code>WS=128</code></td>
          <td>multiply every later window value by 128 (RFC 7323)</td>
      </tr>
  </tbody>
</table>
<p>Window scaling explains a number that looks wrong at first. Frame 8 says <code>Win=64256</code>, but the raw window field
in the packet is 502. Wireshark saw <code>WS=128</code> in the handshake and did the multiplication: 502 × 128 = 64,256.
It can only do that if it captured the SYNs. Start a capture halfway through a connection and Wireshark doesn&rsquo;t
know the scale factor, so the windows it shows are too small.</p>
<p>Options only count if both sides send them. If one side&rsquo;s SYN has no <code>SACK_PERM</code>, the connection doesn&rsquo;t use
SACK. Comparing the options in the SYN and the SYN-ACK is how you find out what the connection really
negotiated.</p>
<hr>
<h2 id="fin-and-rst-two-ways-to-end">FIN and RST: two ways to end</h2>
<p>A connection closes politely with <strong>FIN</strong> (finish). Each direction closes separately: a FIN means &ldquo;I&rsquo;ve
finished sending&rdquo;, not &ldquo;go away&rdquo;. In the lab, curl had the whole page, so it sent the first FIN (frame 15). web
answered with its own FIN (16), and the laptop acknowledged it (17). A FIN counts as one byte, just like a SYN,
which is why the Ack goes from 72 to 73. Whoever sends the first FIN chose to end the conversation. Here it was
the client, even though nginx had offered <code>Connection: keep-alive</code>.</p>
<p><strong>RST</strong> (reset) is the other way: &ldquo;this connection doesn&rsquo;t exist, stop&rdquo;. No goodbye, no acknowledgement. A RST
mid-connection usually means one side crashed, timed out, or decided something was badly wrong. A RST in answer
to a SYN means something much more specific.</p>
<hr>
<h2 id="refused-versus-no-answer">Refused versus no answer</h2>
<p>Here&rsquo;s the laptop trying port 8080 on web, where nothing listens:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">21 0.071235375 192.168.1.175 → 192.168.2.10 TCP 74 55482 → 8080 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM ...
</span></span><span class="line"><span class="cl">22 0.071316959 192.168.2.10 → 192.168.1.175 TCP 54 8080 → 55482 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0
</span></span></code></pre></div><p>Two packets, 0.08 ms apart. web&rsquo;s operating system got a SYN for a port no program has open and answered
straight away with a RST. curl reports <em>Connection refused</em> and stops.</p>
<p>A refusal is good news for troubleshooting. It proves the network path works in both directions, and that
something at the far end is up and answering. The problem is the service: wrong port, program not running,
program listening on a different address.</p>
<p>Compare that with a SYN that gets <strong>no answer at all</strong>. The client can&rsquo;t tell whether the SYN was lost, so it
sends it again, and again, until it gives up. In the lab&rsquo;s broken-path chapter,
the laptop sent the same SYN at 0, 1, 2, 3, 4, 5 and 7 seconds before curl&rsquo;s eight-second limit ran out. From the
application, that&rsquo;s a hang: <em>Connection timed out</em>, eventually.</p>
<table>
  <thead>
      <tr>
          <th>What you see</th>
          <th>What it means</th>
          <th>Where to look</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>SYN, then RST, ACK</td>
          <td>the host is reachable; nothing listens on that port</td>
          <td>the server: is the service running, on that port?</td>
      </tr>
      <tr>
          <td>SYN, SYN again, SYN again…</td>
          <td>something on the path dropped the SYN or the answer</td>
          <td>the network: a filter, a route, a firewall</td>
      </tr>
      <tr>
          <td>SYN, SYN-ACK, then nothing works</td>
          <td>the connection opened; the problem is later</td>
          <td>sizes, the application, lesson 8</td>
      </tr>
  </tbody>
</table>
<p>One caution: a RST proves <em>something</em> answered, not necessarily the server. Some firewalls reject a
connection by sending a RST on the server&rsquo;s behalf. If a refusal makes no sense, capture closer to the server
and see whether the SYN ever arrived.</p>
<hr>
<h2 id="streams-follow-tcp-stream-and-conversations">Streams, Follow TCP Stream and Conversations</h2>
<p>A real capture holds many connections at once. Wireshark numbers each one, in the order it first sees them,
as <code>tcp.stream</code>. In the lab&rsquo;s refused capture, stream 0 is the first page, stream 1 the refused attempt and
stream 2 the page again. The filter <code>tcp.stream == 1</code> shows one connection and nothing else, which is the
fastest way to pull one conversation out of thousands of frames.</p>
<p>Two views build on streams.</p>
<p><strong>Follow › TCP Stream</strong> strips away the packets and shows the bytes each side sent, in order, one color per
side. For plain HTTP that&rsquo;s the request and the response as text:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">GET / HTTP/1.1
</span></span><span class="line"><span class="cl">Host: web.lab
</span></span><span class="line"><span class="cl">User-Agent: curl/8.14.1
</span></span><span class="line"><span class="cl">Accept: */*
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">HTTP/1.1 200 OK
</span></span><span class="line"><span class="cl">Server: nginx/1.28.3
</span></span><span class="line"><span class="cl">Content-Type: text/html
</span></span><span class="line"><span class="cl">Content-Length: 33
</span></span><span class="line"><span class="cl">...
</span></span><span class="line"><span class="cl">Hello from web.lab, 192.168.2.10
</span></span></code></pre></div><p><strong>Statistics › Conversations › TCP</strong> lists every connection in the capture with its addresses, ports, frames,
bytes and duration. For this page: 12 frames and 1,148 bytes, seven frames from the laptop and five from web.
Sort it by bytes and the biggest transfers come to the top. Sort it by duration and the connections that hung
around do.</p>
<hr>
<h2 id="what-i-see-go-wrong">What I see go wrong</h2>
<p><strong>Starting the capture too late.</strong> Without the handshake, Wireshark doesn&rsquo;t know the window scale, the MSS or
the initial RTT, and its analysis is weaker. Start capturing before the connection opens.</p>
<p><strong>Counting packets instead of bytes.</strong> Seq and Ack count bytes. A missing packet shows up as a gap in the byte
numbers, and two segments of very different sizes can carry one HTTP message. Read the numbers, not the
number of lines.</p>
<p><strong>Calling every RST a failure.</strong> A RST answering a SYN is the clearest answer TCP has. It&rsquo;s silence that
should worry you.</p>
<p><strong>Assuming the client closed it.</strong> Whoever sent the first FIN (or the RST) ended the connection. When a
connection &ldquo;drops&rdquo;, check which side did it first. The answer often points straight at a timeout on the other
end.</p>
<hr>
<h2 id="try-it">Try it</h2>
<p><a href="/labs/wireshark-101/#chapter-6">Chapter 6 of the lab</a> has you capture a page load, find the handshake and
the FINs, then pick the refused connection out of three by its stream index.</p>
<h2 id="whats-next">What&rsquo;s next</h2>
<p>The same connection on port 443 starts with a handshake of its own, and most of it is still readable.
<a href="/lessons/wireshark-101/tls-without-decryption/">Lesson 7</a> is about what TLS shows without decrypting
anything, and how to decrypt your own when you need to. Back to <a href="/lessons/wireshark-101/dns/">lesson 5, DNS</a>.</p>
]]></content:encoded></item><item><title>SSH and Keys: Logging In Without a Password (Linux CLI 101, Lesson 6)</title><link>https://networkdoodles.com/lessons/linux-cli-101/ssh-and-keys/</link><pubDate>Mon, 16 Nov 2026 09:00:00 -0500</pubDate><dc:creator>David Henderson</dc:creator><guid>https://networkdoodles.com/lessons/linux-cli-101/ssh-and-keys/</guid><category>linux</category><category>networking-basics</category><category>network-automation</category><category>security</category><description>How SSH checks a server's host key and proves who you are: known_hosts, ed25519 key pairs, authorized_keys, ~/.ssh/config, scp, and a host key that changed.</description><content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[<p><strong>SSH</strong> (Secure Shell) gives you a shell on another machine over an encrypted connection. Two kinds of key
make it safe. The server&rsquo;s <strong>host key</strong> proves to you that you&rsquo;ve reached the right machine, and your
computer remembers it in <code>~/.ssh/known_hosts</code>. Your own <strong>key pair</strong> proves to the server who you are: the
<strong>private key</strong> never leaves your machine, and the <strong>public key</strong> goes into the server&rsquo;s
<code>~/.ssh/authorized_keys</code>. With both in place you log in with no password, and a line in <code>~/.ssh/config</code>
lets you do it by a short name.</p>
<p>You&rsquo;ve probably used SSH to network devices for years, with a password. Automation needs the key-based
version: Ansible logs in to every host over SSH, and Git pushes over it.</p>
<hr>
<h2 id="what-are-the-client-and-the-server">What are the client and the server?</h2>
<p>The <strong>SSH client</strong> is the program you run, <code>ssh</code>. The <strong>SSH server</strong> is a program on the other machine,
<code>sshd</code> (the <em>d</em> is for <em>daemon</em>, a program that runs in the background waiting for work), listening on TCP
port 22. Both come from the same project, <a href="https://www.openssh.org/">OpenSSH</a>, and the versions in this lab
differ, which matters below:</p>
<table>
  <thead>
      <tr>
          <th>Machine</th>
          <th>Role in the lab</th>
          <th><code>ssh -V</code> says</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>the codespace (Debian 12)</td>
          <td>client</td>
          <td><code>OpenSSH_9.2p1 Debian-2+deb12u10</code></td>
      </tr>
      <tr>
          <td>ws1 and srv1 (Debian 13)</td>
          <td>servers (they have the client too)</td>
          <td><code>OpenSSH_10.0p2 Debian-7+deb13u4</code></td>
      </tr>
      <tr>
          <td>r1 (CapeOS)</td>
          <td>a router with an SSH server and a password login</td>
          <td></td>
      </tr>
  </tbody>
</table>
<p>When you connect, the client and server agree on encryption, the server proves its identity with its host
key, and then you prove yours, with a key or a password.</p>
<hr>
<h2 id="what-is-a-host-key-and-whats-the-first-connection-question">What is a host key, and what&rsquo;s the first-connection question?</h2>
<p>A <strong>host key</strong> is a key pair that belongs to the server itself, made when SSH was installed. Its job is to
stop you typing your password into an impostor. The first time you connect, your client has never seen this
server&rsquo;s key, so it asks you. Here is that question from the codespace, connecting to srv1:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">The authenticity of host &#39;srv1 (172.80.102.12)&#39; can&#39;t be established.
</span></span><span class="line"><span class="cl">ED25519 key fingerprint is SHA256:D5T0vLnK5aqJqyXkl0nNrxpCcACGUqPLTgSGR0KO4Eo.
</span></span><span class="line"><span class="cl">This key is not known by any other names.
</span></span><span class="line"><span class="cl">Are you sure you want to continue connecting (yes/no/[fingerprint])?
</span></span></code></pre></div><p>A <strong>fingerprint</strong> is a short hash of the key, easy to compare by eye. The honest way to answer is to compare it
with the fingerprint taken on the server itself, through some other channel you already trust (a console, a
colleague, your build records). On srv1, <code>ssh-keygen -lf</code> reads its own public host key:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">256 SHA256:D5T0vLnK5aqJqyXkl0nNrxpCcACGUqPLTgSGR0KO4Eo root@srv1 (ED25519)
</span></span></code></pre></div><p>They match, so it&rsquo;s srv1. Answer <code>yes</code> and the client saves the key in <code>~/.ssh/known_hosts</code>. Every later
connection checks the server against that saved key and doesn&rsquo;t ask again. This is called <strong>trust on first
use</strong>: the first answer is the one that counts. You can also paste the fingerprint itself as the answer, and
the client checks it for you.</p>
<p><code>ssh-keyscan srv1</code> fetches a server&rsquo;s public host keys without logging in, which is how automation fills
<code>known_hosts</code> for many machines. It saves you typing; it doesn&rsquo;t prove anything, because it trusts whatever
answers.</p>
<h3 id="why-does-known_hosts-look-like-gibberish">Why does known_hosts look like gibberish?</h3>
<p>Because Debian&rsquo;s client settings, in <code>/etc/ssh/ssh_config</code>, include <code>HashKnownHosts yes</code>. The client stores a
hash of each host name instead of the name, so someone who reads your file can&rsquo;t get a list of every server
you use. A line looks like this:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">|1|V5D+TWLxnaBV6Ck3cnAQOfkYWZA=|qeCTHIXQjdcQrnEdhx7ml2NDR5g= ssh-ed25519 AAAAC3NzaC1lZDI1N...
</span></span></code></pre></div><p>So don&rsquo;t search the file with <code>grep srv1</code>: it won&rsquo;t find anything. <code>ssh-keygen -F srv1</code> hashes the name and
finds the line for you. The <a href="https://man.openbsd.org/ssh-keygen">ssh-keygen(1) manual page</a> covers it.</p>
<hr>
<h2 id="what-does-remote-host-identification-has-changed-mean">What does &ldquo;REMOTE HOST IDENTIFICATION HAS CHANGED&rdquo; mean?</h2>
<p>It means the server offered a different host key from the one in your <code>known_hosts</code>. To show it here, this
lesson gave the client a throwaway <code>known_hosts</code> file holding the wrong key for srv1:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
</span></span><span class="line"><span class="cl">@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
</span></span><span class="line"><span class="cl">@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
</span></span><span class="line"><span class="cl">IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
</span></span><span class="line"><span class="cl">Someone could be eavesdropping on you right now (man-in-the-middle attack)!
</span></span><span class="line"><span class="cl">It is also possible that a host key has just been changed.
</span></span><span class="line"><span class="cl">The fingerprint for the ED25519 key sent by the remote host is
</span></span><span class="line"><span class="cl">SHA256:D5T0vLnK5aqJqyXkl0nNrxpCcACGUqPLTgSGR0KO4Eo.
</span></span><span class="line"><span class="cl">...
</span></span><span class="line"><span class="cl">Host key verification failed.
</span></span></code></pre></div><p>The client refuses to connect at all. There are two explanations, and the message names both:</p>
<table>
  <thead>
      <tr>
          <th>Cause</th>
          <th>How often</th>
          <th>What to do</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>the server was rebuilt or reinstalled, so it made new host keys</td>
          <td>almost always</td>
          <td>confirm the new fingerprint with whoever rebuilt it, then replace the old line</td>
      </tr>
      <tr>
          <td>something between you and the server is pretending to be it (a <strong>man-in-the-middle</strong>)</td>
          <td>rarely, and that&rsquo;s the case it exists for</td>
          <td>don&rsquo;t connect; find out what&rsquo;s answering</td>
      </tr>
  </tbody>
</table>
<p>The wrong fix is to
delete the whole <code>known_hosts</code> file, or to switch checking off: both throw away the protection for every
server, not just this one. The right fix removes only the one stale entry, and the full message (trimmed
above) tells you which line it is and how. Then connect again and check the new fingerprint, as on first use.</p>
<hr>
<h2 id="how-does-a-key-pair-log-you-in">How does a key pair log you in?</h2>
<p>A <strong>key pair</strong> is two files made together. The <strong>private key</strong> can sign things, and must stay secret. The
<strong>public key</strong> can only check those signatures, so anyone may have it. When you log in, the server sends a
challenge, your client signs it with the private key, and the server checks the signature with the public key
it already has. Your private key never crosses the network, and there&rsquo;s no password to steal or guess.</p>
<p><code>ssh-keygen</code> makes the pair. Here it is in the codespace, writing to a throwaway path (<code>-N ''</code> means no
passphrase, used only for this demo):</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ ssh-keygen -t ed25519 -N &#39;&#39; -f /tmp/lesson6-demo/id_ed25519 -C demo@codespace
</span></span><span class="line"><span class="cl">Generating public/private ed25519 key pair.
</span></span><span class="line"><span class="cl">Your identification has been saved in /tmp/lesson6-demo/id_ed25519
</span></span><span class="line"><span class="cl">Your public key has been saved in /tmp/lesson6-demo/id_ed25519.pub
</span></span><span class="line"><span class="cl">The key fingerprint is:
</span></span><span class="line"><span class="cl">SHA256:Jc1YWRs4Z7TUlItc+zW2wdGVlMZ9wO3p3+qm02hmOHE demo@codespace
</span></span></code></pre></div><p>The two files, and the public key&rsquo;s one line:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">-rw------- 1 vscode vscode  411 Nov 11 05:55 id_ed25519
</span></span><span class="line"><span class="cl">-rw-r--r-- 1 vscode vscode   96 Nov 11 05:55 id_ed25519.pub
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIF9gi15jt5MNl7110z05ei2oPcxC5i0B7QzCMhXRQPQv demo@codespace
</span></span></code></pre></div><p>The private key is <code>600</code>, readable only by you, as <a href="/lessons/linux-cli-101/users-permissions-sudo/">lesson 4</a>
explained: SSH refuses a private key that others can read. The <code>.pub</code> file is one line: the key type, the key,
and a comment that&rsquo;s only a label.</p>
<p>To log in with it, the public key&rsquo;s line goes into <code>~/.ssh/authorized_keys</code> in your account on the server, one
line per key you trust. <code>ssh-copy-id</code> does that for you over a password login. On srv1 that file and its
folder are locked down as <code>sshd</code> expects:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">700 netops /home/netops/.ssh
</span></span><span class="line"><span class="cl">600 netops /home/netops/.ssh/authorized_keys
</span></span></code></pre></div><table>
  <thead>
      <tr>
          <th>File</th>
          <th>Where</th>
          <th>Holds</th>
          <th>Secret?</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>~/.ssh/id_ed25519</code></td>
          <td>your machine</td>
          <td>your private key</td>
          <td><strong>yes</strong>: never copy it, share it or commit it</td>
      </tr>
      <tr>
          <td><code>~/.ssh/id_ed25519.pub</code></td>
          <td>your machine</td>
          <td>your public key</td>
          <td>no</td>
      </tr>
      <tr>
          <td><code>~/.ssh/authorized_keys</code></td>
          <td>the server</td>
          <td>public keys allowed to log in as this user</td>
          <td>no, but only you may write it</td>
      </tr>
      <tr>
          <td><code>~/.ssh/known_hosts</code></td>
          <td>your machine</td>
          <td>servers&rsquo; public host keys</td>
          <td>no</td>
      </tr>
      <tr>
          <td><code>/etc/ssh/ssh_host_*_key</code></td>
          <td>the server</td>
          <td>the server&rsquo;s own private host keys</td>
          <td>yes</td>
      </tr>
  </tbody>
</table>
<p>A <strong>passphrase</strong> encrypts the private key file, so a stolen copy is useless without it. Use one on your own
laptop. The agent, below, means you type it once, not every time.</p>
<h3 id="why-does-the-lab-type--t-ed25519">Why does the lab type -t ed25519?</h3>
<p><strong>Ed25519</strong> is a modern key type: small, fast and strong. It&rsquo;s been ssh-keygen&rsquo;s default since OpenSSH 9.5,
whose <a href="https://www.openssh.org/txt/release-9.5">release notes</a> (2023-10-04) say &ldquo;generate Ed25519 keys by
default&rdquo;. The codespace runs OpenSSH 9.2, older than that, so there <code>ssh-keygen</code> with no <code>-t</code> still makes RSA:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ ssh-keygen -N &#39;&#39; -f /tmp/lesson6-demo/default_key -C demo@codespace
</span></span><span class="line"><span class="cl">Generating public/private rsa key pair.
</span></span><span class="line"><span class="cl">...
</span></span><span class="line"><span class="cl">3072 SHA256:PSXemktsnew7YRkQIkYNXcNul6piSYKqv4qJ9/jqx08 demo@codespace (RSA)
</span></span></code></pre></div><p>A 3072-bit RSA key still works. But <code>-t ed25519</code> gives you the same key type on every machine, whatever its
OpenSSH version, and it&rsquo;s the type the lab&rsquo;s checker looks for.</p>
<hr>
<h2 id="what-does-permission-denied-publickeypassword-mean">What does &ldquo;Permission denied (publickey,password)&rdquo; mean?</h2>
<p>The server turned you away, and the brackets list the login methods it would have accepted. Here the
codespace tries ws1 with the demo key, which ws1 has never been given, in <strong>batch mode</strong>
(<code>-o BatchMode=yes</code>: never stop to ask a question, fail instead, which is how scripts run ssh):</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">netops@ws1: Permission denied (publickey,password).
</span></span></code></pre></div><p>The key was offered and refused, and batch mode wasn&rsquo;t allowed to fall back to a password prompt. The same
words appear when the key isn&rsquo;t in <code>authorized_keys</code>, when you&rsquo;ve given the wrong user name, or when the
server ignores <code>authorized_keys</code> because its permissions are too open. <code>ssh -v</code> shows each key the client
offers and the server&rsquo;s answer.</p>
<hr>
<h2 id="how-do-i-log-in-by-a-short-name-sshconfig">How do I log in by a short name? ~/.ssh/config</h2>
<p><code>~/.ssh/config</code> is your client&rsquo;s settings file. Each <code>Host</code> block gives a name you can type, and the settings
to use for it. The options are in <a href="https://man.openbsd.org/ssh_config">ssh_config(5)</a>. Here&rsquo;s a block in a
throwaway file that makes <code>web</code> mean &ldquo;srv1, as netops, with this key&rdquo;:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">Host web
</span></span><span class="line"><span class="cl">    HostName srv1
</span></span><span class="line"><span class="cl">    User netops
</span></span><span class="line"><span class="cl">    IdentityFile /tmp/lesson6-demo/id_ed25519
</span></span></code></pre></div><p><code>ssh -G web</code> prints the settings the client would actually use, without connecting. It&rsquo;s the quickest way to
check a block does what you meant:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">user netops
</span></span><span class="line"><span class="cl">hostname srv1
</span></span><span class="line"><span class="cl">port 22
</span></span><span class="line"><span class="cl">identityfile /tmp/lesson6-demo/id_ed25519
</span></span></code></pre></div><table>
  <thead>
      <tr>
          <th>Keyword</th>
          <th>Means</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>Host</code></td>
          <td>the name (or names, or a pattern like <code>*.lab</code>) this block applies to</td>
      </tr>
      <tr>
          <td><code>HostName</code></td>
          <td>the real name or address to connect to, if different</td>
      </tr>
      <tr>
          <td><code>User</code></td>
          <td>the user to log in as, so you can drop <code>user@</code></td>
      </tr>
      <tr>
          <td><code>Port</code></td>
          <td>if it&rsquo;s not 22</td>
      </tr>
      <tr>
          <td><code>IdentityFile</code></td>
          <td>which private key to offer</td>
      </tr>
  </tbody>
</table>
<p>The client reads the file from the top and, for each setting, keeps the <strong>first</strong> value it finds. Put
specific hosts first and a catch-all <code>Host *</code> last. Every tool that uses OpenSSH reads this file, so an alias
works for <code>scp</code>, Git and Ansible too.</p>
<p>A fresh install has no <code>~/.ssh/config</code> at all. If yours already has blocks, they&rsquo;re left from earlier work.</p>
<hr>
<h2 id="how-do-i-copy-a-file-over-ssh-scp">How do I copy a file over SSH? scp</h2>
<p><code>scp</code> copies files over the same connection, with the same keys, <code>known_hosts</code> and aliases. It reads like
<code>cp</code>, with a host name and a colon in front of a remote path. A path after the colon is relative to your home
folder on the other side, and nothing after the colon means the home folder itself. Forget the colon and <code>scp</code>
makes a local copy with a file name that looks like a host.</p>
<hr>
<h2 id="what-does-the-ssh-agent-do">What does the SSH agent do?</h2>
<p>The <strong>agent</strong>, <code>ssh-agent</code>, is a small background process (a process like the ones in
<a href="/lessons/linux-cli-101/processes-environment-path/">lesson 5</a>) that holds your unlocked private keys in memory.
You add a key once with <code>ssh-add</code>, typing its passphrase once, and every <code>ssh</code> after that asks the agent to
sign for it. The client finds the agent through an environment variable, <code>SSH_AUTH_SOCK</code>, so only shells that have
that variable can use it. Here&rsquo;s the demo key added to an agent and listed:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">Identity added: /tmp/lesson6-demo/id_ed25519 (demo@codespace)
</span></span><span class="line"><span class="cl">256 SHA256:Jc1YWRs4Z7TUlItc+zW2wdGVlMZ9wO3p3+qm02hmOHE demo@codespace (ED25519)
</span></span></code></pre></div><p><code>ssh-add -l</code> shows fingerprints, never the keys. The agent is what makes a passphrase bearable, and what lets
Ansible open fifty connections without fifty prompts.</p>
<hr>
<h2 id="why-do-keys-beat-passwords-for-automation">Why do keys beat passwords for automation?</h2>
<table>
  <thead>
      <tr>
          <th></th>
          <th>Password</th>
          <th>Key</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>a script can use it without a human</td>
          <td>only by storing the password somewhere</td>
          <td>yes, through the agent or an unencrypted key kept for that job</td>
      </tr>
      <tr>
          <td>what crosses the network</td>
          <td>the password itself, encrypted</td>
          <td>a signature; the private key never leaves</td>
      </tr>
      <tr>
          <td>guessable</td>
          <td>yes, and servers get guessed at all day</td>
          <td>no</td>
      </tr>
      <tr>
          <td>revoke one machine&rsquo;s access</td>
          <td>change the password everywhere it&rsquo;s used</td>
          <td>delete one line from <code>authorized_keys</code></td>
      </tr>
      <tr>
          <td>one per person or job</td>
          <td>rarely, in practice</td>
          <td>easily: one key each, each with its own comment</td>
      </tr>
  </tbody>
</table>
<p>Servers defend themselves against guessing too. Since <a href="https://www.openssh.org/txt/release-9.8">OpenSSH 9.8</a>
(2024-07-01), <code>sshd</code> penalises client addresses that don&rsquo;t complete a login, through a <code>PerSourcePenalties</code>
setting that&rsquo;s &ldquo;on by default&rdquo;. srv1&rsquo;s effective settings show it:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">persourcepenalties crash:90 authfail:5 noauth:1 grace-exceeded:10 refuseconnection:10 max:600 min:15 ...
</span></span></code></pre></div><p>A script that keeps retrying a wrong password can get its own address refused for a while. With keys, that
doesn&rsquo;t happen.</p>
<p>That&rsquo;s why <a href="/lessons/ansible-101/">Ansible 101</a> logs in to Linux hosts with keys, why
<span class="link-soon">GitOps</span> pushes with one, and why Python 101&rsquo;s
<a href="/lessons/python-101/netmiko/">Netmiko lesson</a> keeps device passwords in environment variables rather than in
code. Many network devices, like r1 here, still use passwords, so those tools also keep passwords out of your files.</p>
<hr>
<h2 id="what-i-see-go-wrong">What I see go wrong</h2>
<p><strong>A private key others can read.</strong> The client won&rsquo;t use it, and says so in capitals. <code>chmod 600</code>, from
lesson 4.</p>
<p><strong>A key that&rsquo;s in authorized_keys and still refused.</strong> The server ignores the file if <code>~/.ssh</code> or
<code>authorized_keys</code> can be written by anyone but you. Or the key went into the wrong user&rsquo;s home folder.</p>
<p><strong>Wrong user.</strong> Without <code>user@</code> or a <code>User</code> line, ssh uses your local user name (<code>vscode</code> in the codespace),
which doesn&rsquo;t exist on the server.</p>
<p><strong>Deleting all of known_hosts after one rebuild.</strong> It works, and it throws away every other server&rsquo;s key with
it. Remove only the stale entry.</p>
<p><strong><code>StrictHostKeyChecking no</code> in a config &ldquo;to make it work&rdquo;.</strong> It turns off the one check that tells you you&rsquo;ve
reached the right machine. Scan and verify the keys instead.</p>
<p><strong>The private key in Git.</strong> Once pushed, assume it&rsquo;s public: remove its line from every <code>authorized_keys</code> and
make a new pair. <a href="/posts/donthardcodepasswords/">Don&rsquo;t hardcode passwords</a> applies to keys too.</p>
<p><strong>Copying the wrong file.</strong> <code>authorized_keys</code> takes the <code>.pub</code> file&rsquo;s line. If you&rsquo;re about to paste something
that starts with <code>-----BEGIN OPENSSH PRIVATE KEY-----</code>, stop.</p>
<hr>
<h2 id="try-it">Try it</h2>
<p><a href="/labs/linux-cli-101/task-6-1/">Chapter 6 of the lab</a> has you log in to ws1 with a password, check and accept
its host key, make an ed25519 key pair and log in again without a password. Then you give ws1, srv1 and r1
short names, copy a file to srv1, and meet &ldquo;host key changed&rdquo; for real when srv1 is rebuilt.</p>
<h2 id="whats-next">What&rsquo;s next</h2>
<p>Typing the same SSH command for every host gets old fast. <a href="/lessons/linux-cli-101/bash-scripting-basics/">Lesson 7</a>
turns commands you&rsquo;d type into a script: a loop over a list of hosts, and an exit code that says whether they
all answered.</p>
]]></content:encoded></item><item><title>Processes, Environment Variables and PATH: Finding What Runs (Linux CLI 101, Lesson 5)</title><link>https://networkdoodles.com/lessons/linux-cli-101/processes-environment-path/</link><pubDate>Mon, 16 Nov 2026 09:00:00 -0500</pubDate><dc:creator>David Henderson</dc:creator><guid>https://networkdoodles.com/lessons/linux-cli-101/processes-environment-path/</guid><category>linux</category><category>networking-basics</category><category>network-automation</category><category>troubleshooting</category><description>Processes and PIDs, background jobs, ps and top, what TERM and KILL do, how exported environment variables reach child processes, and how PATH finds a command.</description><content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[<p>A <strong>process</strong> is a program that&rsquo;s running, and Linux gives every one a number, its <strong>PID</strong> (process ID). You
find processes with <code>ps</code> and <code>top</code>, and you stop one by sending it a <strong>signal</strong>: <code>kill</code> sends TERM, a polite
request to stop, and <code>kill -9</code> sends KILL, which no process can refuse. Each process also carries a set of
<strong>environment variables</strong>, named values it hands down to the processes it starts. One of them, <code>PATH</code>, is the
list of folders the shell searches when you type a command name.</p>
<p>Those three ideas explain why a script can&rsquo;t see a variable you just set, why a tool you installed says
&ldquo;command not found&rdquo;, and what to do when something is eating a CPU. They come back in every later course:
Ansible reads settings from environment variables, and Python&rsquo;s venv works by changing <code>PATH</code>.</p>
<hr>
<h2 id="what-is-a-process-and-whats-a-pid">What is a process, and what&rsquo;s a PID?</h2>
<p>When you type <code>sleep 300</code> and press <kbd>Enter</kbd>, the shell asks the Linux kernel to start the <code>sleep</code>
program. The kernel loads it, gives it a PID and some memory, and lets it run. That running copy is a
process. Run <code>sleep</code> twice and you have two processes of one program, each with its own PID.</p>
<p>Every process except the very first has a <strong>parent</strong>: the process that started it. Your shell is the parent
of every command you type. Here&rsquo;s the codespace&rsquo;s shell and a <code>sleep</code> it started in the background, as <code>ps</code>
shows them:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">  PID  PPID STAT     ELAPSED COMMAND
</span></span><span class="line"><span class="cl">35250     0 Ss         00:00 bash
</span></span><span class="line"><span class="cl">35256 35250 S          00:00 sleep
</span></span></code></pre></div><table>
  <thead>
      <tr>
          <th>Column</th>
          <th>Means</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>PID</code></td>
          <td>this process&rsquo;s number</td>
      </tr>
      <tr>
          <td><code>PPID</code></td>
          <td>its parent&rsquo;s PID: <code>sleep</code>&rsquo;s parent, 35250, is the <code>bash</code> above it (that <code>bash</code> shows 0 because it was started from outside the container)</td>
      </tr>
      <tr>
          <td><code>STAT</code></td>
          <td>its state: <code>R</code> running, <code>S</code> sleeping (waiting for something), <code>T</code> stopped, <code>Z</code> a zombie (finished, but its parent hasn&rsquo;t collected it yet)</td>
      </tr>
      <tr>
          <td><code>ELAPSED</code></td>
          <td>how long it has been running</td>
      </tr>
      <tr>
          <td><code>COMMAND</code></td>
          <td>the program</td>
      </tr>
  </tbody>
</table>
<p>Your PIDs will be different, and a PID is reused after its process ends, so look one up just before you act on it.</p>
<hr>
<h2 id="whats-the-difference-between-foreground-and-background">What&rsquo;s the difference between foreground and background?</h2>
<p>A command runs in the <strong>foreground</strong> by default: the shell waits for it to finish before it gives you the
prompt back. That&rsquo;s what you want for <code>ls</code>. It&rsquo;s not what you want for a capture or a long copy that takes ten
minutes.</p>
<p>Put <code>&amp;</code> at the end of a command and it runs in the <strong>background</strong>: the shell starts it, prints its job number
and PID, and gives you the prompt straight away. The shell&rsquo;s own list of what it started is called its
<strong>jobs</strong>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ sleep 300 &amp;
</span></span><span class="line"><span class="cl">[1] 36465
</span></span><span class="line"><span class="cl">$ jobs -l
</span></span><span class="line"><span class="cl">[1]+ 36465 Running                 sleep 300 &amp;
</span></span></code></pre></div><p><code>[1]</code> is the job number, which only your shell knows. <code>36465</code> is the PID, which the whole system knows.
<code>kill %1</code> stops a job by its number; <code>kill 36465</code> does the same by PID.</p>
<table>
  <thead>
      <tr>
          <th>You want to</th>
          <th>Do this</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>start something in the background</td>
          <td>end the command with <code>&amp;</code></td>
      </tr>
      <tr>
          <td>pause the foreground command</td>
          <td><kbd>Ctrl</kbd>+<kbd>Z</kbd></td>
      </tr>
      <tr>
          <td>list this shell&rsquo;s jobs</td>
          <td><code>jobs</code> (<code>jobs -l</code> adds PIDs)</td>
      </tr>
      <tr>
          <td>bring a job back to the foreground</td>
          <td><code>fg %1</code></td>
      </tr>
      <tr>
          <td>let a paused job carry on in the background</td>
          <td><code>bg %1</code></td>
      </tr>
      <tr>
          <td>stop the foreground command</td>
          <td><kbd>Ctrl</kbd>+<kbd>C</kbd></td>
      </tr>
  </tbody>
</table>
<p><kbd>Ctrl</kbd>+<kbd>Z</kbd> and <kbd>Ctrl</kbd>+<kbd>C</kbd> look alike and do very different things.
<kbd>Ctrl</kbd>+<kbd>Z</kbd> <strong>pauses</strong>: the process is still there, holding its memory, doing nothing, until
you <code>fg</code> or <code>bg</code> it. <kbd>Ctrl</kbd>+<kbd>C</kbd> asks it to <strong>stop</strong>. Background jobs belong to the shell that started them. Close that terminal and they normally go with it.</p>
<hr>
<h2 id="how-do-i-find-the-process-thats-eating-a-cpu">How do I find the process that&rsquo;s eating a CPU?</h2>
<p>Ask for every process, sorted by CPU use. <code>ps -e</code> lists every process on the system, not just yours, and
<code>--sort=-pcpu</code> puts the busiest first:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ ps -eo pid,user,pcpu,pmem,comm --sort=-pcpu | head -6
</span></span><span class="line"><span class="cl">  PID USER     %CPU %MEM COMMAND
</span></span><span class="line"><span class="cl">33433 vscode    200  0.0 ps
</span></span><span class="line"><span class="cl">   26 root     28.5  1.4 dockerd
</span></span><span class="line"><span class="cl">33423 vscode    2.6  0.0 bash
</span></span><span class="line"><span class="cl">26452 root      2.5  0.3 capeosd
</span></span><span class="line"><span class="cl">   37 root      2.0  0.7 containerd
</span></span></code></pre></div><p><code>%CPU</code> is per core, so it can pass 100 on a machine with several cores. And <code>ps</code> averages it over the
process&rsquo;s lifetime, so a command that has only just started (<code>ps</code> itself, here) can look wildly busy. For
&ldquo;what&rsquo;s busy right now&rdquo;, use <code>top</code>.</p>
<p><code>top</code> is a live view that refreshes every few seconds and sorts by current CPU use. Here&rsquo;s one screen of it from
the codespace, taken with <code>top -b -n1</code> (batch mode, one round, so it can be shown here):</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">top - 05:56:30 up 3 days,  7:01,  0 user,  load average: 12.30, 7.90, 7.51
</span></span><span class="line"><span class="cl">Tasks:  46 total,   1 running,  24 sleeping,   0 stopped,  21 zombie
</span></span><span class="line"><span class="cl">%Cpu(s):  5.1 us, 43.6 sy,  0.0 ni, 48.7 id,  2.6 wa,  0.0 hi,  0.0 si,  0.0 st
</span></span><span class="line"><span class="cl">MiB Mem :   7837.1 total,    242.9 free,   1890.9 used,   5971.7 buff/cache
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
</span></span><span class="line"><span class="cl">26452 root      20   0   35820  28584   9012 S   6.2   0.4   0:07.17 capeosd
</span></span><span class="line"><span class="cl">33471 vscode    20   0   12064   4784   2736 R   6.2   0.1   0:00.03 top
</span></span></code></pre></div><p>The <strong>load average</strong> is how many processes, on average, were running or waiting to run over the last 1, 5
and 15 minutes. Compare it with the number of cores (<code>nproc</code>): a load of 2 on a 2-core codespace means it&rsquo;s
fully busy. In the list, <code>PID</code>, <code>%CPU</code>, <code>TIME+</code> (total CPU time used) and <code>COMMAND</code> are the columns that find a
runaway.</p>
<hr>
<h2 id="what-does-kill-actually-do-term-vs-kill">What does kill actually do? TERM vs KILL</h2>
<p><code>kill</code> doesn&rsquo;t kill. It sends a <strong>signal</strong>: a small numbered message from the kernel to a process. The
process can have a <strong>handler</strong> for most signals, a piece of its own code that runs when the signal arrives,
or it can ignore them. The <a href="https://man7.org/linux/man-pages/man7/signal.7.html">signal(7) manual page</a> lists
them all. These are the ones you&rsquo;ll use:</p>
<table>
  <thead>
      <tr>
          <th>Signal</th>
          <th>Number</th>
          <th>Sent by</th>
          <th>What it asks</th>
          <th>Can it be ignored?</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>TERM</code></td>
          <td>15</td>
          <td><code>kill PID</code> (the default)</td>
          <td>please finish up and exit</td>
          <td>yes</td>
      </tr>
      <tr>
          <td><code>INT</code></td>
          <td>2</td>
          <td><kbd>Ctrl</kbd>+<kbd>C</kbd></td>
          <td>interrupt what you&rsquo;re doing</td>
          <td>yes</td>
      </tr>
      <tr>
          <td><code>HUP</code></td>
          <td>1</td>
          <td>the terminal closing</td>
          <td>your terminal has gone</td>
          <td>yes</td>
      </tr>
      <tr>
          <td><code>KILL</code></td>
          <td>9</td>
          <td><code>kill -9 PID</code> or <code>kill -KILL PID</code></td>
          <td>nothing: the kernel ends it</td>
          <td><strong>no</strong></td>
      </tr>
      <tr>
          <td><code>STOP</code></td>
          <td>19</td>
          <td><code>kill -STOP PID</code></td>
          <td>pause, like <kbd>Ctrl</kbd>+<kbd>Z</kbd></td>
          <td><strong>no</strong></td>
      </tr>
  </tbody>
</table>
<p>Send TERM first, because it lets a program clean up: close its files, remove its lock file, write its last
log line. KILL gives it no chance. The kernel just removes it, mid-write if that&rsquo;s where it was. signal(7) puts it plainly: SIGKILL and SIGSTOP &ldquo;cannot be
caught, blocked, or ignored&rdquo;.</p>
<p>Here&rsquo;s a small script, written for this lesson, that spins one CPU and ignores TERM. It&rsquo;s started in the
background, sent TERM, then sent KILL:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ ps -o pid,stat,pcpu,etime,cmd -p 33680
</span></span><span class="line"><span class="cl">  PID STAT %CPU     ELAPSED CMD
</span></span><span class="line"><span class="cl">33680 R     100       00:02 bash /tmp/lesson5-stubborn
</span></span><span class="line"><span class="cl">$ kill 33680
</span></span><span class="line"><span class="cl">$ ps -o pid,stat,pcpu,cmd -p 33680
</span></span><span class="line"><span class="cl">  PID STAT %CPU CMD
</span></span><span class="line"><span class="cl">33680 R    99.0 bash /tmp/lesson5-stubborn
</span></span><span class="line"><span class="cl">$ kill -9 33680
</span></span><span class="line"><span class="cl">bash: line 6: 33680 Killed                  /tmp/lesson5-stubborn
</span></span><span class="line"><span class="cl">$ ps -p 33680 -o pid,cmd
</span></span><span class="line"><span class="cl">  PID CMD
</span></span></code></pre></div><p>After TERM it&rsquo;s still there, still at 99% CPU. <code>kill</code> printed nothing, because sending the signal worked; the
process simply chose to ignore it. After KILL it&rsquo;s gone, and <code>ps</code> finds nothing. The shell reported its exit
status as 137, which is 128 plus 9: &ldquo;ended by signal 9&rdquo;. You&rsquo;ll meet exit codes properly in
<a href="/lessons/linux-cli-101/bash-scripting-basics/">lesson 7</a>.</p>
<p>The habit is: <strong>TERM, wait a few seconds, check, and only then KILL.</strong> Reaching for <code>kill -9</code> first is how you
end up with half-written files and lock files that stop the program starting again.</p>
<hr>
<h2 id="what-is-an-environment-variable">What is an environment variable?</h2>
<p>An <strong>environment variable</strong> is a named value, like <code>HOME=/home/vscode</code>, that a process carries with it and
that programs can read. <code>env</code> lists the ones your shell has; <code>echo &quot;$HOME&quot;</code> prints one. They tell a program something without changing its code or files.</p>
<p>The shell also has plain <strong>shell variables</strong>, which stay inside the shell. The difference is the whole point
of <code>export</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ GREETING=hi
</span></span><span class="line"><span class="cl">$ bash -c &#39;echo child sees: [$GREETING]&#39;
</span></span><span class="line"><span class="cl">child sees: []
</span></span><span class="line"><span class="cl">$ export GREETING
</span></span><span class="line"><span class="cl">$ bash -c &#39;echo child sees: [$GREETING]&#39;
</span></span><span class="line"><span class="cl">child sees: [hi]
</span></span></code></pre></div><p><code>GREETING=hi</code> set a shell variable. The second <code>bash</code>, a <strong>child process</strong> started by the first, couldn&rsquo;t see
it. After <code>export</code>, the shell put it in its environment, and every child it starts from then on gets a copy.</p>
<p>Three rules follow, and they explain most &ldquo;but I set it!&rdquo; problems:</p>
<ul>
<li><strong>Children get a copy, not a link.</strong> A child that changes the variable changes only its own copy. Nothing a
script does to its environment comes back to the shell that ran it.</li>
<li><strong>It goes down, never sideways.</strong> A variable exported in one terminal tab doesn&rsquo;t exist in another. Each
terminal runs its own shell.</li>
<li><strong>It lasts as long as the shell.</strong> Close the terminal and it&rsquo;s gone. To have it in every new terminal, put
the <code>export</code> line in a startup file (below).</li>
</ul>
<h3 id="why-do-secrets-go-in-environment-variables">Why do secrets go in environment variables?</h3>
<p>Because a file you commit to Git is shared with everyone who can read the repository, forever, history
included. A password in an environment variable lives only in the memory of the processes that need it. That&rsquo;s
why Python 101 has Netmiko read credentials from the environment, and why
<a href="/posts/donthardcodepasswords/">Don&rsquo;t hardcode passwords</a> and Ansible 101&rsquo;s
<a href="/lessons/ansible-101/secrets/">lesson on secrets</a> take the same line. They aren&rsquo;t a vault (any process running as you can read them), but they&rsquo;re far better than a password in code.</p>
<hr>
<h2 id="how-does-the-shell-find-a-command-path">How does the shell find a command? PATH</h2>
<p>When you type <code>ssh</code>, the shell needs a file to run. It doesn&rsquo;t search the whole disk. It reads the
environment variable <strong><code>PATH</code></strong>, a list of folders separated by colons, and looks in each in turn. The
first folder holding an executable file called <code>ssh</code> wins. Here&rsquo;s the codespace&rsquo;s <code>PATH</code>, one folder per line:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ echo &#34;$PATH&#34; | tr &#34;:&#34; &#34;\n&#34;
</span></span><span class="line"><span class="cl">/home/vscode/.venvs/lab/bin
</span></span><span class="line"><span class="cl">/home/vscode/.local/bin
</span></span><span class="line"><span class="cl">/home/vscode/.atuin/bin
</span></span><span class="line"><span class="cl">/usr/local/python/current/bin
</span></span><span class="line"><span class="cl">/usr/local/py-utils/bin
</span></span><span class="line"><span class="cl">/usr/local/jupyter
</span></span><span class="line"><span class="cl">/usr/local/share/nvm/current/bin
</span></span><span class="line"><span class="cl">/usr/local/bin
</span></span><span class="line"><span class="cl">/usr/local/sbin
</span></span><span class="line"><span class="cl">/usr/local/bin
</span></span><span class="line"><span class="cl">/usr/sbin
</span></span><span class="line"><span class="cl">/usr/bin
</span></span><span class="line"><span class="cl">/sbin
</span></span><span class="line"><span class="cl">/bin
</span></span></code></pre></div><p>Order matters. The lab&rsquo;s Python venv is first, which is how activating a venv works: it puts its own <code>bin</code>
folder in front, so its <code>python</code> is found before the system&rsquo;s. (Python 101 picks this up.)</p>
<p>To ask which file a name resolves to, use <code>type</code>, which knows everything the shell knows, or <code>which</code>, which
only searches <code>PATH</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ type cd ls ssh
</span></span><span class="line"><span class="cl">cd is a shell builtin
</span></span><span class="line"><span class="cl">ls is /usr/bin/ls
</span></span><span class="line"><span class="cl">ssh is /usr/bin/ssh
</span></span></code></pre></div><p><code>cd</code> isn&rsquo;t a file at all. It&rsquo;s a <strong>builtin</strong>, part of Bash itself, because changing the shell&rsquo;s folder is
something only the shell can do to itself.</p>
<p>A command that isn&rsquo;t in any <code>PATH</code> folder isn&rsquo;t found, even if the file is sitting right there. Here&rsquo;s a
two-line script, <code>hello5</code>, in a folder that&rsquo;s not on <code>PATH</code>, then the same name after that folder is added
to the front:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ hello5
</span></span><span class="line"><span class="cl">bash: line 1: hello5: command not found
</span></span><span class="line"><span class="cl">$ PATH=&#34;/tmp/lesson5-tools:$PATH&#34;
</span></span><span class="line"><span class="cl">$ hello5
</span></span><span class="line"><span class="cl">hello from my own command
</span></span><span class="line"><span class="cl">$ type hello5
</span></span><span class="line"><span class="cl">hello5 is hashed (/tmp/lesson5-tools/hello5)
</span></span></code></pre></div><p>&ldquo;Hashed&rdquo; means Bash remembered where it found it. A name with a <code>/</code> in it, such as <code>./hello5</code>, skips the
<code>PATH</code> search entirely: that&rsquo;s why you type <code>./</code> to run a script in the current folder.</p>
<p>Exit code 127 is the shell&rsquo;s code for &ldquo;command not found&rdquo;. When a script fails with 127, check <code>PATH</code> first.</p>
<hr>
<h2 id="where-do-settings-for-every-new-terminal-go-bashrc">Where do settings for every new terminal go? ~/.bashrc</h2>
<p>A variable you set or a <code>PATH</code> you change lasts until the terminal closes. To have it in every new terminal,
add the line to a startup file that Bash reads when it starts:</p>
<table>
  <thead>
      <tr>
          <th>File</th>
          <th>Read by</th>
          <th>Use it for</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>~/.bashrc</code></td>
          <td>every new interactive shell, such as each terminal tab in the codespace</td>
          <td><code>export</code> lines, <code>PATH</code> changes, aliases</td>
      </tr>
      <tr>
          <td><code>~/.profile</code></td>
          <td>login shells, such as when you log in over SSH</td>
          <td>settings for a login; on Debian it also reads <code>~/.bashrc</code> when Bash is the shell</td>
      </tr>
  </tbody>
</table>
<p>The <a href="https://www.gnu.org/software/bash/manual/html_node/Bash-Startup-Files.html">GNU Bash manual&rsquo;s section on startup files</a>
has the exact rules. For this course, <code>~/.bashrc</code> is the one you edit.</p>
<p>Debian&rsquo;s default <code>~/.profile</code> already adds <code>~/bin</code> to <code>PATH</code> for login shells, if that folder exists. The lab
uses a different folder, <code>~/tools</code>, so you add it yourself and see how it works.</p>
<p>Editing <code>~/.bashrc</code> changes nothing in the terminal you already have open: it&rsquo;s only read when a shell starts.
Open a new terminal, or run <code>source ~/.bashrc</code> to read it into the current one. The checker always tests a new terminal.</p>
<hr>
<h2 id="what-i-see-go-wrong">What I see go wrong</h2>
<p><strong>&ldquo;I set it, but the script can&rsquo;t see it.&rdquo;</strong> It&rsquo;s a shell variable, not exported. Use <code>export NAME=value</code>.</p>
<p><strong>&ldquo;I exported it, but the other tab can&rsquo;t see it.&rdquo;</strong> Each terminal is its own shell. Put it in <code>~/.bashrc</code>
and open a new terminal.</p>
<p><strong>&ldquo;I added it to ~/.bashrc and nothing changed.&rdquo;</strong> The file is read when a shell starts. Open a new terminal.</p>
<p><strong>&ldquo;command not found&rdquo;, but the file is right there.</strong> Its folder isn&rsquo;t on <code>PATH</code>, or the file isn&rsquo;t
executable (<code>chmod +x</code>, from <a href="/lessons/linux-cli-101/users-permissions-sudo/">lesson 4</a>). <code>type name</code> tells
you what the shell would run.</p>
<p><strong>The wrong version runs.</strong> Two folders on <code>PATH</code> have a file of that name, and the earlier one wins.
<code>type -a name</code> lists every match, in order.</p>
<p><strong>A broken PATH.</strong> <code>PATH=/home/me/tools</code> without <code>:$PATH</code> replaces the whole list, and suddenly <code>ls</code> is &ldquo;not
found&rdquo;. Always add to it: <code>PATH=&quot;$HOME/tools:$PATH&quot;</code>. If you&rsquo;ve done it in a terminal, close that terminal.
If you&rsquo;ve done it in <code>~/.bashrc</code>, fix the file with its full path, such as <code>/usr/bin/nano ~/.bashrc</code>.</p>
<p><strong><code>kill -9</code> first.</strong> It works, and it skips the program&rsquo;s clean-up. Send TERM, wait, check with <code>ps</code>, then
KILL.</p>
<hr>
<h2 id="try-it">Try it</h2>
<p><a href="/labs/linux-cli-101/task-5-1/">Chapter 5 of the lab</a> has you start and pause jobs, read <code>ps</code> and <code>top</code>, set
a variable that every new terminal and the programs it starts can see, then find a runaway process that
doesn&rsquo;t stop when it&rsquo;s asked, and make a command of your own runnable from any folder.</p>
<h2 id="whats-next">What&rsquo;s next</h2>
<p>So far everything has run on one machine. <a href="/lessons/linux-cli-101/ssh-and-keys/">Lesson 6</a> is about getting a
shell on another one with SSH: host keys, key pairs, and logging in by a short name with no password.</p>
]]></content:encoded></item><item><title>DNS on the Wire: Queries, Answers and the Lookups You Didn't Type (Wireshark 101, Lesson 5)</title><link>https://networkdoodles.com/lessons/wireshark-101/dns/</link><pubDate>Mon, 16 Nov 2026 09:00:00 -0500</pubDate><dc:creator>David Henderson</dc:creator><guid>https://networkdoodles.com/lessons/wireshark-101/dns/</guid><category>wireshark</category><category>packet-capture</category><category>networking-basics</category><category>dns</category><description>DNS queries and responses in a capture: transaction IDs, flags, NXDOMAIN vs REFUSED, A, AAAA and CNAME records, TTLs, and the search domain's lookups.</description><content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[<p>A DNS lookup on the wire is usually two packets: a <strong>query</strong> sent over UDP to port 53, and a <strong>response</strong> that
copies the question back and adds the answers. A 16-bit <strong>transaction ID</strong> ties the two together. Everything
else you need to read a lookup is in a handful of fields: the flags, the reply code, the records and their
TTLs. And the most useful thing a capture shows you about DNS is often what the computer asked for that you
never typed.</p>
<p><a href="/lessons/networking-101/">Networking 101</a> described DNS as the network&rsquo;s phone book. This lesson opens the
phone book&rsquo;s envelope. Every example comes from the
<a href="/labs/wireshark-101/">Wireshark 101 lab</a>, where a laptop looks up names from a small DNS server two routers
away.</p>
<hr>
<h2 id="what-does-a-dns-query-look-like-in-wireshark">What does a DNS query look like in Wireshark?</h2>
<p>Here are three lookups from the lab: <code>dig</code> asking for <code>web.lab</code>, <code>intranet.lab</code> and <code>nas.lab</code>.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">4 1.186582542 192.168.1.175 → 192.168.2.53 DNS 90 Standard query 0x6aec A web.lab OPT
</span></span><span class="line"><span class="cl">5 1.190101167 192.168.2.53 → 192.168.1.175 DNS 94 Standard query response 0x6aec A web.lab A 192.168.2.10 OPT
</span></span><span class="line"><span class="cl">6 1.240939542 192.168.1.175 → 192.168.2.53 DNS 95 Standard query 0xc2f2 A intranet.lab OPT
</span></span><span class="line"><span class="cl">7 1.241206167 192.168.2.53 → 192.168.1.175 DNS 120 Standard query response 0xc2f2 A intranet.lab CNAME web.lab A 192.168.2.10 OPT
</span></span><span class="line"><span class="cl">8 1.302522250 192.168.1.175 → 192.168.2.53 DNS 90 Standard query 0x822d A nas.lab OPT
</span></span><span class="line"><span class="cl">9 1.303036709 192.168.2.53 → 192.168.1.175 DNS 78 Standard query response 0x822d No such name A nas.lab OPT
</span></span></code></pre></div><p>Six packets, three conversations. Each line tells you most of the story before you open anything: the
transaction ID (<code>0x6aec</code>), the record type asked for (<code>A</code>), the name, and in the response, the answer or the
reason there isn&rsquo;t one.</p>
<h3 id="why-dns-uses-udp-and-what-the-transaction-id-is-for">Why DNS uses UDP, and what the transaction ID is for</h3>
<p><strong>UDP</strong> (User Datagram Protocol) sends one packet and forgets about it. There&rsquo;s no handshake and no
connection, which is exactly what you want for a question that fits in one packet: a lookup costs one round
trip, not three. The price is that UDP doesn&rsquo;t match answers to questions for you.</p>
<p>That&rsquo;s the job of the <strong>transaction ID</strong>, a 16-bit number the client picks at random for each query. The server
copies it into the response, and the client accepts only a response whose ID (and ports) match a question it
asked. <a href="https://www.rfc-editor.org/rfc/rfc1035">RFC 1035</a> defines it. The randomness matters: a predictable ID
makes it easier for someone else to slip in a forged answer first.</p>
<p>Wireshark does the matching for you. Open a response and you&rsquo;ll find <code>[Request In: 4]</code> and
<code>[Time: 3.518625 milliseconds]</code>. That time is <code>dns.time</code>, the gap between question and answer, and it&rsquo;s the
quickest way to find a slow DNS server: filter <code>dns.time &gt; 0.1</code> to see every answer that took longer than 100
ms. In the lab, the first answer took 3.5 ms and the next two under 0.6 ms. A capture on the laptop&rsquo;s cable
can&rsquo;t tell you why the first one was slower. It can only tell you that it was, which is where every
investigation starts.</p>
<hr>
<h2 id="what-do-the-dns-flags-mean">What do the DNS flags mean?</h2>
<p>Every DNS message has a 16-bit flags field. Wireshark shows it as one hex number and then breaks it out bit by
bit. Here&rsquo;s the answer for <code>web.lab</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">Flags: 0x8580 Standard query response, No error
</span></span><span class="line"><span class="cl">    1... .... .... .... = Response: Message is a response
</span></span><span class="line"><span class="cl">    .000 0... .... .... = Opcode: Standard query (0)
</span></span><span class="line"><span class="cl">    .... .1.. .... .... = Authoritative: Server is an authority for domain
</span></span><span class="line"><span class="cl">    .... ..0. .... .... = Truncated: Message is not truncated
</span></span><span class="line"><span class="cl">    .... ...1 .... .... = Recursion desired: Do query recursively
</span></span><span class="line"><span class="cl">    .... .... 1... .... = Recursion available: Server can do recursive queries
</span></span><span class="line"><span class="cl">    .... .... .... 0000 = Reply code: No error (0)
</span></span></code></pre></div><table>
  <thead>
      <tr>
          <th>Flag</th>
          <th>Set by</th>
          <th>Means</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><strong>QR</strong> (Response)</td>
          <td>both</td>
          <td>0 in a query, 1 in a response. Filter <code>dns.flags.response == 0</code> for questions only</td>
      </tr>
      <tr>
          <td><strong>AA</strong> (Authoritative)</td>
          <td>server</td>
          <td>&ldquo;I own this name. This answer comes from my own data, not from a cache&rdquo;</td>
      </tr>
      <tr>
          <td><strong>TC</strong> (Truncated)</td>
          <td>server</td>
          <td>&ldquo;the full answer didn&rsquo;t fit; ask again over TCP&rdquo;</td>
      </tr>
      <tr>
          <td><strong>RD</strong> (Recursion desired)</td>
          <td>client</td>
          <td>&ldquo;if you don&rsquo;t know, go and find out for me&rdquo;</td>
      </tr>
      <tr>
          <td><strong>RA</strong> (Recursion available)</td>
          <td>server</td>
          <td>&ldquo;I&rsquo;m willing to do that&rdquo;</td>
      </tr>
  </tbody>
</table>
<p>The two recursion flags describe the two kinds of DNS server. A <strong>recursive resolver</strong> (your home router, your
company&rsquo;s DNS servers, a public resolver) takes your question and chases it across the internet for you. An
<strong>authoritative server</strong> holds the records for a zone and only answers for those. The lab&rsquo;s server is a small
<code>dnsmasq</code> that does a bit of both, which is why its answer for <code>web.lab</code> has AA and RA set at once.</p>
<p>Notice that the answer for <code>nas.lab</code> came back with flags <code>0x8183</code>: AA is <strong>off</strong>. The server is saying &ldquo;no
such name&rdquo; without claiming to be the authority for it. Small details like that tell you what kind of server
you&rsquo;re talking to.</p>
<hr>
<h2 id="nxdomain-refused-servfail-what-does-each-reply-code-tell-you">NXDOMAIN, REFUSED, SERVFAIL: what does each reply code tell you?</h2>
<p>The last four bits of the flags are the <strong>reply code</strong> (rcode). It&rsquo;s the server&rsquo;s one-word verdict on your
question, and it&rsquo;s worth more than anything else in the packet when a lookup fails.</p>
<table>
  <thead>
      <tr>
          <th>rcode</th>
          <th>Name</th>
          <th>Wireshark shows</th>
          <th>What it tells you</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>0</td>
          <td>NOERROR</td>
          <td>No error</td>
          <td>the server answered. It may still have no records of the type you asked for</td>
      </tr>
      <tr>
          <td>2</td>
          <td>SERVFAIL</td>
          <td>Server failure</td>
          <td>the server tried and couldn&rsquo;t get an answer: often a broken upstream or a DNSSEC failure</td>
      </tr>
      <tr>
          <td>3</td>
          <td>NXDOMAIN</td>
          <td>No such name</td>
          <td>the name doesn&rsquo;t exist. The server is sure</td>
      </tr>
      <tr>
          <td>5</td>
          <td>REFUSED</td>
          <td>Refused</td>
          <td>the server won&rsquo;t answer this question for you, by policy</td>
      </tr>
  </tbody>
</table>
<p>The difference between NXDOMAIN and REFUSED is the difference between &ldquo;the name is wrong&rdquo; and &ldquo;you asked the
wrong server&rdquo;. NXDOMAIN is about the <strong>name</strong>: check the spelling, the domain, the record. REFUSED is about
the <strong>server</strong>: it isn&rsquo;t configured to answer that question, or to answer you. A capture tells them apart in
one glance, while most applications report both as the same vague &ldquo;host not found&rdquo;.</p>
<p>One code that catches people out is NOERROR with an empty answer section. It means the name exists but has no
record of the type you asked for, such as a name with an IPv4 address and no IPv6 one.
<a href="https://www.rfc-editor.org/rfc/rfc2308">RFC 2308</a> calls this NODATA. It isn&rsquo;t an error, even though nothing
useful came back.</p>
<hr>
<h2 id="a-aaaa-and-cname-records">A, AAAA and CNAME records</h2>
<p>An answer is a list of <strong>resource records</strong>. Each has a name, a type, a class (always <code>IN</code>, for internet), a
<strong>TTL</strong> and a value.</p>
<table>
  <thead>
      <tr>
          <th>Type</th>
          <th>Value</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><strong>A</strong></td>
          <td>an IPv4 address, 4 bytes</td>
      </tr>
      <tr>
          <td><strong>AAAA</strong></td>
          <td>an IPv6 address, 16 bytes</td>
      </tr>
      <tr>
          <td><strong>CNAME</strong></td>
          <td>another name: &ldquo;this name is an alias; the real one is over there&rdquo;</td>
      </tr>
  </tbody>
</table>
<p>The <code>intranet.lab</code> response shows a CNAME at work:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">Answers
</span></span><span class="line"><span class="cl">    intranet.lab: type CNAME, class IN, cname web.lab
</span></span><span class="line"><span class="cl">        Time to live: 60 (1 minute)
</span></span><span class="line"><span class="cl">    web.lab: type A, class IN, addr 192.168.2.10
</span></span><span class="line"><span class="cl">        Time to live: 60 (1 minute)
</span></span></code></pre></div><p>Two records in one response. The server didn&rsquo;t just say &ldquo;it&rsquo;s really web.lab, go and ask about that&rdquo;. It
followed the alias itself and put web.lab&rsquo;s address in the same packet, so the laptop got its answer in one
round trip. That&rsquo;s why the response is 120 bytes and web.lab&rsquo;s alone is 94.</p>
<p>CNAMEs are everywhere on the real internet. A company&rsquo;s <code>www</code> name is often a CNAME for a content delivery
network&rsquo;s name, which can be a CNAME for another. A chain of three or four aliases in one response is normal.
When a site resolves to an address you didn&rsquo;t expect, the CNAME chain in the response is the first thing to
read.</p>
<hr>
<h2 id="how-long-can-an-answer-be-cached-ttl">How long can an answer be cached? TTL</h2>
<p>The <strong>TTL</strong> (time to live) on each record is the number of seconds the answer may be kept and reused before
asking again. It has nothing to do with the TTL field in the IP header, which counts router hops. Same name,
different job.</p>
<p>TTLs are a trade-off. A low TTL means a change (a new server address, a failover) reaches everyone quickly,
at the cost of more lookups. A high TTL means fewer lookups and faster page loads, and a change that takes
hours to reach everyone. The lab&rsquo;s server hands out 60 seconds for everything.</p>
<p>Now the part that confuses people with their first DNS capture: <strong>caching happens off the wire</strong>. Windows,
macOS and most Linux desktops keep a local cache, and browsers keep their own on top. If the answer is cached,
the second lookup never leaves the machine, and your capture shows nothing at all. That&rsquo;s not a failure to
capture. It&rsquo;s the cache doing its job. To see a lookup, clear the cache first, or capture on a host that
doesn&rsquo;t have one.</p>
<p>The lab&rsquo;s hosts don&rsquo;t cache. They&rsquo;re small Alpine Linux machines with no caching service, so every lookup goes
on the wire, every time. That&rsquo;s handy for learning, and it&rsquo;s worth remembering that your laptop at work
behaves differently.</p>
<p>Negative answers can be cached too. RFC 2308 has an authoritative server send its zone&rsquo;s SOA record (the
zone&rsquo;s &ldquo;start of authority&rdquo;, with its timers) along with an NXDOMAIN, and that record says how long the &ldquo;no&rdquo;
may be kept. The lab&rsquo;s NXDOMAIN carries no SOA at all: <code>Authority RRs: 0</code>.</p>
<hr>
<h2 id="what-is-the-opt-record-on-the-end-of-every-line">What is the OPT record on the end of every line?</h2>
<p>Every <code>dig</code> query and answer in the lab ends with <code>OPT</code>. That&rsquo;s <strong>EDNS</strong> (Extension Mechanisms for DNS,
<a href="https://www.rfc-editor.org/rfc/rfc6891">RFC 6891</a>). The original DNS limited a UDP answer to 512 bytes. EDNS
adds a pseudo-record, type OPT (41), to the additional section, where the client says how big an answer it can
take, and leaves room for options:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">&lt;Root&gt;: type OPT
</span></span><span class="line"><span class="cl">    UDP payload size: 1232
</span></span><span class="line"><span class="cl">    EDNS0 version: 0
</span></span><span class="line"><span class="cl">    Option: COOKIE
</span></span></code></pre></div><p>dig offers 1,232 bytes and adds a DNS cookie, a small token that helps a server tell a real client from a
spoofed one. None of it is part of the answer. If an answer is still too big, the server sets the TC flag and
the client asks again over TCP.</p>
<hr>
<h2 id="why-did-one-short-name-cause-six-queries">Why did one short name cause six queries?</h2>
<p><code>dig</code> asks exactly what you tell it. Almost nothing else does. Ping, curl, your browser and your mail client
ask the operating system&rsquo;s <strong>resolver</strong>, the library code that does lookups for programs, and the resolver
has its own ideas.</p>
<p>In the lab, the laptop got its resolver settings from DHCP (lesson 4):</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">search lab
</span></span><span class="line"><span class="cl">nameserver 192.168.2.53
</span></span></code></pre></div><p><code>nameserver</code> came from DHCP option 6. <code>search lab</code> is the <strong>search domain</strong>, from option 15: when a name looks
short, try it with <code>.lab</code> on the end. The rule for &ldquo;short&rdquo; is <code>ndots</code>, which defaults to 1 (see the
<a href="https://man7.org/linux/man-pages/man5/resolv.conf.5.html">resolv.conf man page</a>): a name with fewer dots than
that gets the search list tried first.</p>
<p>So here&rsquo;s what the laptop sent when it ran <code>ping nas</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl"> 3 0.219553792 192.168.1.175 → 192.168.2.53 DNS 67 Standard query 0xc991 A nas.lab
</span></span><span class="line"><span class="cl"> 4 0.220102917 192.168.1.175 → 192.168.2.53 DNS 67 Standard query 0xcb5b AAAA nas.lab
</span></span><span class="line"><span class="cl"> 5 0.221055167 192.168.2.53 → 192.168.1.175 DNS 67 Standard query response 0xc991 No such name A nas.lab
</span></span><span class="line"><span class="cl"> 6 0.221235334 192.168.2.53 → 192.168.1.175 DNS 67 Standard query response 0xcb5b No such name AAAA nas.lab
</span></span><span class="line"><span class="cl"> 7 0.221357750 192.168.1.175 → 192.168.2.53 DNS 63 Standard query 0x968f A nas
</span></span><span class="line"><span class="cl"> 8 0.221519209 192.168.1.175 → 192.168.2.53 DNS 63 Standard query 0x97b8 AAAA nas
</span></span><span class="line"><span class="cl"> 9 0.221737250 192.168.2.53 → 192.168.1.175 DNS 63 Standard query response 0x968f Refused A nas
</span></span><span class="line"><span class="cl">10 0.221890417 192.168.2.53 → 192.168.1.175 DNS 63 Standard query response 0x97b8 Refused AAAA nas
</span></span><span class="line"><span class="cl">13 2.729063751 192.168.1.175 → 192.168.2.53 DNS 63 Standard query 0x968f A nas
</span></span><span class="line"><span class="cl">14 2.729521918 192.168.1.175 → 192.168.2.53 DNS 63 Standard query 0x97b8 AAAA nas
</span></span><span class="line"><span class="cl">15 2.730466668 192.168.2.53 → 192.168.1.175 DNS 63 Standard query response 0x968f Refused A nas
</span></span><span class="line"><span class="cl">16 2.730674918 192.168.2.53 → 192.168.1.175 DNS 63 Standard query response 0x97b8 Refused AAAA nas
</span></span></code></pre></div><p>Read it in four steps:</p>
<ol>
<li><strong>The search domain first.</strong> <code>nas</code> has no dots, so the resolver tried <code>nas.lab</code> before <code>nas</code>.</li>
<li><strong>A and AAAA, side by side.</strong> The resolver doesn&rsquo;t know whether the program wants IPv4 or IPv6, so it asks
for both at once, two queries per name, even on a network where nobody uses IPv6.</li>
<li><strong>The bare name.</strong> <code>nas.lab</code> was NXDOMAIN, so it tried plain <code>nas</code>. The lab&rsquo;s server only answers for names
under <code>lab</code> and has nowhere else to ask, so it refused.</li>
<li><strong>A retry.</strong> 2.5 seconds later the resolver sent the same two questions again, with the <strong>same transaction
IDs</strong> (<code>0x968f</code>, <code>0x97b8</code>), got refused again, and gave up.</li>
</ol>
<p>Six queries and two and a half seconds, for one word. Ping just said <code>Try again</code>.</p>
<p>Two more things in that listing are worth noticing. The queries are 67 and 63 bytes, against 90 for dig&rsquo;s,
because the laptop&rsquo;s resolver (Alpine&rsquo;s C library, musl) sends no OPT record at all. Different clients send
different DNS, and that&rsquo;s one reason &ldquo;it works with dig&rdquo; doesn&rsquo;t prove an application&rsquo;s lookup will work. And
the retry reused its transaction IDs, which is how you can tell a retry from a new lookup at a glance.</p>
<hr>
<h2 id="what-you-wont-see-in-a-capture">What you won&rsquo;t see in a capture</h2>
<p>Plain DNS over UDP port 53 is the easy case. Three others show up less clearly, or not at all:</p>
<table>
  <thead>
      <tr>
          <th>Transport</th>
          <th>What the capture shows</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>DNS over TCP, port 53 (<a href="https://www.rfc-editor.org/rfc/rfc7766">RFC 7766</a>)</td>
          <td>a normal TCP connection carrying DNS. Wireshark decodes it. Used after a truncated answer, and for zone transfers</td>
      </tr>
      <tr>
          <td>DNS over TLS, port 853 (<a href="https://www.rfc-editor.org/rfc/rfc7858">RFC 7858</a>)</td>
          <td>a TLS connection to port 853. You can see that DNS happened, not what was asked</td>
      </tr>
      <tr>
          <td>DNS over HTTPS (<a href="https://www.rfc-editor.org/rfc/rfc8484">RFC 8484</a>)</td>
          <td>HTTPS on port 443, like any other web traffic. Nothing marks it as DNS</td>
      </tr>
  </tbody>
</table>
<p>The last one matters for troubleshooting. A browser that uses DNS over HTTPS doesn&rsquo;t ask the resolver in
<code>resolv.conf</code> at all, so a capture filtered on <code>dns</code> stays empty while pages load fine. If you see no DNS and
expected some, ask whether the application is doing its own. <a href="/lessons/wireshark-101/tls-without-decryption/">Lesson 7</a>
covers what an encrypted connection still shows.</p>
<hr>
<h2 id="what-i-see-go-wrong">What I see go wrong</h2>
<p><strong>&ldquo;DNS is slow&rdquo; that&rsquo;s really the search list.</strong> A short or mistyped name that walks the search list, gets
refused or times out, and retries, can cost seconds before the application gives up or falls back. From the
application it looks like a slow network. In a capture, it&rsquo;s a run of queries for names nobody typed. Count
the queries a single lookup causes before blaming the server.</p>
<p><strong>Reading only the first answer.</strong> Wireshark&rsquo;s Info column shows the answers in order, and the first one is
often a CNAME. The address you care about is at the end of the chain.</p>
<p><strong>Treating REFUSED like NXDOMAIN.</strong> They send you to different places. NXDOMAIN: fix the name or the record.
REFUSED: fix which server is asked, or what that server is allowed to answer.</p>
<p><strong>Expecting every lookup on the wire.</strong> A cached answer never leaves the machine. An empty capture right after
a successful lookup usually means the cache worked, not that capturing failed.</p>
<p><strong>Forgetting AAAA.</strong> Hosts ask for both. A broken or slow AAAA answer can delay an application that only ever
uses IPv4, because it waits for both answers before it connects.</p>
<hr>
<h2 id="try-it">Try it</h2>
<p><a href="/labs/wireshark-101/#chapter-5">Chapter 5 of the lab</a> has you capture three lookups, read an address, a TTL,
an alias and a name that doesn&rsquo;t exist, then count what one short name really costs.</p>
<h2 id="whats-next">What&rsquo;s next</h2>
<p>Once the laptop has an address for <code>web.lab</code>, it opens a connection to it.
<a href="/lessons/wireshark-101/tcp/">Lesson 6</a> follows that connection from the first SYN to the last FIN.</p>
]]></content:encoded></item><item><title>Users, Permissions and sudo: Reading ls -l (Linux CLI 101, Lesson 4)</title><link>https://networkdoodles.com/lessons/linux-cli-101/users-permissions-sudo/</link><pubDate>Mon, 16 Nov 2026 09:00:00 -0500</pubDate><dc:creator>David Henderson</dc:creator><guid>https://networkdoodles.com/lessons/linux-cli-101/users-permissions-sudo/</guid><category>linux</category><category>networking-basics</category><category>security</category><category>devops</category><description>Owners, groups and the nine permission bits in ls -l, chmod with 600, 644 and 755, why SSH refuses a key others can read, and when sudo and apt need root.</description><content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[<p>Every file on a Linux system belongs to one <strong>user</strong> and one <strong>group</strong>, and carries nine permission bits: read,
write and execute (<code>rwx</code>), once for the owner, once for the group and once for everyone else. <code>ls -l</code> shows them
as letters, like <code>-rw-r--r--</code>; <code>chmod</code> sets them, usually as three digits: <code>644</code> for a normal file, <code>755</code> for a
program or folder, <code>600</code> for anything private. <strong>root</strong> is the one user the rules don&rsquo;t apply to, and <code>sudo</code> is
how you borrow root&rsquo;s power for one command, which is what installing software with <code>apt</code> needs.</p>
<p>You&rsquo;ll meet this the first time SSH refuses your key, a script won&rsquo;t run, or a tool says &ldquo;Permission denied&rdquo;.
Each of those is a permission bit, and each takes a few seconds to fix once you can read <code>ls -l</code>.</p>
<hr>
<h2 id="what-are-users-and-groups">What are users and groups?</h2>
<p>A <strong>user</strong> is an identity the system checks before it lets anything happen. Every process runs as some user,
and every file is owned by one. To the kernel, a user is a number, the <strong>UID</strong> (user ID). The name is just a
label for it, kept in <code>/etc/passwd</code>.</p>
<p>A <strong>group</strong> is a named set of users, with its own number, the <strong>GID</strong>. Groups let you give several people the
same access without listing each of them on every file.</p>
<p><code>id</code> shows who you are. In the lab&rsquo;s codespace:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ id
</span></span><span class="line"><span class="cl">uid=1000(vscode) gid=1000(vscode) groups=1000(vscode),104(docker),997(clab_admins),998(pipx),999(nvm),1001(ssh)
</span></span></code></pre></div><p>You&rsquo;re user <code>vscode</code>, number 1000. Your <strong>primary group</strong> (the one new files get) is also called <code>vscode</code>, and
you&rsquo;re in a few more. Being in the <code>docker</code> group is what lets you run <code>docker</code> without asking for root.</p>
<p>On ws1, the lab&rsquo;s workstation, the same command says something different:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ id
</span></span><span class="line"><span class="cl">uid=1000(netops) gid=1000(netops) groups=1000(netops)
</span></span></code></pre></div><p>Same number, different name. Users are <strong>local to each machine</strong>: UID 1000 on the codespace and UID 1000 on
ws1 have nothing to do with each other. That matters later, when you copy files between machines with SSH
(<a href="/lessons/linux-cli-101/ssh-and-keys/">lesson 6</a>).</p>
<p>Each line of <code>/etc/passwd</code> is one user, with fields split by colons:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">root:x:0:0:root:/root:/bin/bash
</span></span><span class="line"><span class="cl">vscode:x:1000:1000::/home/vscode:/bin/bash
</span></span></code></pre></div><p>Name, a placeholder <code>x</code> (the password hash lives in <code>/etc/shadow</code>, which only root can read), UID, GID, a
comment, the home folder and the login shell.</p>
<hr>
<h2 id="how-do-i-read-ls--l">How do I read ls -l?</h2>
<p><code>ls -l</code> prints one line per file. The first ten characters are the type and the permissions:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ ls -l
</span></span><span class="line"><span class="cl">-rw-r--r-- 1 vscode vscode  3 Nov 11 05:56 notes.txt
</span></span><span class="line"><span class="cl">drwxr-xr-x 2 vscode vscode 40 Nov 11 05:56 reports
</span></span></code></pre></div><p>Take <code>-rw-r--r--</code> apart:</p>
<table>
  <thead>
      <tr>
          <th>Characters</th>
          <th><code>-</code></th>
          <th><code>rw-</code></th>
          <th><code>r--</code></th>
          <th><code>r--</code></th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>Meaning</td>
          <td>type</td>
          <td><strong>user</strong> (the owner)</td>
          <td><strong>group</strong></td>
          <td><strong>other</strong> (everyone else)</td>
      </tr>
      <tr>
          <td>Here</td>
          <td>a regular file</td>
          <td>read, write</td>
          <td>read</td>
          <td>read</td>
      </tr>
  </tbody>
</table>
<p>The type is <code>-</code> for a file and <code>d</code> for a directory. Then three sets of three, always in the order <code>r</code>, <code>w</code>, <code>x</code>.
A letter means the permission is there; a dash means it isn&rsquo;t.</p>
<p>After the permissions come a link count (ignore it for now), the <strong>owner</strong> (<code>vscode</code>), the <strong>group</strong>
(<code>vscode</code>), the size in bytes, the time it was last changed, and the name.</p>
<p>When you try to open a file, Linux checks <strong>one</strong> set: the user set if you own it, otherwise the group set if
you&rsquo;re in its group, otherwise the other set. It doesn&rsquo;t add them up. If you own a file with <code>---r--r--</code>, you
can&rsquo;t read it, even though everyone else can.</p>
<hr>
<h2 id="what-do-r-w-and-x-mean-on-files-and-on-folders">What do r, w and x mean on files and on folders?</h2>
<p>The same three letters mean slightly different things on a directory:</p>
<table>
  <thead>
      <tr>
          <th>Bit</th>
          <th>On a file</th>
          <th>On a directory</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>r</code> read</td>
          <td>read its contents</td>
          <td>list the names in it</td>
      </tr>
      <tr>
          <td><code>w</code> write</td>
          <td>change its contents</td>
          <td>create, delete and rename files in it</td>
      </tr>
      <tr>
          <td><code>x</code> execute</td>
          <td>run it as a program</td>
          <td>go into it, and reach the files inside</td>
      </tr>
  </tbody>
</table>
<p>Two consequences catch people out. Deleting a file needs <code>w</code> on the <strong>folder</strong>, not on the file. And a folder
without <code>x</code> is a locked door, even if you can read the names through the keyhole. Here&rsquo;s a folder set to <code>600</code>
(read and write, no execute) by its own owner:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ ls -l locked
</span></span><span class="line"><span class="cl">ls: cannot access &#39;locked/f&#39;: Permission denied
</span></span><span class="line"><span class="cl">total 0
</span></span><span class="line"><span class="cl">-????????? ? ? ? ?            ? f
</span></span><span class="line"><span class="cl">$ cat locked/f
</span></span><span class="line"><span class="cl">cat: locked/f: Permission denied
</span></span></code></pre></div><p><code>ls</code> could read the name <code>f</code>, because of <code>r</code>, but not look at the file, because there&rsquo;s no <code>x</code>. That&rsquo;s why
folders are almost always <code>755</code> or <code>700</code>, never <code>644</code> or <code>600</code>.</p>
<hr>
<h2 id="what-do-644-755-and-600-mean">What do 644, 755 and 600 mean?</h2>
<p>Each permission is worth a number: <strong>r = 4, w = 2, x = 1</strong>. Add them up for each set and you get one digit per
set, user then group then other.</p>
<table>
  <thead>
      <tr>
          <th>Mode</th>
          <th>Letters</th>
          <th>Who can do what</th>
          <th>Use it for</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>644</code></td>
          <td><code>rw-r--r--</code></td>
          <td>you read and write; everyone reads</td>
          <td>ordinary files, configs, notes</td>
      </tr>
      <tr>
          <td><code>755</code></td>
          <td><code>rwxr-xr-x</code></td>
          <td>you do anything; everyone reads and runs</td>
          <td>scripts, programs, folders</td>
      </tr>
      <tr>
          <td><code>600</code></td>
          <td><code>rw-------</code></td>
          <td>only you, read and write</td>
          <td>private keys, files with passwords</td>
      </tr>
      <tr>
          <td><code>700</code></td>
          <td><code>rwx------</code></td>
          <td>only you, everything</td>
          <td>private folders, like <code>~/.ssh</code></td>
      </tr>
      <tr>
          <td><code>666</code></td>
          <td><code>rw-rw-rw-</code></td>
          <td>everyone reads and writes</td>
          <td>almost never what you want</td>
      </tr>
      <tr>
          <td><code>777</code></td>
          <td><code>rwxrwxrwx</code></td>
          <td>everyone does anything</td>
          <td>never: it&rsquo;s the &ldquo;I gave up&rdquo; mode</td>
      </tr>
  </tbody>
</table>
<p>So <code>6</code> is <code>4 + 2</code>, read and write; <code>5</code> is <code>4 + 1</code>, read and execute; <code>7</code> is all three.</p>
<p><code>chmod</code> (change mode) sets them. You can give the number, which sets all nine bits at once, or letters, which
change only what you name: <code>u</code>, <code>g</code>, <code>o</code> for user, group and other (<code>a</code> for all), then <code>+</code> or <code>-</code>, then the bits.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ chmod 600 notes.txt; ls -l notes.txt
</span></span><span class="line"><span class="cl">-rw------- 1 vscode vscode 3 Nov 11 05:56 notes.txt
</span></span><span class="line"><span class="cl">$ chmod g+r,o+r notes.txt; ls -l notes.txt
</span></span><span class="line"><span class="cl">-rw-r--r-- 1 vscode vscode 3 Nov 11 05:56 notes.txt
</span></span><span class="line"><span class="cl">$ chmod o+w notes.txt; ls -l notes.txt
</span></span><span class="line"><span class="cl">-rw-r--rw- 1 vscode vscode 3 Nov 11 05:56 notes.txt
</span></span></code></pre></div><p>The last one is the kind of thing you want to spot: anyone can change that file. <code>stat -c &quot;%a %A %n&quot;</code> shows
the number and the letters side by side, if you&rsquo;d rather not do the arithmetic:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ stat -c &#34;%a %A %U %G %n&#34; notes.txt hello.sh
</span></span><span class="line"><span class="cl">646 -rw-r--rw- vscode vscode notes.txt
</span></span><span class="line"><span class="cl">755 -rwxr-xr-x vscode vscode hello.sh
</span></span></code></pre></div><p>Use numbers for the whole result, letters for &ldquo;add execute, leave the rest alone&rdquo; (<code>chmod +x</code>). The <a href="https://www.gnu.org/software/coreutils/manual/html_node/File-permissions.html">coreutils manual on file
permissions</a> has every form.</p>
<h3 id="why-do-new-files-start-as-644">Why do new files start as 644?</h3>
<p>Because of the <strong>umask</strong>, a mask of bits your shell takes away from every new file and folder. In the codespace
it&rsquo;s <code>0022</code>, which removes write for group and other. A new file starts from <code>666</code> and ends up <code>644</code>; a new
folder starts from <code>777</code> and ends up <code>755</code>. New files never get <code>x</code>, which is why you <code>chmod +x</code> a script you&rsquo;ve
just written.</p>
<hr>
<h2 id="why-wont-my-script-run">Why won&rsquo;t my script run?</h2>
<p>Because running a file is a permission, and a new file doesn&rsquo;t have it:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ ls -l hello.sh
</span></span><span class="line"><span class="cl">-rw-r--r-- 1 vscode vscode 33 Nov 11 05:56 hello.sh
</span></span><span class="line"><span class="cl">$ ./hello.sh
</span></span><span class="line"><span class="cl">bash: line 1: ./hello.sh: Permission denied
</span></span><span class="line"><span class="cl">$ bash hello.sh
</span></span><span class="line"><span class="cl">it runs
</span></span><span class="line"><span class="cl">$ chmod 755 hello.sh; ./hello.sh
</span></span><span class="line"><span class="cl">it runs
</span></span></code></pre></div><p><code>bash hello.sh</code> works without <code>x</code> because there you&rsquo;re running <code>bash</code>, which has <code>x</code>, and it only <em>reads</em> the
script. <code>./hello.sh</code> asks the system to run the file itself, which needs <code>x</code>. The shell&rsquo;s exit code for &ldquo;found
it, can&rsquo;t run it&rdquo; is 126, which you&rsquo;ll see again in <a href="/lessons/linux-cli-101/bash-scripting-basics/">lesson 7</a>.</p>
<hr>
<h2 id="whats-the-difference-between-owner-and-permission">What&rsquo;s the difference between owner and permission?</h2>
<p>The <strong>owner</strong> decides who the user set applies to. The <strong>permissions</strong> decide what each set may do. You change
permissions with <code>chmod</code>, and only on files you own. You change the owner with <code>chown</code>, and only root can do
that, even for your own files:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ chown root notes.txt
</span></span><span class="line"><span class="cl">chown: changing ownership of &#39;notes.txt&#39;: Operation not permitted
</span></span></code></pre></div><p>When a file has the right owner and the wrong access, fix the mode. When it has the wrong owner (usually root,
after a careless <code>sudo</code>), that&rsquo;s a job for <code>sudo chown</code>.</p>
<hr>
<h2 id="why-does-ssh-refuse-a-key-file-others-can-read">Why does SSH refuse a key file others can read?</h2>
<p>An SSH <strong>private key</strong> is the secret half of a key pair: whoever has the file can log in as you wherever the
matching public key is accepted. <a href="/lessons/linux-cli-101/ssh-and-keys/">Lesson 6</a> covers the pair. The
permission part is simple: if anyone but you can read the file, SSH assumes it may already be copied, and it
refuses to use it. Here&rsquo;s a freshly made key set to <code>644</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">Permissions 0644 for &#39;demo_key&#39; are too open.
</span></span><span class="line"><span class="cl">It is required that your private key files are NOT accessible by others.
</span></span><span class="line"><span class="cl">This private key will be ignored.
</span></span><span class="line"><span class="cl">Load key &#34;demo_key&#34;: bad permissions
</span></span></code></pre></div><p>Above those lines, SSH prints a box of <code>@</code> signs with a warning in capitals. The <a href="https://man.openbsd.org/ssh#FILES">ssh manual</a>
says it plainly: a private key file should be readable by you and &ldquo;not accessible by others&rdquo;, and ssh
simply ignores one that is.</p>
<p>The fix is <code>chmod 600</code> on the key. Nothing else. Don&rsquo;t copy the key, don&rsquo;t make a new one, don&rsquo;t change the
owner. SSH applies the same idea to the folder around it: in the codespace, <code>~/.ssh</code> is <code>drwx------</code> (700), and
on ws1 <code>authorized_keys</code> is <code>-rw-------</code> (600). SSH (OpenSSH 9.2 in the codespace, 10.0 on ws1) is strict on
purpose, because automation tools like Ansible and Git log in with these same files and nobody is watching.</p>
<hr>
<h2 id="what-is-root-and-what-does-sudo-do">What is root, and what does sudo do?</h2>
<p><strong>root</strong> is the user with UID 0, and the kernel skips permission checks for it. root can read <code>/etc/shadow</code>,
change any file&rsquo;s owner and write anywhere. That&rsquo;s why you don&rsquo;t work as root day to day: one typo has no
safety net.</p>
<p><code>sudo</code> (&ldquo;superuser do&rdquo;) runs <strong>one command</strong> as root, if the system&rsquo;s rules in <code>/etc/sudoers</code> say you may. The
rules file is itself readable only by root:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ ls -l /etc/sudoers
</span></span><span class="line"><span class="cl">-r--r----- 1 root root 1714 Apr 11  2026 /etc/sudoers
</span></span><span class="line"><span class="cl">$ cat /etc/sudoers
</span></span><span class="line"><span class="cl">cat: /etc/sudoers: Permission denied
</span></span><span class="line"><span class="cl">$ whoami
</span></span><span class="line"><span class="cl">vscode
</span></span><span class="line"><span class="cl">$ sudo whoami
</span></span><span class="line"><span class="cl">root
</span></span></code></pre></div><p><code>sudo -l</code> lists what you&rsquo;re allowed. In the codespace the answer is everything, with no password:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">User vscode may run the following commands on codespace:
</span></span><span class="line"><span class="cl">    (root) NOPASSWD: ALL
</span></span></code></pre></div><p>That&rsquo;s a lab convenience. On a normal server, <code>sudo</code> asks for <strong>your</strong> password (not root&rsquo;s) and logs every
command. The codespace has sudo 1.9.13; the <a href="https://www.sudo.ws/docs/man/sudo.man/">sudo
manual</a> covers the options.</p>
<h3 id="why-doesnt-sudo-work-with-a-redirect">Why doesn&rsquo;t sudo work with a redirect?</h3>
<p>Because the redirect isn&rsquo;t part of the command. From <a href="/lessons/linux-cli-101/pipes-redirection-grep/">lesson 3</a>:
the shell sets up <code>&gt;</code> before the command starts, and your shell is running as you.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ sudo echo hi &gt; /etc/lesson4-demo.txt
</span></span><span class="line"><span class="cl">bash: line 1: /etc/lesson4-demo.txt: Permission denied
</span></span></code></pre></div><p><code>echo</code> ran as root. The file was opened by your shell, as <code>vscode</code>, and <code>/etc</code> doesn&rsquo;t let <code>vscode</code> write. The
error even says <code>bash:</code>, not <code>echo:</code>. The usual fix is <code>echo hi | sudo tee /etc/file</code>, so that the program
writing the file is the one running as root.</p>
<hr>
<h2 id="why-does-installing-software-need-root">Why does installing software need root?</h2>
<p>On Debian and Ubuntu, software comes in <strong>packages</strong>, and <code>apt</code> is the tool that fetches them from Debian&rsquo;s
servers (the <strong>repositories</strong>) and installs them, along with anything they depend on. Installing means writing
to folders like <code>/usr/bin</code> and to the package database in <code>/var/lib/dpkg</code>, and those belong to root:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ ls -ld /usr/bin
</span></span><span class="line"><span class="cl">drwxr-xr-x 1 root root 4096 Nov 11 05:52 /usr/bin
</span></span><span class="line"><span class="cl">$ ls -l /var/lib/dpkg/lock-frontend
</span></span><span class="line"><span class="cl">-rw-r----- 1 root root 0 Nov  7 10:44 /var/lib/dpkg/lock-frontend
</span></span></code></pre></div><p>Anyone can read <code>/usr/bin</code> and run what&rsquo;s in it (<code>r-x</code> for other). Only root can add to it, because a program
there runs for every user. Hence <code>sudo apt</code>.</p>
<p>Two steps, always in this order. <code>sudo apt update</code> downloads the current list of packages and versions; it
installs nothing. <code>sudo apt install &lt;name&gt;</code> then installs a package from that list. Skip the update on a fresh
machine and <code>apt</code> may not find the package at all, because its list is empty or out of date. The <a href="https://manpages.debian.org/bookworm/apt/apt.8.en.html">apt
manual</a> has the rest.</p>
<p>Asking what&rsquo;s installed needs no root:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ apt list --installed 2&gt;/dev/null | grep -E &#34;^(grep|openssh-client|sudo)/&#34;
</span></span><span class="line"><span class="cl">grep/oldstable,now 3.8-5 arm64 [installed,automatic]
</span></span><span class="line"><span class="cl">openssh-client/oldstable,now 1:9.2p1-2+deb12u10 arm64 [installed]
</span></span><span class="line"><span class="cl">sudo/oldstable,now 1.9.13p3-1+deb12u4 arm64 [installed]
</span></span></code></pre></div><p><code>oldstable</code> is Debian&rsquo;s name for the previous release: the codespace runs Debian 12, while the lab&rsquo;s ws1 and
srv1 run Debian 13. <code>automatic</code> means <code>apt</code> installed it as something another package needed.</p>
<hr>
<h2 id="what-i-see-go-wrong">What I see go wrong</h2>
<p><strong><code>chmod 777</code> to make an error go away.</strong> Now every user can change the file or replace the script. Find the
bit that was missing and set only that.</p>
<p><strong><code>chmod -R</code> on a whole tree.</strong> <code>chmod -R 644</code> on a folder removes <code>x</code> from every directory inside it, and
they all become locked doors. Files and folders need different modes; set them separately.</p>
<p><strong><code>sudo</code> for everything.</strong> Running an editor, <code>git</code> or <code>pip</code> with <code>sudo</code> in your home folder leaves files
owned by root there. Later, as yourself, you get &ldquo;Permission denied&rdquo; on your own project. <code>ls -l</code> shows
<code>root</code> in the owner column; <code>sudo chown</code> puts it back. Use <code>sudo</code> for system changes, not to silence errors.</p>
<p><strong>A key that&rsquo;s 644 after a copy.</strong> Keys copied from a Windows share, unzipped, or pulled from a repo often
arrive with loose permissions. SSH then ignores them, and asks for a password instead if the server allows one. <code>chmod 600</code>.</p>
<h2 id="every-folder-on-the-path-with-ls--ld-without-x-on-one-of-them-you-cant-reach-it"><strong>Fixing the file when the folder is the problem.</strong> &ldquo;Permission denied&rdquo; on a file you can plainly read? Check
every folder on the path with <code>ls -ld</code>. Without <code>x</code> on one of them, you can&rsquo;t reach it.</h2>
<h2 id="try-it">Try it</h2>
<p><a href="/labs/linux-cli-101/task-4-1/">Chapter 4 of the lab</a> starts with reading <code>ls -l</code> on a file only root may
read, then has you make a private file and a runnable script with the right modes, and install a missing tool
with <code>apt</code>. The challenge breaks three files on purpose: a private key SSH won&rsquo;t use, a script that won&rsquo;t run,
and a notes file anyone can change. You fix all three without changing who owns them, and the checker logs in
to ws1 with the key to prove it.</p>
<p>If the checker or <code>make break</code> is new to you, <a href="/labs/basics/">Meet the lab</a> explains both. Permissions come
up again in every lab that uses SSH, including <a href="/lessons/python-101/">Python 101</a> and Ansible.</p>
<h2 id="whats-next">What&rsquo;s next</h2>
<p><a href="/lessons/linux-cli-101/processes-environment-path/">Lesson 5</a> is about the programs themselves once they&rsquo;re
running: processes and their IDs, stopping one that won&rsquo;t stop, environment variables, and <code>PATH</code>, the list
of folders the shell searches when you type a command&rsquo;s name.</p>
]]></content:encoded></item><item><title>DHCP on the Wire: How a Host Gets an Address (Wireshark 101, Lesson 4)</title><link>https://networkdoodles.com/lessons/wireshark-101/dhcp/</link><pubDate>Mon, 16 Nov 2026 09:00:00 -0500</pubDate><dc:creator>David Henderson</dc:creator><guid>https://networkdoodles.com/lessons/wireshark-101/dhcp/</guid><category>wireshark</category><category>packet-capture</category><category>networking-basics</category><category>dhcp</category><description>DHCP's four messages in a capture, the options that matter, leases and renewal, and how a relay turns a broadcast into unicast with giaddr in it.</description><content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[<p>DHCP, the Dynamic Host Configuration Protocol, gives a host its address and everything else it needs to use
the network, in four messages: <strong>Discover</strong>, <strong>Offer</strong>, <strong>Request</strong>, <strong>ACK</strong>. The first is sent before the
host has any address, from <code>0.0.0.0</code> to <code>255.255.255.255</code>. The address is only ever <strong>leased</strong>, so the host
renews it, directly with the server, before the lease runs out. And because routers don&rsquo;t pass broadcasts on,
a router acting as a <strong>DHCP relay</strong> carries those messages to a server on another network, writing its own
address into them so the server knows which network the client is on.</p>
<p>Networking 101&rsquo;s <a href="/lessons/networking-101/task-4-1/">DHCP lesson</a> walked through DORA as an animation. This
lesson is the same story in real packets from the lab, where the DHCP server is two hops away from the laptop
and r1 relays every message.</p>
<hr>
<h2 id="how-does-a-host-send-anything-without-an-address">How does a host send anything without an address?</h2>
<p>It cheats, politely. A host that has no IP address still has a MAC address, and it can still send an IP
packet: it just has to make up the addresses. The <strong>Discover</strong> goes:</p>
<ul>
<li><strong>from</strong> <code>0.0.0.0</code>, which means <em>this host, no address yet</em>;</li>
<li><strong>to</strong> <code>255.255.255.255</code>, the <strong>limited broadcast</strong>: everyone on this network;</li>
<li>in an Ethernet frame to <code>ff:ff:ff:ff:ff:ff</code>, the broadcast MAC address;</li>
<li>over UDP, from port <strong>68</strong> (the client&rsquo;s port) to port <strong>67</strong> (the server&rsquo;s).</li>
</ul>
<p>Here are the DHCP frames of the lab&rsquo;s capture, taken on the laptop&rsquo;s cable:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">    6 0.613249583 192.168.1.175 → 192.168.2.53 DHCP 342 DHCP Release  - Transaction ID 0x54c3a747
</span></span><span class="line"><span class="cl">    7 2.240006876      0.0.0.0 → 255.255.255.255 DHCP 342 DHCP Discover - Transaction ID 0xd9e69210
</span></span><span class="line"><span class="cl">    8 2.246596042  192.168.1.1 → 192.168.1.175 DHCP 342 DHCP Offer    - Transaction ID 0xd9e69210
</span></span><span class="line"><span class="cl">    9 2.273590042      0.0.0.0 → 255.255.255.255 DHCP 342 DHCP Request  - Transaction ID 0xd9e69210
</span></span><span class="line"><span class="cl">   10 2.281617209  192.168.1.1 → 192.168.1.175 DHCP 347 DHCP ACK      - Transaction ID 0xd9e69210
</span></span></code></pre></div><p>The lab makes the laptop give its address back first (the Release in frame 6, more on that below), then ask
for a new one. Frames 7 to 10 are DORA. From Discover to ACK took 41.6 milliseconds.</p>
<p>The four messages share one <strong>transaction ID</strong>, <code>0xd9e69210</code>: a random number the client picks for each
exchange. Every answer carries it back. On a network where twenty laptops boot at once, it&rsquo;s how each one
picks its own answers out of the broadcast noise, and in Wireshark it&rsquo;s how you pick one client&rsquo;s exchange out
of a busy capture: filter <code>dhcp.id == 0xd9e69210</code>.</p>
<table>
  <thead>
      <tr>
          <th></th>
          <th>Message</th>
          <th>IP from → to</th>
          <th>What it says</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><strong>D</strong></td>
          <td>Discover</td>
          <td><code>0.0.0.0</code> → <code>255.255.255.255</code></td>
          <td><em>I need an address. Any servers out there?</em></td>
      </tr>
      <tr>
          <td><strong>O</strong></td>
          <td>Offer</td>
          <td>server (here, the relay) → offered address</td>
          <td><em>You can have 192.168.1.175, with these settings</em></td>
      </tr>
      <tr>
          <td><strong>R</strong></td>
          <td>Request</td>
          <td><code>0.0.0.0</code> → <code>255.255.255.255</code></td>
          <td><em>I&rsquo;ll take 192.168.1.175, from server 192.168.2.53</em></td>
      </tr>
      <tr>
          <td><strong>A</strong></td>
          <td>ACK</td>
          <td>server (the relay) → the address</td>
          <td><em>It&rsquo;s yours, for 600 seconds</em></td>
      </tr>
  </tbody>
</table>
<p>Why is the Request still a broadcast, when the laptop already knows the server? Because more than one server
may have made an offer. The Request names the one the laptop chose (option 54, the server identifier) and
the address it&rsquo;s taking (option 50), and broadcasting it tells <em>every</em> server at once: the chosen one
confirms, and the others can put their offered addresses back.</p>
<p>The Offer and ACK go to the offered address directly, not to everyone. The laptop&rsquo;s Discover had its
<strong>broadcast flag</strong> set to 0, which tells the server the client can receive a unicast before its address is
configured. A client that can&rsquo;t would set it to 1, and the answers would be broadcast.</p>
<hr>
<h2 id="which-dhcp-options-matter">Which DHCP options matter?</h2>
<p>The address itself is in a fixed field near the top of the message, <strong>Your (client) IP address</strong>. Everything
else is an <strong>option</strong>: a numbered setting with a length and a value. The numbers come from
<a href="https://www.rfc-editor.org/rfc/rfc2132">RFC 2132</a>, and they&rsquo;re worth learning, because Wireshark shows them
by number. Here&rsquo;s the lab&rsquo;s Offer:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">Dynamic Host Configuration Protocol (Offer)
</span></span><span class="line"><span class="cl">    Message type: Boot Reply (2)
</span></span><span class="line"><span class="cl">    Hops: 1
</span></span><span class="line"><span class="cl">    Transaction ID: 0xd9e69210
</span></span><span class="line"><span class="cl">    Your (client) IP address: 192.168.1.175
</span></span><span class="line"><span class="cl">    Relay agent IP address: 192.168.1.1
</span></span><span class="line"><span class="cl">    Client MAC address: aa:aa:aa:11:11:11 (aa:aa:aa:11:11:11)
</span></span><span class="line"><span class="cl">    Option: (53) DHCP Message Type (Offer)
</span></span><span class="line"><span class="cl">    Option: (54) DHCP Server Identifier (192.168.2.53)
</span></span><span class="line"><span class="cl">    Option: (51) IP Address Lease Time
</span></span><span class="line"><span class="cl">        IP Address Lease Time: 10 minutes (600)
</span></span><span class="line"><span class="cl">    Option: (58) Renewal Time Value
</span></span><span class="line"><span class="cl">        Renewal Time Value: 5 minutes (300)
</span></span><span class="line"><span class="cl">    Option: (59) Rebinding Time Value
</span></span><span class="line"><span class="cl">        Rebinding Time Value: 8 minutes, 45 seconds (525)
</span></span><span class="line"><span class="cl">    Option: (1) Subnet Mask (255.255.255.0)
</span></span><span class="line"><span class="cl">    Option: (15) Domain Name
</span></span><span class="line"><span class="cl">        Domain Name: lab
</span></span><span class="line"><span class="cl">    Option: (6) Domain Name Server
</span></span><span class="line"><span class="cl">        Domain Name Server: 192.168.2.53
</span></span><span class="line"><span class="cl">    Option: (3) Router
</span></span><span class="line"><span class="cl">        Router: 192.168.1.1
</span></span></code></pre></div><p>(Some lines left out.) The ones to know:</p>
<table>
  <thead>
      <tr>
          <th>Option</th>
          <th>Name</th>
          <th>Lab value</th>
          <th>Why it matters</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><strong>53</strong></td>
          <td>DHCP Message Type</td>
          <td>Offer</td>
          <td>which of the messages this is: Wireshark&rsquo;s summary comes from it</td>
      </tr>
      <tr>
          <td><strong>54</strong></td>
          <td>Server Identifier</td>
          <td><code>192.168.2.53</code></td>
          <td>the real server, whoever delivered the message</td>
      </tr>
      <tr>
          <td><strong>51</strong></td>
          <td>Lease Time</td>
          <td>600 s</td>
          <td>how long the address is lent for</td>
      </tr>
      <tr>
          <td><strong>58</strong></td>
          <td>Renewal Time (T1)</td>
          <td>300 s</td>
          <td>when to start renewing</td>
      </tr>
      <tr>
          <td><strong>59</strong></td>
          <td>Rebinding Time (T2)</td>
          <td>525 s</td>
          <td>when to ask any server, if the first won&rsquo;t answer</td>
      </tr>
      <tr>
          <td><strong>1</strong></td>
          <td>Subnet Mask</td>
          <td><code>255.255.255.0</code></td>
          <td>which addresses are neighbors (ARP) and which go to the gateway</td>
      </tr>
      <tr>
          <td><strong>3</strong></td>
          <td>Router</td>
          <td><code>192.168.1.1</code></td>
          <td>the default gateway</td>
      </tr>
      <tr>
          <td><strong>6</strong></td>
          <td>DNS Server</td>
          <td><code>192.168.2.53</code></td>
          <td>where to send name lookups</td>
      </tr>
      <tr>
          <td><strong>15</strong></td>
          <td>Domain Name</td>
          <td><code>lab</code></td>
          <td>added to short names, so <code>web</code> becomes <code>web.lab</code></td>
      </tr>
  </tbody>
</table>
<p>The client has its own say, too. Its Discover carries option <strong>55</strong>, the Parameter Request List: the
options it wants. The lab&rsquo;s laptop asks for 1, 3, 6, 12, 15, 28 and 42. A server only has to send what it has,
and a missing option is a common reason a host &ldquo;has an address but can&rsquo;t reach anything&rdquo;: no option 3, no
gateway.</p>
<p>Every setting the laptop has, apart from its MAC address, came from those few hundred bytes. When a host&rsquo;s
network settings look wrong, the Offer and the ACK are the first place to look, because they&rsquo;re the only
record of what it was told.</p>
<hr>
<h2 id="how-do-leases-and-renewal-work">How do leases and renewal work?</h2>
<p>An address is lent, not given. <a href="https://www.rfc-editor.org/rfc/rfc2131">RFC 2131</a>, the DHCP standard, gives
the client two timers, and the lab&rsquo;s numbers show its defaults exactly:</p>
<table>
  <thead>
      <tr>
          <th>Time</th>
          <th>Lab value</th>
          <th>Share of the lease</th>
          <th>What the client does</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><strong>T1</strong>, renewal</td>
          <td>300 s</td>
          <td>50%</td>
          <td>sends a Request <strong>unicast</strong> to the server that gave the lease</td>
      </tr>
      <tr>
          <td><strong>T2</strong>, rebinding</td>
          <td>525 s</td>
          <td>87.5%</td>
          <td>no answer yet: broadcasts a Request to any server</td>
      </tr>
      <tr>
          <td>lease end</td>
          <td>600 s</td>
          <td>100%</td>
          <td>still no answer: stops using the address</td>
      </tr>
  </tbody>
</table>
<p>The renewal is the interesting one on the wire. By T1 the client has an address and knows the server&rsquo;s
(option 54), so it sends an ordinary unicast packet, from its own address to the server&rsquo;s, routed like any
other traffic. No broadcast, no relay. If you capture a renewal, it looks nothing like DORA: one Request and one
ACK, with real addresses on both.</p>
<p>Ten minutes is a short lease. It suits a lab where you want to watch a renewal, and networks where devices
come and go quickly. Longer leases mean fewer renewals and a pool that empties more slowly when devices leave
without saying goodbye.</p>
<h3 id="release">Release</h3>
<p>A client that&rsquo;s leaving can hand the address back with a <strong>Release</strong>, and the lab&rsquo;s capture starts with one.
Frame 6 goes from the laptop&rsquo;s address, <code>192.168.1.175</code>, to the server&rsquo;s, <code>192.168.2.53</code>: unicast, like a
renewal. Just before it, the laptop sent an ARP request for <code>192.168.1.1</code>. The server is on another network,
so the Release went to the gateway&rsquo;s MAC address like any other routed packet (that&rsquo;s
<a href="/lessons/wireshark-101/arp/">lesson 3</a>&rsquo;s rule at work). A Release is optional (RFC 2131 says a client <em>may</em>
send one), and a device that&rsquo;s simply unplugged can&rsquo;t send one at all. That&rsquo;s why the lease time, not the
Release, is what really frees addresses.</p>
<hr>
<h2 id="why-do-dhcp-broadcasts-need-a-relay">Why do DHCP broadcasts need a relay?</h2>
<p>Because the Discover is a broadcast, and <strong>routers don&rsquo;t forward broadcasts</strong>. That&rsquo;s the point of a router,
and it&rsquo;s what keeps a broadcast domain to one network (Networking 101&rsquo;s
<a href="/lessons/networking-101/task-2-2/">lesson on domains</a>). Without help, every network would need its own DHCP
server.</p>
<p>Real networks don&rsquo;t work like that. A company with dozens of offices runs a handful of DHCP servers in its data
centers. What connects them is the <strong>DHCP relay</strong>: the router on the client&rsquo;s network listens for DHCP
broadcasts and passes each one on to a server, as a unicast packet that can be routed anywhere.</p>
<p>On Cisco-style routers, CapeOS included, one line on the client-facing interface turns it on. Here&rsquo;s r1&rsquo;s:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">interface GigabitEthernet1
</span></span><span class="line"><span class="cl"> description home network (sw1)
</span></span><span class="line"><span class="cl"> ip address 192.168.1.1 255.255.255.0
</span></span><span class="line"><span class="cl"> ip helper-address 192.168.2.53
</span></span></code></pre></div><p><code>ip helper-address 192.168.2.53</code> means: <em>DHCP broadcasts that arrive here, send to 192.168.2.53</em>.</p>
<hr>
<h2 id="what-does-a-relayed-discover-look-like">What does a relayed Discover look like?</h2>
<p>The lab captures both sides of r1 at once: its home side (sw1&rsquo;s port towards r1) and its side towards r2. Here
are the first DHCP frames, as frame, interface, IP source and destination, UDP ports, hops, the relay address
and the summary:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">7	sw1:eth1	0.0.0.0	255.255.255.255	68	67	0	0.0.0.0	DHCP Discover - Transaction ID 0x5fc77b2d
</span></span><span class="line"><span class="cl">8	r1:eth2	192.168.1.1	192.168.2.53	67	67	1	192.168.1.1	DHCP Discover - Transaction ID 0x5fc77b2d
</span></span><span class="line"><span class="cl">9	r1:eth2	192.168.2.53	192.168.1.1	67	67	1	192.168.1.1	DHCP Offer    - Transaction ID 0x5fc77b2d
</span></span><span class="line"><span class="cl">10	sw1:eth1	192.168.1.1	192.168.1.175	67	68	1	192.168.1.1	DHCP Offer    - Transaction ID 0x5fc77b2d
</span></span></code></pre></div><p>It&rsquo;s a different exchange from the one above, so a different transaction ID, but one ID throughout: one
conversation, carried in two hops. Follow the Discover through r1:</p>
<table>
  <thead>
      <tr>
          <th></th>
          <th>On the home network (<code>sw1:eth1</code>)</th>
          <th>After r1 (<code>r1:eth2</code>)</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>IP source</td>
          <td><code>0.0.0.0</code></td>
          <td><code>192.168.1.1</code>, r1</td>
      </tr>
      <tr>
          <td>IP destination</td>
          <td><code>255.255.255.255</code></td>
          <td><code>192.168.2.53</code>, the server</td>
      </tr>
      <tr>
          <td>UDP ports</td>
          <td>68 → 67</td>
          <td>67 → 67</td>
      </tr>
      <tr>
          <td>Hops</td>
          <td>0</td>
          <td>1</td>
      </tr>
      <tr>
          <td>Relay agent IP address (giaddr)</td>
          <td><code>0.0.0.0</code></td>
          <td><code>192.168.1.1</code></td>
      </tr>
  </tbody>
</table>
<p>r1 didn&rsquo;t forward the broadcast. It received it, and sent a <strong>new</strong> packet with the same DHCP message inside,
changed in two places. It added one to <strong>hops</strong>, a counter that stops messages being relayed in circles. And
it wrote its own address on the receiving interface into the <strong>Relay agent IP address</strong> field, which RFC 2131
calls <strong>giaddr</strong> (<em>gateway IP address</em>).</p>
<h3 id="how-giaddr-picks-the-pool">How giaddr picks the pool</h3>
<p>giaddr is the most important field in a relayed message. The server can&rsquo;t tell where the client is from the
packet&rsquo;s source: the client has no address, and the packet arrived from r1 anyway. giaddr tells it: <em>this
client is on the network where 192.168.1.1 lives</em>. The server finds the range that contains that address and
offers from it.</p>
<p>The lab&rsquo;s DHCP server, dnsmasq on the dns host, isn&rsquo;t on <code>192.168.1.0/24</code> at all. It has one range, for that
network, <code>192.168.1.100</code> to <code>192.168.1.199</code>, and it picks that range because giaddr is <code>192.168.1.1</code>. The
laptop got <code>192.168.1.175</code>.</p>
<h3 id="why-the-offer-comes-from-the-relay">Why the Offer comes from the relay</h3>
<p>The server sends its answer back to giaddr, to UDP port 67 on the relay (frame 9). r1 then delivers it on the
home network, from its own address, <code>192.168.1.1</code>, to the client (frame 10). So the laptop only ever talks to
r1: its Offer and ACK arrive <strong>from 192.168.1.1</strong>, while option 54 inside them names the real server,
<code>192.168.2.53</code>.</p>
<p>That&rsquo;s not a contradiction, and it&rsquo;s useful. The IP source tells you who delivered the message. Option 54
tells you who made the decision. When something&rsquo;s wrong with what a client was given, option 54 is the
server whose logs you need.</p>
<hr>
<h2 id="what-i-see-go-wrong">What I see go wrong</h2>
<p><strong>DHCP works on one network and not another.</strong> Nearly always a relay problem. Capture both sides of the
router. No relayed copy on the server side means the router isn&rsquo;t relaying: a missing <code>ip helper-address</code>, or
one on the wrong interface. A relayed copy with no answer means the server has no range for that giaddr, or
something between the relay and the server blocks UDP port 67.</p>
<p><strong>Two servers answering.</strong> A second DHCP server on the network, often a home router or a test box plugged into
the wrong port, sends Offers too. In a capture that&rsquo;s two Offers with the same transaction ID and different
option 54 values. Clients take whichever they like, and some of them end up with the wrong gateway or DNS server.
Filter <code>dhcp.option.dhcp == 2</code> and look at the server identifiers.</p>
<p><strong>A client with an address and no connectivity.</strong> Read the ACK. A missing or wrong option 3 (no gateway) or 6
(no DNS server) explains it before you touch anything else.</p>
<p><strong>Discovers with no Offer at all.</strong> The pool may be full, especially with long leases and a lot of
short-lived devices. The client keeps sending Discovers, each with a new transaction ID. Many operating
systems then give themselves a link-local address from <code>169.254.0.0/16</code>
(<a href="https://www.rfc-editor.org/rfc/rfc3927">RFC 3927</a>), which is a reliable sign that DHCP failed.</p>
<p><strong>Expecting to see renewals in a broadcast capture.</strong> Renewals are unicast to the server. A capture on another
host&rsquo;s switch port won&rsquo;t show them. A capture on the client&rsquo;s cable, or near the server, will.</p>
<hr>
<h2 id="try-it">Try it</h2>
<p><a href="/labs/wireshark-101/#chapter-4">Chapter 4 of the lab</a> has you capture the laptop releasing its address and
asking for a new one, read the lease, the gateway and the DNS server out of the Offer, and then capture both
sides of r1 at once to watch the Discover change as it&rsquo;s relayed.</p>
<h2 id="whats-next">What&rsquo;s next</h2>
<p>With an address, a gateway and a DNS server, the laptop is ready to look up a name. <a href="/lessons/wireshark-101/dns/">Lesson 5</a> is about DNS:
the question and answer behind every name you type, and the extra lookups the domain name from option 15 adds.</p>
]]></content:encoded></item><item><title>Pipes, Redirection and grep: One Question, One Pipeline (Linux CLI 101, Lesson 3)</title><link>https://networkdoodles.com/lessons/linux-cli-101/pipes-redirection-grep/</link><pubDate>Mon, 16 Nov 2026 09:00:00 -0500</pubDate><dc:creator>David Henderson</dc:creator><guid>https://networkdoodles.com/lessons/linux-cli-101/pipes-redirection-grep/</guid><category>linux</category><category>networking-basics</category><category>syslog</category><category>troubleshooting</category><category>capeos</category><description>stdin, stdout and stderr, redirecting them to files, and pipes that join grep, cut, sort and uniq to answer real questions about a router's log in one line.</description><content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[<p>Every command you run in a Linux shell has three streams: one coming in (<strong>stdin</strong>), one for its results
(<strong>stdout</strong>) and one for its complaints (<strong>stderr</strong>). Redirection, with <code>&gt;</code>, <code>&gt;&gt;</code> and <code>2&gt;</code>, sends a stream to a
file. A pipe, <code>|</code>, connects one command&rsquo;s stdout to the next command&rsquo;s stdin. Put small tools in a row like that,
<code>grep</code> to pick lines, <code>sort</code> to order them, <code>uniq -c</code> to count them, and one line answers a question that would
take you ten minutes with a mouse.</p>
<p>This is the lesson where the shell stops being a place to look at files and starts being a tool for questions
about them. The examples all run on <code>r1.log</code>, a real syslog captured from the lab&rsquo;s CapeOS router.</p>
<hr>
<h2 id="what-are-stdin-stdout-and-stderr">What are stdin, stdout and stderr?</h2>
<p>A <strong>stream</strong> is a flow of text into or out of a program. Every program starts with three of them already open:</p>
<table>
  <thead>
      <tr>
          <th>Stream</th>
          <th>Number</th>
          <th>Default</th>
          <th>Used for</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><strong>stdin</strong> (standard input)</td>
          <td>0</td>
          <td>your keyboard</td>
          <td>what the program reads</td>
      </tr>
      <tr>
          <td><strong>stdout</strong> (standard output)</td>
          <td>1</td>
          <td>your terminal</td>
          <td>the results</td>
      </tr>
      <tr>
          <td><strong>stderr</strong> (standard error)</td>
          <td>2</td>
          <td>your terminal</td>
          <td>errors and warnings</td>
      </tr>
  </tbody>
</table>
<p>stdout and stderr both land on your screen, so they look like one thing. They aren&rsquo;t. Ask <code>ls</code> for one file
that exists and one that doesn&rsquo;t:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ ls nosuch r1.log
</span></span><span class="line"><span class="cl">ls: cannot access &#39;nosuch&#39;: No such file or directory
</span></span><span class="line"><span class="cl">r1.log
</span></span></code></pre></div><p>The first line went to stderr, the second to stdout. The split is deliberate: it means you can save or process
the results without the error messages mixed in, and still see the errors on screen.</p>
<p>The numbers matter because redirection uses them. <code>2&gt;</code> means &ldquo;stream 2&rdquo;, which is stderr.</p>
<hr>
<h2 id="how-do-i-send-output-to-a-file">How do I send output to a file?</h2>
<p>With <strong>redirection</strong>: you tell the shell, before the command even starts, to point one of its streams somewhere
else. The program doesn&rsquo;t know or care. It writes to stdout as usual, and the shell has already connected stdout
to a file.</p>
<table>
  <thead>
      <tr>
          <th>You write</th>
          <th>What happens</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>cmd &gt; file</code></td>
          <td>stdout goes to <code>file</code>. The file is created, or <strong>emptied first</strong> if it exists</td>
      </tr>
      <tr>
          <td><code>cmd &gt;&gt; file</code></td>
          <td>stdout is added to the end of <code>file</code></td>
      </tr>
      <tr>
          <td><code>cmd 2&gt; file</code></td>
          <td>stderr goes to <code>file</code></td>
      </tr>
      <tr>
          <td><code>cmd &gt; file 2&gt;&amp;1</code></td>
          <td>stdout goes to <code>file</code>, then stderr goes wherever stdout goes: both in one file</td>
      </tr>
      <tr>
          <td><code>cmd 2&gt; /dev/null</code></td>
          <td>stderr is thrown away. <code>/dev/null</code> is a special file that swallows everything</td>
      </tr>
      <tr>
          <td><code>cmd &lt; file</code></td>
          <td>stdin comes from <code>file</code> instead of the keyboard</td>
      </tr>
  </tbody>
</table>
<p>Here&rsquo;s the same <code>ls</code> with stdout sent to a file. The error still reaches the screen, because only stream 1 moved:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ ls nosuch r1.log &gt; out.txt
</span></span><span class="line"><span class="cl">ls: cannot access &#39;nosuch&#39;: No such file or directory
</span></span><span class="line"><span class="cl">$ cat out.txt
</span></span><span class="line"><span class="cl">r1.log
</span></span></code></pre></div><p>And with stderr sent to a file instead, the screen stays quiet and the complaint is in <code>err.txt</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ ls nosuch 2&gt; err.txt
</span></span><span class="line"><span class="cl">$ cat err.txt
</span></span><span class="line"><span class="cl">ls: cannot access &#39;nosuch&#39;: No such file or directory
</span></span></code></pre></div><p><code>2&gt;&amp;1</code> reads as &ldquo;send stream 2 to wherever stream 1 is going right now&rdquo;. Order matters: the shell processes
redirections left to right, so <code>&gt; file 2&gt;&amp;1</code> puts both in the file, and <code>2&gt;&amp;1 &gt; file</code> doesn&rsquo;t (stderr was
pointed at the terminal before stdout moved). The <a href="https://www.gnu.org/software/bash/manual/html_node/Redirections.html">Bash manual&rsquo;s redirection
section</a> has the full list.</p>
<p>A real use for <code>2&gt; /dev/null</code>: <code>apt</code>, Debian&rsquo;s package tool, warns on stderr whenever its output goes anywhere
but a terminal:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ apt list --installed 2&gt;&amp;1 | head -3
</span></span><span class="line"><span class="cl">WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">Listing...
</span></span></code></pre></div><p>Add <code>2&gt;/dev/null</code> and the warning is gone, while the list itself is untouched.</p>
<hr>
<h2 id="what-does-a-pipe-do">What does a pipe do?</h2>
<p>A <strong>pipe</strong>, written <code>|</code>, connects the stdout of the command on its left to the stdin of the command on its right.
No file in between: the text flows straight from one program into the next. A row of commands joined by pipes is
a <strong>pipeline</strong>.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ wc -l &lt; r1.log
</span></span><span class="line"><span class="cl">37
</span></span></code></pre></div><p><code>wc -l</code> counts lines. The log has 37. Now count only the successful logins: <code>grep</code> picks the matching lines and
hands them on.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ grep LOGIN_OK r1.log
</span></span><span class="line"><span class="cl">*Nov 11 05:42:32.023: %CAPEOS-5-LOGIN_OK: Login by admin from 172.80.102.1 port 60330
</span></span><span class="line"><span class="cl">*Nov 11 05:43:18.153: %CAPEOS-5-LOGIN_OK: Login by admin from 172.80.102.1 port 41092
</span></span><span class="line"><span class="cl">*Nov 11 05:44:03.607: %CAPEOS-5-LOGIN_OK: Login by admin from 172.80.102.1 port 35784
</span></span><span class="line"><span class="cl">$ grep LOGIN_OK r1.log | wc -l
</span></span><span class="line"><span class="cl">3
</span></span></code></pre></div><p>This is the Unix idea in one line: <strong>small tools that each do one job</strong>, joined together. <code>grep</code> doesn&rsquo;t count,
<code>wc</code> doesn&rsquo;t search, and neither needs to.</p>
<p>Only stdout goes through a pipe. stderr still goes to the screen, which is usually what you want:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ ls nosuch r1.log | wc -l
</span></span><span class="line"><span class="cl">ls: cannot access &#39;nosuch&#39;: No such file or directory
</span></span><span class="line"><span class="cl">1
</span></span><span class="line"><span class="cl">$ ls nosuch r1.log 2&gt;&amp;1 | wc -l
</span></span><span class="line"><span class="cl">2
</span></span></code></pre></div><p>The first <code>wc</code> counted one line, because the error never entered the pipe. With <code>2&gt;&amp;1</code> it counted both.</p>
<hr>
<h2 id="which-small-tools-are-worth-knowing-first">Which small tools are worth knowing first?</h2>
<p>Five do most of the work in log pipelines. All of them read stdin when you don&rsquo;t give them a file name, which is
what makes them pipe-friendly. They&rsquo;re part of <a href="https://www.gnu.org/software/coreutils/manual/">GNU coreutils</a>
(version 9.1 in the codespace).</p>
<table>
  <thead>
      <tr>
          <th>Tool</th>
          <th>Job</th>
          <th>Options you&rsquo;ll use</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>wc</code></td>
          <td>counts lines, words, bytes</td>
          <td><code>-l</code> lines only</td>
      </tr>
      <tr>
          <td><code>sort</code></td>
          <td>puts lines in order</td>
          <td><code>-n</code> by number, <code>-r</code> reverse, <code>-u</code> drop repeats</td>
      </tr>
      <tr>
          <td><code>uniq</code></td>
          <td>collapses <strong>adjacent</strong> repeated lines</td>
          <td><code>-c</code> prefix each with its count</td>
      </tr>
      <tr>
          <td><code>cut</code></td>
          <td>keeps some columns of each line</td>
          <td><code>-d</code> the separator, <code>-f</code> which fields</td>
      </tr>
      <tr>
          <td><code>tee</code></td>
          <td>copies stdin to a file <strong>and</strong> to stdout</td>
          <td><code>-a</code> append</td>
      </tr>
  </tbody>
</table>
<p><code>cut</code> treats each line as fields split by one character. Split the log&rsquo;s lines on <code>:</code> and the fourth field is the
message type:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ cut -d: -f4 r1.log | head -3
</span></span><span class="line"><span class="cl"> %CAPEOS-5-RESTART
</span></span><span class="line"><span class="cl"> %CAPEOS-5-LOGIN_OK
</span></span><span class="line"><span class="cl"> %CAPEOS-3-LINK_DOWN
</span></span></code></pre></div><p>(The time stamp holds two colons of its own, which is why the type is field 4, not 2. Counting fields is the
fiddly part of <code>cut</code>.)</p>
<p><code>tee</code> is for when you want to keep a copy of what&rsquo;s flowing through a pipe and keep going. Here it saves the list
of message types to a file while <code>wc</code> counts them:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ grep -o &#34;%CAPEOS-[0-9]-[A-Z_]*&#34; r1.log | sort -u | tee types.txt | wc -l
</span></span><span class="line"><span class="cl">8
</span></span></code></pre></div><hr>
<h2 id="how-does-sort--uniq--c-count-things">How does sort | uniq -c count things?</h2>
<p>This is the pattern you&rsquo;ll use more than any other: pull out the thing you want to count, sort it so that equal
lines sit together, count each run with <code>uniq -c</code>, then sort by the count, biggest first. Here&rsquo;s every message
type in the router&rsquo;s log, counted:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ grep -o &#34;%CAPEOS-[0-9]-[A-Z_]*&#34; r1.log | sort | uniq -c | sort -rn
</span></span><span class="line"><span class="cl">      9 %CAPEOS-5-CONFIG_DONE
</span></span><span class="line"><span class="cl">      8 %CAPEOS-4-LOGIN_FAILED
</span></span><span class="line"><span class="cl">      4 %CAPEOS-5-PROTO_UP
</span></span><span class="line"><span class="cl">      4 %CAPEOS-5-PROTO_DOWN
</span></span><span class="line"><span class="cl">      4 %CAPEOS-3-LINK_UP
</span></span><span class="line"><span class="cl">      4 %CAPEOS-3-LINK_DOWN
</span></span><span class="line"><span class="cl">      3 %CAPEOS-5-LOGIN_OK
</span></span><span class="line"><span class="cl">      1 %CAPEOS-5-RESTART
</span></span></code></pre></div><p>Read it left to right:</p>
<ol>
<li><code>grep -o</code> prints only the part of each line that matched, not the whole line: one message type per line.</li>
<li><code>sort</code> puts identical types next to each other.</li>
<li><code>uniq -c</code> turns each run of identical lines into one line with a count in front.</li>
<li><code>sort -rn</code> sorts by that number (<code>-n</code>), largest first (<code>-r</code>).</li>
</ol>
<p>In one line you&rsquo;ve learned something useful: eight failed logins in a log of 37 lines, more than the successful
ones. That&rsquo;s the kind of thing worth a second look on a real router. (The four ties come out Z to A because <code>-r</code>
reverses the whole comparison, including the alphabetical tie-break.)</p>
<p>Why the first <code>sort</code>? Because <code>uniq</code> only compares each line with the one before it. Skip the sort and it counts
runs, not totals:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ grep -o &#34;%CAPEOS-[0-9]-[A-Z_]*&#34; r1.log | uniq -c | head -5
</span></span><span class="line"><span class="cl">      1 %CAPEOS-5-RESTART
</span></span><span class="line"><span class="cl">      1 %CAPEOS-5-LOGIN_OK
</span></span><span class="line"><span class="cl">      1 %CAPEOS-3-LINK_DOWN
</span></span><span class="line"><span class="cl">      1 %CAPEOS-5-PROTO_DOWN
</span></span><span class="line"><span class="cl">      2 %CAPEOS-5-CONFIG_DONE
</span></span></code></pre></div><p>No error, just wrong numbers. That&rsquo;s the worst kind of mistake, so make <code>sort | uniq -c</code> a single habit in your
head.</p>
<hr>
<h2 id="what-can-grep-do-besides-find-a-word">What can grep do besides find a word?</h2>
<p><code>grep</code> prints the lines that match a pattern. The codespace has GNU grep 3.8, and the <a href="https://www.gnu.org/software/grep/manual/grep.html">GNU grep
manual</a> covers everything. These options cover most days:</p>
<table>
  <thead>
      <tr>
          <th>Option</th>
          <th>Means</th>
          <th>Example on the lab&rsquo;s files</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>-i</code></td>
          <td>ignore case</td>
          <td><code>grep -i &quot;login by&quot;</code> matches <code>Login by</code></td>
      </tr>
      <tr>
          <td><code>-v</code></td>
          <td>invert: lines that <strong>don&rsquo;t</strong> match</td>
          <td><code>grep -v CAPEOS-5</code> drops the severity 5 noise</td>
      </tr>
      <tr>
          <td><code>-c</code></td>
          <td>print a count of matching lines</td>
          <td><code>grep -c LOGIN_OK r1.log</code> prints <code>3</code></td>
      </tr>
      <tr>
          <td><code>-r</code></td>
          <td>search every file under a folder</td>
          <td><code>grep -r hostname files/configs</code></td>
      </tr>
      <tr>
          <td><code>-o</code></td>
          <td>print only the matching part</td>
          <td>used in the counting pipeline above</td>
      </tr>
      <tr>
          <td><code>-E</code></td>
          <td>extended regular expressions</td>
          <td><code>grep -E &quot;LINK_(UP|DOWN)&quot;</code></td>
      </tr>
      <tr>
          <td><code>-n</code></td>
          <td>add line numbers</td>
          <td>handy when you&rsquo;ll open the file next</td>
      </tr>
  </tbody>
</table>
<p>Case matters by default. The log says <code>Login by</code>, so this finds nothing:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ grep &#34;login by&#34; r1.log | wc -l
</span></span><span class="line"><span class="cl">0
</span></span><span class="line"><span class="cl">$ grep -i &#34;login by&#34; r1.log | wc -l
</span></span><span class="line"><span class="cl">3
</span></span></code></pre></div><p><code>-r</code> turns <code>grep</code> into a search across a whole tree, and it prefixes each match with the file it came from.
On the lab&rsquo;s saved configs:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ grep -r hostname files/configs
</span></span><span class="line"><span class="cl">files/configs/nyc-leaf1.cfg:hostname nyc-leaf1
</span></span><span class="line"><span class="cl">files/configs/nyc-leaf2.cfg:hostname nyc-leaf2
</span></span><span class="line"><span class="cl">files/configs/chi leaf2.cfg:hostname chi-leaf2
</span></span><span class="line"><span class="cl">files/configs/chi-leaf1.cfg:hostname chi-leaf1
</span></span></code></pre></div><p>That&rsquo;s &ldquo;which file has the config for chi-leaf2?&rdquo;, answered without opening any of them. Notice <code>chi leaf2.cfg</code>,
with a space in its name, and a hostname that doesn&rsquo;t match it. Real backup folders look like this.</p>
<p><code>-c</code> counts <strong>lines</strong>, not matches. A line that matches twice counts once. Usually that&rsquo;s what you want with
logs, where one event is one line.</p>
<hr>
<h2 id="what-is-a-regular-expression">What is a regular expression?</h2>
<p>A <strong>regular expression</strong>, or <strong>regex</strong>, is a pattern that describes text instead of spelling it out. Every
<code>grep</code> pattern is one: <code>LOGIN_OK</code> is a regex that happens to match only itself. The power is in the characters
that mean something else.</p>
<table>
  <thead>
      <tr>
          <th>Pattern</th>
          <th>Matches</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>.</code></td>
          <td>any one character</td>
      </tr>
      <tr>
          <td><code>[0-9]</code></td>
          <td>one digit; <code>[A-Z_]</code> one capital letter or underscore</td>
      </tr>
      <tr>
          <td><code>*</code></td>
          <td>the thing before it, any number of times, including none</td>
      </tr>
      <tr>
          <td><code>^</code> and <code>$</code></td>
          <td>the start and the end of the line</td>
      </tr>
      <tr>
          <td><code>(UP|DOWN)</code></td>
          <td><code>UP</code> or <code>DOWN</code> (with <code>-E</code>)</td>
      </tr>
      <tr>
          <td><code>+</code>, <code>?</code></td>
          <td>one or more; zero or one (with <code>-E</code>)</td>
      </tr>
  </tbody>
</table>
<p>So <code>%CAPEOS-[0-9]-[A-Z_]*</code> reads: a percent sign, <code>CAPEOS-</code>, one digit, a dash, then any run of capitals and
underscores. That&rsquo;s every message type, without listing them. And <code>^\*Nov 11 05:43</code> would match only lines whose
time stamp starts in that minute.</p>
<p>Two flavours matter. Plain <code>grep</code> uses <strong>basic</strong> regular expressions, where <code>+</code>, <code>?</code>, <code>|</code> and <code>()</code> are ordinary
characters unless you put a backslash before them. <code>grep -E</code> uses <strong>extended</strong> ones, where they&rsquo;re special
straight away. Use <code>-E</code> whenever your pattern has an &ldquo;or&rdquo; in it; it&rsquo;s easier to read:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ grep -cE &#34;LINK_(UP|DOWN)&#34; r1.log
</span></span><span class="line"><span class="cl">8
</span></span></code></pre></div><p>Quote your patterns. Without the quotes, the shell sees <code>*</code>, <code>|</code> and <code>(</code> first and tries to use them itself,
as a glob, a pipe and a syntax error.</p>
<p>This is a first taste. Python uses the same ideas, with more power (named groups, pulling fields out instead of
just matching lines), and <a href="/lessons/python-101/regex-and-textfsm/">Python 101&rsquo;s regex lesson</a> picks up from
here. The difference in practice: <code>grep</code> answers &ldquo;which lines?&rdquo;, and Python answers &ldquo;which values, as data?&rdquo;.</p>
<hr>
<h2 id="what-i-see-go-wrong">What I see go wrong</h2>
<p><strong><code>&gt;</code> when you meant <code>&gt;&gt;</code>.</strong> <code>&gt;</code> empties the file before the command runs. Running a report twice with <code>&gt;</code> keeps
only the second run. That&rsquo;s fine for a report you regenerate, and a disaster for a log you meant to add to.</p>
<p><strong>Redirecting a file onto itself.</strong> <code>sort r1.log &gt; r1.log</code> doesn&rsquo;t sort the file, it destroys it. The shell
empties <code>r1.log</code> to set up the redirect before <code>sort</code> starts reading, so <code>sort</code> reads an empty file:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ sort copy.log &gt; copy.log
</span></span><span class="line"><span class="cl">$ wc -l copy.log
</span></span><span class="line"><span class="cl">0 copy.log
</span></span></code></pre></div><p>Write to a new file, then rename it.</p>
<p><strong><code>uniq</code> without <code>sort</code>.</strong> Wrong counts, no warning. See above.</p>
<p><strong>Errors that vanish into a pipe, or don&rsquo;t.</strong> If a pipeline&rsquo;s output looks short, check whether the first
command wrote errors to the screen that you scrolled past. If you&rsquo;re saving to a file and the file has errors
in it, you used <code>2&gt;&amp;1</code> and maybe didn&rsquo;t mean to.</p>
<p><strong><code>grep -c</code> on several files.</strong> It prints one count per file (<code>r1.log:3</code>), not a total. For a total, pipe
<code>grep</code> into <code>wc -l</code>.</p>
<p><strong>Unquoted patterns.</strong> <code>grep LINK_(UP|DOWN) r1.log</code> without quotes is a shell syntax error, not a grep one.
If an error message mentions <code>bash:</code> and not <code>grep:</code>, the shell got there first.</p>
<p><strong>Case.</strong> The log says <code>LOGIN_FAILED</code> and <code>Login failed</code>. A pattern in the wrong case matches nothing, quietly.
<code>-i</code> when in doubt.</p>
<hr>
<h2 id="try-it">Try it</h2>
<p><a href="/labs/linux-cli-101/task-3-1/">Chapter 3 of the lab</a> gives you the same <code>r1.log</code> and a list of questions about
it: lines about one interface, lines that aren&rsquo;t config changes, link-down events per interface, the severe
lines saved to a report with nothing on the screen. Then a challenge: who keeps failing to log in to the router,
and how often. Each answer is a one-line pipeline, and the checker runs yours on logs it has never seen, so it
has to answer the question, not just match the numbers in this file.</p>
<p><a href="/lessons/linux-cli-101/working-with-files/">Lesson 2</a> covered reading these files with <code>less</code>, <code>head</code> and
<code>tail -f</code>; <code>tail -f r1.log | grep LINK</code> is the live version of everything above. If this was your first
time in a terminal, the <a href="/labs/basics/">Meet the lab</a> page explains the checker and <code>make</code>.</p>
<h2 id="whats-next">What&rsquo;s next</h2>
<p><a href="/lessons/linux-cli-101/users-permissions-sudo/">Lesson 4</a> is about who may read and change a file: users,
groups, the <code>rwx</code> columns in <code>ls -l</code>, and why some files, like a private SSH key, must be readable by you alone.</p>
]]></content:encoded></item><item><title>ARP on the Wire: Who Has This Address? (Wireshark 101, Lesson 3)</title><link>https://networkdoodles.com/lessons/wireshark-101/arp/</link><pubDate>Mon, 16 Nov 2026 09:00:00 -0500</pubDate><dc:creator>David Henderson</dc:creator><guid>https://networkdoodles.com/lessons/wireshark-101/arp/</guid><category>wireshark</category><category>packet-capture</category><category>networking-basics</category><category>arp</category><description>ARP on the wire: the broadcast request and unicast reply, the neighbor cache, ARP for the gateway, gratuitous ARP, and a duplicate IP address caught.</description><content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[<p>ARP, the Address Resolution Protocol, is how a host finds the MAC address that goes with an IP address on its
own network. It&rsquo;s one question, sent to everyone (<em>who has 192.168.1.20?</em>), and one answer, sent only to the
host that asked (<em>192.168.1.20 is at aa:aa:aa:22:22:22</em>). Hosts remember the answer for a while in a
<strong>neighbor cache</strong>, and only ask for addresses on their own network: for anything further away, they ask
for their gateway.</p>
<p>It&rsquo;s the first thing you&rsquo;ll see in almost every capture you take, and it&rsquo;s where a lot of &ldquo;the network is
flaky&rdquo; problems turn out to live. This lesson reads ARP field by field from the lab&rsquo;s captures, then covers
the cache, the gateway, gratuitous ARP, and what two hosts with one address look like on the wire.</p>
<hr>
<h2 id="why-does-ip-need-arp">Why does IP need ARP?</h2>
<p>An IP packet never travels on its own. On an Ethernet network it&rsquo;s carried inside a <strong>frame</strong>, and the frame
needs a destination MAC address: the address of the network card that should pick it up. The laptop knows
the printer&rsquo;s IP address, <code>192.168.1.20</code>, because someone typed it or DNS gave it. It doesn&rsquo;t know the
printer&rsquo;s MAC address. Nothing in IP tells it.</p>
<p>ARP fills that gap. It&rsquo;s defined in <a href="https://www.rfc-editor.org/rfc/rfc826">RFC 826</a>, from 1982, and it
hasn&rsquo;t needed to change much since. You met it in <a href="/lessons/wireshark-101/what-a-capture-shows/">lesson 1</a>:
before the laptop could send its first ping, it sent an ARP request. Every first conversation with a
neighbor starts like that.</p>
<hr>
<h2 id="whats-in-an-arp-request-and-reply">What&rsquo;s in an ARP request and reply?</h2>
<p>Here&rsquo;s the start of the lab&rsquo;s ARP capture, taken on the laptop&rsquo;s cable while it pings the printer:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">    2 1.071035042 aa:aa:aa:11:11:11 → Broadcast    ARP 42 Who has 192.168.1.20? Tell 192.168.1.175
</span></span><span class="line"><span class="cl">    3 1.071288709 aa:aa:aa:22:22:22 → aa:aa:aa:11:11:11 ARP 42 192.168.1.20 is at aa:aa:aa:22:22:22
</span></span><span class="line"><span class="cl">    4 1.071292167 192.168.1.175 → 192.168.1.20 ICMP 98 Echo (ping) request  id=0x092b, seq=1/256, ttl=64
</span></span></code></pre></div><p>The question, the answer 254 microseconds later, and the ping straight after: 3.5 microseconds after the
answer arrived. The laptop was holding the ping until it knew where to send it.</p>
<p>Open the request in Wireshark&rsquo;s details pane and the ARP layer has nine fields. Four of them are the ones
that matter: two pairs of addresses.</p>
<table>
  <thead>
      <tr>
          <th>Field</th>
          <th>In the request</th>
          <th>In the reply</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><strong>Opcode</strong></td>
          <td><code>request (1)</code></td>
          <td><code>reply (2)</code></td>
      </tr>
      <tr>
          <td><strong>Sender MAC address</strong></td>
          <td>the laptop&rsquo;s, <code>aa:aa:aa:11:11:11</code></td>
          <td>the printer&rsquo;s: <strong>the answer</strong></td>
      </tr>
      <tr>
          <td><strong>Sender IP address</strong></td>
          <td>the laptop&rsquo;s, <code>192.168.1.175</code></td>
          <td>the printer&rsquo;s, <code>192.168.1.20</code></td>
      </tr>
      <tr>
          <td><strong>Target MAC address</strong></td>
          <td><code>00:00:00:00:00:00</code>: unknown, that&rsquo;s the question</td>
          <td>the laptop&rsquo;s</td>
      </tr>
      <tr>
          <td><strong>Target IP address</strong></td>
          <td><code>192.168.1.20</code>: the address being asked about</td>
          <td>the laptop&rsquo;s</td>
      </tr>
  </tbody>
</table>
<p>The other five (hardware type Ethernet, protocol type IPv4, sizes 6 and 4) say what kind of addresses are
being mapped. ARP was designed to work for more than Ethernet and IPv4, and those fields are where that
shows. On an Ethernet network carrying IPv4 they never change.</p>
<p>Notice that the MAC addresses are in the frame twice: once in the <strong>Ethernet header</strong>, which the switch reads
to deliver the frame, and once in the <strong>ARP body</strong>, which the hosts read to learn the mapping. They usually
agree. When you&rsquo;re investigating something odd, check that they do.</p>
<p>An ARP message is small: 42 bytes on the wire, header included. That&rsquo;s 14 bytes of Ethernet header and 28 bytes
of ARP.</p>
<hr>
<h2 id="why-is-the-request-a-broadcast-and-the-reply-unicast">Why is the request a broadcast and the reply unicast?</h2>
<p>The request goes to <code>ff:ff:ff:ff:ff:ff</code>, the <strong>broadcast</strong> MAC address, which Wireshark shows as
<code>Broadcast</code>. It has to: the laptop doesn&rsquo;t know who has the address, so it asks everyone. Every host on the
network receives it, and every host except the owner ignores it.</p>
<p>The reply goes <strong>unicast</strong>, straight to the laptop&rsquo;s MAC address. The printer knows exactly who asked,
because the request told it (the Sender fields). There&rsquo;s no reason to bother anyone else.</p>
<p>That difference decides what a capture sees. In lesson 1&rsquo;s capture of two switch ports, the printer&rsquo;s cable
carried the laptop&rsquo;s ARP request for r1, because it was a broadcast, but not r1&rsquo;s reply, which the switch sent
only to the laptop. If you&rsquo;re capturing on a third host&rsquo;s port and see an ARP request with no reply, that&rsquo;s
normal. It&rsquo;s only a problem if you&rsquo;re on the <em>asker&rsquo;s</em> cable.</p>
<p>The broadcast is also why ARP stops at the router. A router doesn&rsquo;t forward broadcasts: the edge of a
broadcast domain, from Networking 101&rsquo;s <a href="/lessons/networking-101/task-2-2/">lesson on domains</a>, is the edge of
where ARP can ask.</p>
<hr>
<h2 id="why-does-a-host-arp-for-its-gateway-and-not-the-far-host">Why does a host ARP for its gateway and not the far host?</h2>
<p>Because the far host can&rsquo;t hear the question. A host only ARPs for addresses <strong>on its own network</strong>, which it
works out from its own address and subnet mask. For anything else, it sends the packet to its <strong>default
gateway</strong>, the router on its network, so the MAC address it needs is the gateway&rsquo;s.</p>
<p>The lab&rsquo;s capture shows the laptop doing both. After pinging the printer, it pings web, <code>192.168.2.10</code>, on
another network:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">    8 1.423931125 aa:aa:aa:11:11:11 → Broadcast    ARP 42 Who has 192.168.1.1? Tell 192.168.1.175
</span></span><span class="line"><span class="cl">    9 1.424048667 aa:c1:ab:d8:b6:be → aa:aa:aa:11:11:11 ARP 42 192.168.1.1 is at aa:c1:ab:d8:b6:be
</span></span><span class="line"><span class="cl">   10 1.424050834 192.168.1.175 → 192.168.2.10 ICMP 98 Echo (ping) request  id=0x092c, seq=1/256, ttl=64
</span></span></code></pre></div><p>It never asks <em>who has 192.168.2.10?</em>. It asks for <code>192.168.1.1</code>, r1. And the ping in frame 10 carries two
destinations that look like they disagree:</p>
<table>
  <thead>
      <tr>
          <th>Layer</th>
          <th>Destination</th>
          <th>Means</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>Ethernet</td>
          <td><code>aa:c1:ab:d8:b6:be</code> (r1)</td>
          <td>the next stop</td>
      </tr>
      <tr>
          <td>IP</td>
          <td><code>192.168.2.10</code> (web)</td>
          <td>the final stop</td>
      </tr>
  </tbody>
</table>
<p>That one frame is the whole of routing. The IP destination stays the same for the whole journey. The Ethernet
destination is rewritten at every hop, and ARP is how each sender finds the next one. (r1&rsquo;s MAC address is
generated when the lab starts, so yours will be different.)</p>
<p>This is also how a wrong subnet mask shows up in a capture. Give the laptop a mask that&rsquo;s too wide and it
thinks web is a neighbor, so it broadcasts <em>who has 192.168.2.10?</em>, and nobody on its network answers (unless the router is set up to
answer on the far host&rsquo;s behalf, which is called <em>proxy ARP</em>). The
capture shows the same question, over and over, with no reply, and no ping ever leaves.</p>
<hr>
<h2 id="what-is-the-neighbor-cache-and-why-do-entries-go-stale">What is the neighbor cache, and why do entries go stale?</h2>
<p>Asking before every packet would be wasteful, so every host keeps the answers in a table: the <strong>ARP cache</strong>,
or <strong>neighbor cache</strong>. On Linux, <code>ip neigh</code> shows it. On Cisco-style routers, <code>show ip arp</code>. When the
laptop wants to send to an address that&rsquo;s in the table, it uses the stored MAC address and sends no ARP at
all.</p>
<p>That&rsquo;s why the lab empties the laptop&rsquo;s cache before each ARP task. Without that, a capture of the laptop
pinging the printer would show the ping and no ARP, because the laptop already knew. It&rsquo;s worth remembering
when you&rsquo;re troubleshooting: <strong>no ARP in a capture doesn&rsquo;t mean ARP is broken</strong>. Usually it means the
answer was already cached.</p>
<p>Entries don&rsquo;t last forever, because the answer can change: a host gets a new network card, a virtual
machine moves to another server, a device is replaced and its address reused. A cached entry that&rsquo;s wrong
sends traffic to a MAC address that isn&rsquo;t there any more. So hosts let entries age, and check them again
before trusting them. How long an entry lives varies by operating system and is configurable. Linux&rsquo;s
<code>ip neigh</code> shows each entry&rsquo;s state, such as <code>REACHABLE</code> (recently confirmed) or <code>STALE</code> (needs checking
before it&rsquo;s trusted again).</p>
<p>You can see a recheck on the wire. The lab&rsquo;s DHCP capture starts with this:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">    1 0.000000000 aa:aa:aa:33:33:33 → aa:aa:aa:11:11:11 ARP 42 Who has 192.168.1.175? Tell 192.168.1.30
</span></span><span class="line"><span class="cl">    2 0.000042375 aa:aa:aa:11:11:11 → aa:aa:aa:33:33:33 ARP 42 192.168.1.175 is at aa:aa:aa:33:33:33
</span></span></code></pre></div><p>Look at the destination of frame 1: not <code>Broadcast</code>, but the laptop&rsquo;s own MAC address. broken-pc already had
an entry for the laptop and was checking it with a <strong>unicast</strong> request, straight to the MAC address it
expected. The laptop confirmed it. Nobody else on the network had to hear the question.</p>
<hr>
<h2 id="what-is-gratuitous-arp">What is gratuitous ARP?</h2>
<p>A <strong>gratuitous ARP</strong> is an ARP message nobody asked for: a host announcing its own mapping to everyone. In the
lab it looks like this:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">    1 0.000000000 aa:aa:aa:33:33:33 → Broadcast    ARP 42 Gratuitous ARP for 192.168.1.20 (Request)
</span></span></code></pre></div><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">Address Resolution Protocol (request/gratuitous ARP)
</span></span><span class="line"><span class="cl">    Opcode: request (1)
</span></span><span class="line"><span class="cl">    [Is gratuitous: True]
</span></span><span class="line"><span class="cl">    Sender MAC address: aa:aa:aa:33:33:33 (aa:aa:aa:33:33:33)
</span></span><span class="line"><span class="cl">    Sender IP address: 192.168.1.20
</span></span><span class="line"><span class="cl">    Target MAC address: Broadcast (ff:ff:ff:ff:ff:ff)
</span></span><span class="line"><span class="cl">    Target IP address: 192.168.1.20
</span></span></code></pre></div><p>The giveaway is that the sender IP and the target IP are the same address. It isn&rsquo;t a question; it&rsquo;s an
announcement: <em>192.168.1.20 is me, at this MAC address</em>. <code>[Is gratuitous: True]</code> is in square brackets
because Wireshark worked it out; it isn&rsquo;t a field in the packet. You can filter on it with
<code>arp.isgratuitous == 1</code>.</p>
<p>Hosts send them for good reasons:</p>
<ul>
<li><strong>Bringing an address up.</strong> A host announces its new address so that anyone with an old entry for it
updates straight away.</li>
<li><strong>Failover.</strong> When a standby device takes over an address from a failed one (a redundant router pair, a
server cluster), it sends a gratuitous ARP so every neighbor starts sending to the new MAC address
immediately, instead of waiting for their caches to age.</li>
<li><strong>Conflict detection.</strong> <a href="https://www.rfc-editor.org/rfc/rfc5227">RFC 5227</a> describes how a host can check
that nobody else is using an address before claiming it, and announce it afterwards.</li>
</ul>
<hr>
<h2 id="what-does-a-duplicate-ip-address-look-like-on-the-wire">What does a duplicate IP address look like on the wire?</h2>
<p>Two hosts with the same IP address is a common mistake: a static address typed onto a device that DHCP had
already given out, a cloned virtual machine, a replacement device configured from the old one&rsquo;s notes. The
network doesn&rsquo;t stop it. Both hosts carry on, and both answer ARP for the address.</p>
<p>In the lab, broken-pc briefly takes the printer&rsquo;s address and announces it (the gratuitous ARP above). Then
the laptop looks for the printer:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">    3 0.584692875 aa:aa:aa:11:11:11 → Broadcast    ARP 42 Who has 192.168.1.20? Tell 192.168.1.175
</span></span><span class="line"><span class="cl">    4 0.584883875 aa:aa:aa:33:33:33 → aa:aa:aa:11:11:11 ARP 42 192.168.1.20 is at aa:aa:aa:33:33:33
</span></span><span class="line"><span class="cl">    5 0.584884584 aa:aa:aa:22:22:22 → aa:aa:aa:11:11:11 ARP 42 192.168.1.20 is at aa:aa:aa:22:22:22
</span></span></code></pre></div><p><strong>One question, two different answers</strong>, less than a microsecond apart. Wireshark notices, and marks
frame 5 with <strong>expert info</strong>, its name for problems it spots across packets rather than inside one:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">[Duplicate IP address detected for 192.168.1.20 (aa:aa:aa:22:22:22) - also in use by aa:aa:aa:33:33:33 (frame 4)]
</span></span><span class="line"><span class="cl">    [Expert Info (Warning/Sequence): Duplicate IP address configured (192.168.1.20)]
</span></span></code></pre></div><p>It&rsquo;s a Warning in the Sequence group, so it shows up in <strong>Analyze › Expert Information</strong> and with the
filter <code>_ws.expert</code>. It&rsquo;s worth opening that window on any capture you&rsquo;re handed: it&rsquo;s the quickest way to
find what&rsquo;s odd.</p>
<p>Which answer does the laptop believe? That depends on the operating system. In the lab&rsquo;s reference capture,
the laptop&rsquo;s pings went to the <em>first</em> reply. Here are the pings that followed, as frame number, Ethernet
source and destination, IP source and destination, and ICMP type (8 is a request, 0 a reply):</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">6	aa:aa:aa:11:11:11	aa:aa:aa:33:33:33	192.168.1.175	192.168.1.20	8
</span></span><span class="line"><span class="cl">7	aa:aa:aa:33:33:33	aa:aa:aa:11:11:11	192.168.1.20	192.168.1.175	0
</span></span></code></pre></div><p>broken-pc answered the pings meant for the printer, with the printer&rsquo;s IP address. From the laptop&rsquo;s point of
view, everything worked. That&rsquo;s what makes duplicate addresses so miserable to troubleshoot: depending on
who answered last time and whose cache entry has aged, traffic goes to one host, then the other. Users report
&ldquo;it works, then it doesn&rsquo;t&rdquo;. The ping succeeds, and still the print job vanishes. The capture is the only
place you can see two MAC addresses claiming one IP.</p>
<hr>
<h2 id="why-does-arps-trust-matter">Why does ARP&rsquo;s trust matter?</h2>
<p>ARP has no authentication. A host believes any reply, and often any announcement, without checking who sent
it. The duplicate address above was an accident, but nothing stops a host doing the same thing on purpose:
claiming the gateway&rsquo;s address so that its neighbors send it their traffic. That&rsquo;s <strong>ARP spoofing</strong>, and
it&rsquo;s the reason enterprise switches offer features such as dynamic ARP inspection, which check ARP messages
against the addresses DHCP actually handed out. For troubleshooting, the lesson is simpler: an ARP reply is
a claim, not a proof. When two MAC addresses claim one IP, find out which one should.</p>
<hr>
<h2 id="what-i-see-go-wrong">What I see go wrong</h2>
<p><strong>&ldquo;There&rsquo;s no ARP in the capture, so ARP is the problem.&rdquo;</strong> Usually the answer was cached. Look for the
traffic that follows instead, and at the cache itself.</p>
<p><strong>A request with no reply, read on the wrong cable.</strong> Replies are unicast. On a third host&rsquo;s port you&rsquo;ll
never see one. On the asker&rsquo;s own cable, a request with no reply means nobody has that address, or the host
that has it is down, or the asker has the wrong subnet mask and is asking for a host that isn&rsquo;t on its
network.</p>
<p><strong>Replaced hardware, old entries.</strong> A device is swapped, keeps its IP address, and gets a new MAC address.
Neighbors keep sending to the old one until their entries age out, unless the new device sends a gratuitous
ARP. A capture shows frames to a MAC address that never answers.</p>
<p><strong>Duplicate addresses that come and go.</strong> The symptom moves between hosts and times of day. Capture on the
asker&rsquo;s cable and look for two different <strong>is at</strong> answers, or open Expert Information.</p>
<p><strong>Ignoring the Ethernet header.</strong> The ARP body says one thing, and the Ethernet source says another? Then
something is answering on another host&rsquo;s behalf. Find out what, and why.</p>
<hr>
<h2 id="try-it">Try it</h2>
<p><a href="/labs/wireshark-101/#chapter-3">Chapter 3 of the lab</a> has you capture the laptop finding the printer and
then the web server, read which address it asked for each time, and then catch broken-pc claiming the
printer&rsquo;s address, with Wireshark&rsquo;s expert info pointing at the frame.</p>
<h2 id="whats-next">What&rsquo;s next</h2>
<p>ARP needs the host to have an IP address already. <a href="/lessons/wireshark-101/dhcp/">Lesson 4</a> is about how it
gets one: the four DHCP messages, sent before the laptop has any address at all, and the router that carries
them to a server two hops away.</p>
]]></content:encoded></item><item><title>Working with Files: Read, Edit, Move and Find (Linux CLI 101, Lesson 2)</title><link>https://networkdoodles.com/lessons/linux-cli-101/working-with-files/</link><pubDate>Mon, 16 Nov 2026 09:00:00 -0500</pubDate><dc:creator>David Henderson</dc:creator><guid>https://networkdoodles.com/lessons/linux-cli-101/working-with-files/</guid><category>linux</category><category>networking-basics</category><category>network-automation</category><category>syslog</category><description>Read logs and configs with less, head and tail -f, edit in nano, copy, move and delete safely, match names with globs, quote spaces, and find files by name.</description><content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[<p>On Linux, nearly everything you care about is a <strong>text file</strong>: logs, configs, inventories, scripts. You read a
short file with <code>cat</code>, a long one with <code>less</code>, the start with <code>head</code>, the end with <code>tail</code>, and a log that&rsquo;s
still growing with <code>tail -f</code>. You change files in an editor (nano is the friendly one), move them with <code>cp</code>,
<code>mv</code> and <code>rm</code>, pick out groups of them with <strong>globs</strong> like <code>*.cfg</code>, and find them anywhere in the tree with
<code>find</code>.</p>
<p>There&rsquo;s no mouse and no recycle bin, so the habits matter as much as the commands. This lesson covers what
each tool is for, how the shell treats names, and the mistakes that cost people files.</p>
<hr>
<h2 id="what-is-the-difference-between-a-text-file-and-a-binary-file">What is the difference between a text file and a binary file?</h2>
<p>A <strong>text file</strong> is bytes that mean characters: letters, digits, spaces and line breaks. You can read it with
any tool, change it in any editor, and compare two versions line by line. A <strong>binary file</strong> is bytes that mean
something only to the program that reads it: a compiled program, an image, a packet capture.</p>
<p>The <code>file</code> command looks inside a file and tells you which it is. In the lab:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">files/logs/r1.log:           ASCII text
</span></span><span class="line"><span class="cl">files/configs/nyc-leaf1.cfg: ASCII text
</span></span><span class="line"><span class="cl">/usr/bin/ls:                 ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, ...
</span></span></code></pre></div><p>The router&rsquo;s log and a device config are plain text. <code>ls</code> itself is a program in <strong>ELF</strong>, Linux&rsquo;s format for
executables. Its first bytes are a marker, not words:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">0000000 177   E   L   F 002 001 001  \0  \0  \0  \0  \0  \0  \0  \0  \0
</span></span></code></pre></div><p>Linux doesn&rsquo;t care about extensions. <code>.cfg</code>, <code>.log</code> and <code>.txt</code> are for humans: <code>file</code> reads the content.
That&rsquo;s why a file with no extension at all, such as <code>/etc/hostname</code>, can still be a perfectly good text file.</p>
<p>Configs, syslog, YAML inventories and scripts are all text, which is why <code>grep</code>, Git and Ansible can work on
them. Print a binary file with <code>cat</code> and the terminal fills with garbage, sometimes enough
to scramble the display. If that happens, type <code>reset</code> and press <kbd>Enter</kbd>.</p>
<hr>
<h2 id="how-do-you-read-a-file-cat-less-head-or-tail">How do you read a file: cat, less, head or tail?</h2>
<p>They all print a file. They differ in how much and how you move around it.</p>
<table>
  <thead>
      <tr>
          <th>Tool</th>
          <th>Shows</th>
          <th>Use it for</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>cat file</code></td>
          <td>the whole file, all at once</td>
          <td>short files: a config, a hosts list</td>
      </tr>
      <tr>
          <td><code>less file</code></td>
          <td>one screen at a time, and you scroll</td>
          <td>anything longer than a screen</td>
      </tr>
      <tr>
          <td><code>head file</code></td>
          <td>the first 10 lines (<code>-n 5</code> for five)</td>
          <td>the start of a log, a CSV&rsquo;s header row</td>
      </tr>
      <tr>
          <td><code>tail file</code></td>
          <td>the last 10 lines</td>
          <td>the most recent log entries</td>
      </tr>
      <tr>
          <td><code>tail -f file</code></td>
          <td>the end, then new lines as they&rsquo;re written</td>
          <td>watching a log live</td>
      </tr>
  </tbody>
</table>
<p>Here&rsquo;s <code>head -n 6</code> on one of the lab&rsquo;s practice configs, a CapeOS <code>show running-config</code> saved to a file:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">Building configuration...
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">Current configuration : 411 bytes
</span></span><span class="line"><span class="cl">!
</span></span><span class="line"><span class="cl">hostname nyc-leaf1
</span></span><span class="line"><span class="cl">!
</span></span></code></pre></div><p>And <code>tail -n 3</code> on ws1&rsquo;s package log, the newest lines last:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">2026-11-11 05:33:18 trigproc ca-certificates:all 20250419+deb13u1 &lt;none&gt;
</span></span><span class="line"><span class="cl">2026-11-11 05:33:18 status half-configured ca-certificates:all 20250419+deb13u1
</span></span><span class="line"><span class="cl">2026-11-11 05:33:19 status installed ca-certificates:all 20250419+deb13u1
</span></span></code></pre></div><p>Log files are written in time order and only ever added to at the end, which is why <code>tail</code> is the first thing
you run on one. <code>wc -l</code> counts lines, so you know whether <code>cat</code> will fit on the screen: the lab&rsquo;s router log
is <code>37 files/logs/r1.log</code>, short enough for either tool.</p>
<h3 id="how-do-you-move-around-in-less">How do you move around in less?</h3>
<p><code>less</code> is a <strong>pager</strong>: a program that shows text one page at a time. <code>man</code> uses one too, so the keys are worth
learning once.</p>
<table>
  <thead>
      <tr>
          <th>Key</th>
          <th>Does</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><kbd>Space</kbd> / <kbd>b</kbd></td>
          <td>a page forward / back</td>
      </tr>
      <tr>
          <td><kbd>↓</kbd> <kbd>↑</kbd></td>
          <td>a line at a time</td>
      </tr>
      <tr>
          <td><kbd>g</kbd> / <kbd>G</kbd></td>
          <td>the start / the end of the file</td>
      </tr>
      <tr>
          <td><code>/text</code> then <kbd>Enter</kbd></td>
          <td>search forward for <em>text</em></td>
      </tr>
      <tr>
          <td><kbd>n</kbd> / <kbd>N</kbd></td>
          <td>the next / previous match</td>
      </tr>
      <tr>
          <td><kbd>F</kbd></td>
          <td>follow the end, like <code>tail -f</code> (<kbd>Ctrl</kbd>+<kbd>C</kbd> to stop)</td>
      </tr>
      <tr>
          <td><kbd>q</kbd></td>
          <td>quit</td>
      </tr>
  </tbody>
</table>
<p>Searching is the reason to use <code>less</code> on a log rather than scrolling: <code>/LINK_DOWN</code> jumps straight to the first
interface that went down. The lab has <code>less</code> 590.</p>
<h3 id="what-does-tail--f-do">What does tail -f do?</h3>
<p><code>tail -f</code> (<strong>follow</strong>) prints the last lines and then waits. Each time something adds a line to the file, it
appears on your screen. It doesn&rsquo;t end on its own: press <kbd>Ctrl</kbd>+<kbd>C</kbd> to stop watching. The
file is untouched.</p>
<p>It&rsquo;s how you watch a problem happen: run <code>tail -f</code> on a log in one terminal, bounce the interface or rerun the
job in another, and see the lines arrive. When a log file gets replaced rather than added to (log rotation
does this), <code>tail -F</code>, with a capital F, keeps following the new file under the same name.</p>
<hr>
<h2 id="which-editor-should-you-use-and-how-do-you-get-out-of-vim">Which editor should you use, and how do you get out of vim?</h2>
<p>You&rsquo;ll edit text files on machines that have no VS Code. There are three editors worth knowing, and you only
need to be good at one.</p>
<table>
  <thead>
      <tr>
          <th>Editor</th>
          <th>Where you&rsquo;ll find it</th>
          <th>Why</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><strong>VS Code</strong></td>
          <td>the codespace; <code>code file</code> opens a file in it</td>
          <td>the easiest: mouse, tabs, colors, undo</td>
      </tr>
      <tr>
          <td><strong>nano</strong></td>
          <td>almost every Linux machine, including ws1 and srv1</td>
          <td>works in any terminal, and shows its keys on screen</td>
      </tr>
      <tr>
          <td><strong>vim</strong> (or <code>vi</code>)</td>
          <td>most servers, often as the default editor</td>
          <td>powerful, but it has <em>modes</em>, which surprise everyone once</td>
      </tr>
  </tbody>
</table>
<p><strong>nano</strong> is the one to learn for remote machines. The two bottom lines list its commands, where <code>^</code> means
<kbd>Ctrl</kbd>: <code>^O</code> writes the file (<kbd>Ctrl</kbd>+<kbd>S</kbd> also saves), <code>^X</code> exits, <code>^W</code> searches,
<code>^K</code> cuts a line and <code>^U</code> pastes it. It asks before throwing away changes. The codespace has GNU nano 7.2.</p>
<p><strong>vim</strong> starts in <em>normal mode</em>, where letters are commands, not text. That&rsquo;s why typing into it seems to do
random things. If you land in vim by accident (a Git commit message, <code>crontab -e</code>, a <code>visudo</code>), this gets you
out:</p>
<table>
  <thead>
      <tr>
          <th>You want to</th>
          <th>Type</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>leave and throw away changes</td>
          <td><kbd>Esc</kbd>, then <code>:q!</code> and <kbd>Enter</kbd></td>
      </tr>
      <tr>
          <td>save and leave</td>
          <td><kbd>Esc</kbd>, then <code>:wq</code> and <kbd>Enter</kbd></td>
      </tr>
  </tbody>
</table>
<p><kbd>Esc</kbd> first, always: it gets you back to normal mode from wherever you are. In the codespace, <code>vi</code> is
<code>vim.tiny</code>, a small build of vim. The slim ws1 container has no <code>vi</code> at all, only nano.</p>
<hr>
<h2 id="how-do-you-copy-move-rename-and-delete-files">How do you copy, move, rename and delete files?</h2>
<p>Four commands, from GNU coreutils (9.1 in the codespace):</p>
<table>
  <thead>
      <tr>
          <th>Command</th>
          <th>Does</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>cp source target</code></td>
          <td>copies; <code>-r</code> to copy a folder and everything in it</td>
      </tr>
      <tr>
          <td><code>mv source target</code></td>
          <td>moves, and renames: a rename is a move within the same folder</td>
      </tr>
      <tr>
          <td><code>rm file</code></td>
          <td>deletes; <code>-r</code> for a folder and everything in it</td>
      </tr>
      <tr>
          <td><code>mkdir folder</code></td>
          <td>makes a folder; <code>-p</code> makes the parents too, and doesn&rsquo;t complain if it exists</td>
      </tr>
  </tbody>
</table>
<p>There&rsquo;s no separate rename command. <code>mv r1.bak r1-before.cfg</code> renames, and <code>mv r2.cfg old/</code> moves the file
into the folder <code>old</code>. The trailing <code>/</code> is a good habit: if <code>old</code> doesn&rsquo;t exist, <code>mv r2.cfg old</code> quietly
renames the file to <code>old</code> instead, and the slash makes <code>mv</code> fail loudly.</p>
<h3 id="why-is-rm-dangerous">Why is rm dangerous?</h3>
<p><strong>There is no recycle bin on the command line.</strong> <code>rm</code> removes the name, and the space is free for reuse
straight away. No undo, no Trash folder: in the codespace, <code>~/.Trash</code> doesn&rsquo;t exist. <code>rm -r</code> takes a whole
tree with it, without asking.</p>
<p><code>cp</code> and <code>mv</code> have a quieter version of the same problem: if the target already exists, they <strong>overwrite it
without asking</strong>. The <code>-i</code> (<strong>interactive</strong>) option makes them check first:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">mv: overwrite &#39;r10.cfg&#39;?
</span></span></code></pre></div><p>Answer <code>n</code> and nothing changes. Habits that help:</p>
<ul>
<li><code>ls</code> a pattern first, and only then <code>rm</code> the same pattern: you see exactly what will go.</li>
<li>Use <code>rm -i</code> when you&rsquo;re unsure, and <code>cp -i</code> and <code>mv -i</code> when the target might exist.</li>
<li>Keep anything you&rsquo;d miss in Git (<span class="link-soon">the GitOps course</span>): then a mistake is one
<code>git restore</code> away.</li>
</ul>
<hr>
<h2 id="what-are-globs-and-who-expands-them">What are globs, and who expands them?</h2>
<p>A <strong>glob</strong> is a pattern for file names. The two you&rsquo;ll use most:</p>
<table>
  <thead>
      <tr>
          <th>Pattern</th>
          <th>Matches</th>
          <th>Example</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>*</code></td>
          <td>any number of characters, including none</td>
          <td><code>*.cfg</code>: every name ending <code>.cfg</code></td>
      </tr>
      <tr>
          <td><code>?</code></td>
          <td>exactly one character</td>
          <td><code>r?.cfg</code>: <code>r1.cfg</code> and <code>r2.cfg</code>, but not <code>r10.cfg</code></td>
      </tr>
      <tr>
          <td><code>[12]</code></td>
          <td>one of the listed characters</td>
          <td><code>r[12].cfg</code>: <code>r1.cfg</code> and <code>r2.cfg</code></td>
      </tr>
  </tbody>
</table>
<p>The important part: <strong>the shell expands the glob, not the command</strong>. Before <code>ls</code> runs, Bash looks in the folder,
swaps the pattern for the matching names, and passes those. You can watch it happen with <code>set -x</code>, which makes
Bash print each command after expansion. In a scratch folder:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">+ ls &#39;nyc spine1.cfg&#39; r1.cfg r10.cfg r2.cfg sw1.cfg
</span></span><span class="line"><span class="cl">+ ls r1.cfg r2.cfg
</span></span></code></pre></div><p>That&rsquo;s <code>ls *.cfg</code> and <code>ls r?.cfg</code> after Bash had finished with them. <code>ls</code> never saw a <code>*</code>. It&rsquo;s why globs work
the same with every command: <code>cp *.cfg backup/</code>, <code>rm *.tmp</code>, <code>mv r*.cfg r1/</code>.</p>
<p>Two consequences:</p>
<ul>
<li><strong>A glob that matches nothing is left as it is.</strong> <code>echo *.yml</code> in a folder with no YAML files prints <code>*.yml</code>,
and the command then complains that the file <code>*.yml</code> doesn&rsquo;t exist.</li>
<li>Globs don&rsquo;t match hidden files: <code>*</code> skips names that start with a dot, the same way <code>ls</code> does.</li>
</ul>
<hr>
<h2 id="how-do-you-handle-file-names-with-spaces">How do you handle file names with spaces?</h2>
<p>The shell splits a command line at spaces, so a name with a space in it turns into two arguments. With a file
called <code>nyc spine1.cfg</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">ls: cannot access &#39;nyc&#39;: No such file or directory
</span></span><span class="line"><span class="cl">ls: cannot access &#39;spine1.cfg&#39;: No such file or directory
</span></span></code></pre></div><p><code>ls</code> went looking for two files that don&rsquo;t exist. Either of these keeps the name in one piece:</p>
<table>
  <thead>
      <tr>
          <th>Write</th>
          <th>How it works</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>&quot;nyc spine1.cfg&quot;</code></td>
          <td>quotes: everything inside is one argument</td>
      </tr>
      <tr>
          <td><code>nyc\ spine1.cfg</code></td>
          <td>a backslash: the next character is taken literally</td>
      </tr>
  </tbody>
</table>
<p>Tab completion writes the backslash for you, which is one more reason to let it finish names. Single quotes
work too, and are stricter: inside <code>'...'</code>, nothing is special at all. Better still, don&rsquo;t create names with
spaces: use <code>-</code> or <code>_</code>. Other people&rsquo;s files will still arrive with them.</p>
<hr>
<h2 id="how-do-you-find-a-file-when-you-dont-know-where-it-is">How do you find a file when you don&rsquo;t know where it is?</h2>
<p><code>find</code> walks down a folder tree and prints every path that matches the tests you give it. The shape is always
<code>find where tests</code>. The lab has GNU findutils 4.9.0.</p>
<table>
  <thead>
      <tr>
          <th>Test</th>
          <th>Matches</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>-name '*.cfg'</code></td>
          <td>the name matches the glob (<code>-iname</code> ignores case)</td>
      </tr>
      <tr>
          <td><code>-type f</code> / <code>-type d</code></td>
          <td>files only / folders only</td>
      </tr>
      <tr>
          <td><code>-mmin -60</code></td>
          <td>changed in the last 60 minutes</td>
      </tr>
      <tr>
          <td><code>-mtime -1</code></td>
          <td>changed in the last day</td>
      </tr>
      <tr>
          <td><code>-mtime +1</code></td>
          <td>changed longer ago (find counts whole days, so <code>+1</code> means at least two days)</td>
      </tr>
      <tr>
          <td><code>-empty</code></td>
          <td>empty files and folders</td>
      </tr>
      <tr>
          <td><code>-size +1M</code></td>
          <td>bigger than a mebibyte</td>
      </tr>
  </tbody>
</table>
<p>Several tests together must all match. In the scratch folder, after a few files were moved around:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ find . -name &#39;*.cfg&#39;
</span></span><span class="line"><span class="cl">./r1.cfg
</span></span><span class="line"><span class="cl">./r10.cfg
</span></span><span class="line"><span class="cl">./sw1.cfg
</span></span><span class="line"><span class="cl">./nyc spine1.cfg
</span></span><span class="line"><span class="cl">./old/r2.cfg
</span></span><span class="line"><span class="cl">./r1-before.cfg
</span></span><span class="line"><span class="cl">$ find . -type d
</span></span><span class="line"><span class="cl">.
</span></span><span class="line"><span class="cl">./old
</span></span><span class="line"><span class="cl">$ find . -mtime +1
</span></span><span class="line"><span class="cl">./old/r2.cfg
</span></span></code></pre></div><p><code>find</code> looks inside subfolders, which <code>ls *.cfg</code> doesn&rsquo;t: it found <code>old/r2.cfg</code>, the one file old enough to
match <code>-mtime +1</code>. Unlike <code>ls</code>, its output isn&rsquo;t sorted. It prints paths in the order it walks the tree.</p>
<p>Notice the quotes around <code>'*.cfg'</code>. Here the glob is for <code>find</code>, not the shell, so it must reach <code>find</code>
unexpanded. Leave them off in a folder that has <code>.cfg</code> files, and the shell expands the pattern first:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">find: paths must precede expression: `r1-before.cfg&#39;
</span></span><span class="line"><span class="cl">find: possible unquoted pattern after predicate `-name&#39;?
</span></span></code></pre></div><p>Searching by age is the one network engineers reach for most: <em>which configs changed since yesterday?</em> is
<code>find</code> with <code>-name</code> and <code>-mtime -1</code>. Searching <em>inside</em> files, for a line rather than a name, is <code>grep</code>, in
<a href="/lessons/linux-cli-101/pipes-redirection-grep/">lesson 3</a>.</p>
<hr>
<h2 id="what-i-see-go-wrong">What I see go wrong</h2>
<p><strong><code>rm</code> with a typo in the pattern.</strong> <code>rm * .cfg</code>, with a stray space, is two arguments: <code>*</code>, everything, and
<code>.cfg</code>. <code>ls</code> the pattern first.</p>
<p><strong>A copy that overwrote the original.</strong> <code>cp new.cfg old.cfg</code> replaces <code>old.cfg</code> silently. <code>-i</code>, or Git.</p>
<p><strong><code>mv</code> into a folder that wasn&rsquo;t there.</strong> The file is now called the folder&rsquo;s name. Write the trailing <code>/</code>.</p>
<p><strong>Stuck in vim.</strong> <kbd>Esc</kbd>, <code>:q!</code>, <kbd>Enter</kbd>.</p>
<p><strong><code>tail -f</code> that never ends.</strong> It&rsquo;s not broken; it&rsquo;s waiting for new lines. <kbd>Ctrl</kbd>+<kbd>C</kbd>.</p>
<p><strong>Unquoted names and patterns.</strong> A name with a space splits in two; a <code>find -name</code> pattern without quotes is
expanded by the shell. Quote both.</p>
<p><strong>Editing a config in Windows and copying it over.</strong> Windows ends lines with two characters, Linux with one.
<code>file</code> reports <em>with CRLF line terminators</em>, and some tools choke on the extra character. VS Code&rsquo;s status bar
shows <code>CRLF</code> or <code>LF</code>, and one click changes it.</p>
<hr>
<h2 id="try-it">Try it</h2>
<p><a href="/labs/linux-cli-101/task-2-1/">Chapter 2 of the lab</a> gives you a real CapeOS syslog from the lab&rsquo;s router.
You read it with <code>less</code> and its search, pick out the first and last events with <code>head</code> and <code>tail</code>, watch new
lines arrive with <code>tail -f</code>, and write a note in nano. The challenge is a messy folder of config backups,
saved by hand: you sort it into folders, fix a name with a space in it, clear out the empty files, and use
<code>find</code> to count what&rsquo;s there. The same skills come back in <a href="/lessons/python-101/files-and-errors/">Python 101&rsquo;s lesson on
files</a>, where a script does the reading.</p>
<h2 id="whats-next">What&rsquo;s next</h2>
<p><a href="/lessons/linux-cli-101/pipes-redirection-grep/">Lesson 3</a> connects commands together: send one command&rsquo;s
output into the next, save it to a file, and use <code>grep</code> to answer a question about a log in one line.</p>
]]></content:encoded></item><item><title>Capture Filters vs Display Filters (Wireshark 101, Lesson 2)</title><link>https://networkdoodles.com/lessons/wireshark-101/capture-vs-display-filters/</link><pubDate>Mon, 16 Nov 2026 09:00:00 -0500</pubDate><dc:creator>David Henderson</dc:creator><guid>https://networkdoodles.com/lessons/wireshark-101/capture-vs-display-filters/</guid><category>wireshark</category><category>packet-capture</category><category>networking-basics</category><category>bpf</category><description>Capture filters (BPF) decide what's saved; display filters decide what's shown. Field names, operators, the != change in Wireshark 3.6, and precedence.</description><content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[<p>Wireshark has two kinds of filter, written in two different languages, and they work at two different
times. A <strong>capture filter</strong> runs while you capture and decides which frames are saved at all: whatever it
rejects is gone for good. A <strong>display filter</strong> runs on a capture you already have and only hides frames:
clear it and they&rsquo;re all back. So the rule is simple. Capture wide, with no filter or a loose one, and filter
narrow afterwards.</p>
<p><a href="/lessons/wireshark-101/what-a-capture-shows/">Lesson 1</a> was about what a capture is and where to take it.
This one is about finding the frames you care about in it. Every count below comes from tshark 4.6.5,
Wireshark&rsquo;s command-line twin, run against the lab&rsquo;s reference capture of r1&rsquo;s switch port: 64 frames of ARP,
DNS, pings, HTTP, HTTPS and the switch&rsquo;s own STP messages, from about four seconds of a quiet network.</p>
<hr>
<h2 id="what-is-a-capture-filter">What is a capture filter?</h2>
<p>A capture filter is written in <strong>BPF</strong>, the <em>Berkeley Packet Filter</em> language: the same one tcpdump uses.
It&rsquo;s old, small and fast, because of where it runs. The capture library compiles your filter into a tiny
program, and the operating system&rsquo;s kernel runs that program on every frame before it&rsquo;s handed to the
capture tool. A frame the program rejects is never copied out of the kernel, never written to disk, and can
never be recovered.</p>
<p>You can see the program. <code>tcpdump -d 'udp port 53'</code> prints it, and it starts like this:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">(000) ldh      [12]
</span></span><span class="line"><span class="cl">(001) jeq      #0x86dd          jt 2	jf 8
</span></span><span class="line"><span class="cl">...
</span></span><span class="line"><span class="cl">(018) ret      #262144
</span></span><span class="line"><span class="cl">(019) ret      #0
</span></span></code></pre></div><p>Load the two bytes at offset 12 of the frame (the Ethernet type), check whether they say IPv6, and so on.
<code>ret #262144</code> means <em>keep up to 262,144 bytes of this frame</em>, and <code>ret #0</code> means <em>drop it</em>. That&rsquo;s the whole
idea: a capture filter is arithmetic on header bytes at fixed positions.</p>
<p>Which is also its limit. BPF knows addresses, ports and protocol numbers, and that&rsquo;s about it:</p>
<table>
  <thead>
      <tr>
          <th>BPF capture filter</th>
          <th>Keeps</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>host 192.168.2.10</code></td>
          <td>frames to or from that address</td>
      </tr>
      <tr>
          <td><code>src host 192.168.1.20</code></td>
          <td>frames from it</td>
      </tr>
      <tr>
          <td><code>udp port 53</code></td>
          <td>UDP to or from port 53</td>
      </tr>
      <tr>
          <td><code>tcp port 443</code></td>
          <td>TCP to or from port 443</td>
      </tr>
      <tr>
          <td><code>ether host aa:aa:aa:11:11:11</code></td>
          <td>frames to or from that MAC address</td>
      </tr>
      <tr>
          <td><code>arp</code>, <code>not arp</code></td>
          <td>ARP, or everything else</td>
      </tr>
      <tr>
          <td><code>host 192.168.2.10 and tcp port 80</code></td>
          <td>both must be true (<code>and</code>, <code>or</code>, <code>not</code>)</td>
      </tr>
  </tbody>
</table>
<p>BPF can&rsquo;t say <em>DNS</em>. It can say <em>port 53</em> and trust that whatever uses port 53 is DNS. It can&rsquo;t say
<em>the HTTP request for /login</em>, because that&rsquo;s inside the payload, at no fixed offset. If the answer you need
is past the headers, a capture filter can&rsquo;t find it.</p>
<p>One BPF detail catches people. <code>host 192.168.1.1</code> matches the address wherever BPF knows to look, and that
includes ARP. On the reference capture it keeps 4 frames, all of them ARP: the laptop and the printer asking
for r1, and r1 answering. Write <code>ip host 192.168.1.1</code> and it keeps 0, because nothing in that capture sent an
IP packet to or from r1 itself. The pings went <em>through</em> r1, to a server further away.</p>
<hr>
<h2 id="what-is-a-display-filter">What is a display filter?</h2>
<p>A display filter is written in Wireshark&rsquo;s own language, and it works on the decoded capture. Every line in
the packet details pane is a <strong>field</strong> with a name, made by the <strong>dissector</strong>, the part of Wireshark that
decodes one protocol. Click a field and its name appears in the status bar: <code>ip.src</code>, <code>dns.qry.name</code>,
<code>tcp.flags.syn</code>, <code>tls.handshake.extensions_server_name</code>. tshark 4.6.5 lists 269,301 of them, from
every protocol it can decode.</p>
<p>So a display filter can ask anything a dissector understood:</p>
<table>
  <thead>
      <tr>
          <th>Display filter</th>
          <th>On the reference capture</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>dns</code></td>
          <td>12 frames: every frame with a DNS layer</td>
      </tr>
      <tr>
          <td><code>ip.src == 192.168.1.20</code></td>
          <td>3: the printer&rsquo;s own packets</td>
      </tr>
      <tr>
          <td><code>ip.addr == 192.168.1.20</code></td>
          <td>6: the printer as source <strong>or</strong> destination</td>
      </tr>
      <tr>
          <td><code>ip.addr == 192.168.2.0/24</code></td>
          <td>57: anything to or from the server network</td>
      </tr>
      <tr>
          <td><code>tcp.port in {80, 443}</code></td>
          <td>33: both web connections</td>
      </tr>
      <tr>
          <td><code>http.host == &quot;web.lab&quot;</code></td>
          <td>1: the HTTP request that asked for that name</td>
      </tr>
      <tr>
          <td><code>tcp.flags.syn == 1 &amp;&amp; tcp.flags.ack == 0</code></td>
          <td>2: the first packet of each TCP connection</td>
      </tr>
      <tr>
          <td><code>tls.handshake.extensions_server_name == &quot;web.lab&quot;</code></td>
          <td>1: the HTTPS request&rsquo;s name, before encryption</td>
      </tr>
  </tbody>
</table>
<p>That last one is a field inside an encrypted connection&rsquo;s handshake, at a position that changes from packet
to packet. No capture filter could reach it. A display filter just names it.</p>
<p>A filter with a typo is an error, not a silent empty result, which is good:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">tshark: &#34;ip.adr&#34; is not a valid protocol or protocol field.
</span></span><span class="line"><span class="cl">    ip.adr == 1.2.3.4
</span></span><span class="line"><span class="cl">    ^~~~~~
</span></span></code></pre></div><p>In Wireshark the filter bar turns red instead, and green when the filter makes sense.</p>
<table>
  <thead>
      <tr>
          <th></th>
          <th>Capture filter</th>
          <th>Display filter</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>Language</td>
          <td>BPF (tcpdump&rsquo;s)</td>
          <td>Wireshark&rsquo;s display filter language</td>
      </tr>
      <tr>
          <td>Runs</td>
          <td>in the kernel, while capturing</td>
          <td>in Wireshark, on a saved capture</td>
      </tr>
      <tr>
          <td>Can see</td>
          <td>header fields at fixed offsets</td>
          <td>every field any dissector decoded</td>
      </tr>
      <tr>
          <td>What it does to rejected frames</td>
          <td>never saved</td>
          <td>hidden, still in the file</td>
      </tr>
      <tr>
          <td>Undo</td>
          <td>no</td>
          <td>clear the filter</td>
      </tr>
      <tr>
          <td>Cost</td>
          <td>very cheap, so it keeps up with busy links</td>
          <td>needs the whole capture decoded first</td>
      </tr>
  </tbody>
</table>
<hr>
<h2 id="how-do-contains-and-matches-work">How do contains and matches work?</h2>
<p>Two operators search for text rather than compare values.</p>
<p><code>contains</code> looks for an exact sequence of bytes or characters in a field, or in the whole frame.
<code>frame contains &quot;web&quot;</code> keeps 13 frames on the reference capture: the DNS questions and answers for <code>web.lab</code>,
the HTTP request (its Host header), the page that came back, and the TLS Client Hello, whose server name is
sent in clear text. It&rsquo;s case-sensitive: <code>dns.qry.name contains &quot;PRINT&quot;</code> finds nothing, because the name is
<code>printer.lab</code>.</p>
<p><code>matches</code> takes a regular expression, and it&rsquo;s case-insensitive by default.
<code>dns.qry.name matches &quot;^PRINT&quot;</code> finds both printer frames. Put <code>(?-i)</code> at the front of the expression and it
becomes case-sensitive again, and finds none.</p>
<p>Use <code>contains</code> when you know the exact text, and <code>matches</code> when you need a pattern or don&rsquo;t trust the case.</p>
<hr>
<h2 id="why-is-ipaddr-a-two-way-field">Why is ip.addr a two-way field?</h2>
<p>A frame has one IP source and one IP destination, but Wireshark also gives it <code>ip.addr</code>, a field that appears
<strong>twice</strong>: once with the source and once with the destination. <code>ip.addr == 192.168.1.20</code> is true if <em>either</em>
copy matches, which is exactly what you want for &ldquo;everything to or from the printer&rdquo;. <code>eth.addr</code>, <code>tcp.port</code>
and <code>udp.port</code> work the same way.</p>
<p>It&rsquo;s the most useful shortcut in Wireshark, and it&rsquo;s the reason for the most famous trap in it.</p>
<hr>
<h2 id="what-does-ipaddr--x-really-match">What does ip.addr != x really match?</h2>
<p>The question is what <em>not equal</em> means for a field with two values. Is <code>ip.addr != 192.168.2.10</code> true when
<em>any</em> copy is different, or only when <em>all</em> of them are?</p>
<p>For years, Wireshark used <em>any</em>. Every packet between the laptop and the web server has one address that isn&rsquo;t
the web server&rsquo;s, so <code>ip.addr != 192.168.2.10</code> matched nearly everything, including all the traffic you were
trying to remove. Every guide warned you never to write it.</p>
<p><strong>Wireshark 3.6 changed that.</strong> Its <a href="https://www.wireshark.org/docs/relnotes/wireshark-3.6.0.html">release notes</a>
say &ldquo;a != b&rdquo; now has the &ldquo;same meaning as !(a == b)&rdquo;, and kept the old meaning available as <code>~=</code>. Wireshark
4.0 <a href="https://www.wireshark.org/docs/relnotes/wireshark-4.0.0.html">deprecated <code>~=</code></a> in favor of <code>!==</code>. In
tshark 4.6.5, <code>~=</code> is a syntax error.</p>
<p>So the old warning is out of date. But there&rsquo;s one difference left, and it&rsquo;s the one that bites now. Here are
the three versions on the reference capture:</p>
<table>
  <thead>
      <tr>
          <th>Filter</th>
          <th>Frames</th>
          <th>What it means</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>ip.addr != 192.168.2.10</code></td>
          <td>20</td>
          <td>frames that <strong>have</strong> an IP address, none of them web&rsquo;s</td>
      </tr>
      <tr>
          <td><code>!(ip.addr == 192.168.2.10)</code></td>
          <td>27</td>
          <td>every frame that doesn&rsquo;t mention web, IP or not</td>
      </tr>
      <tr>
          <td><code>ip.addr !== 192.168.2.10</code></td>
          <td>57</td>
          <td>the old meaning: any address that isn&rsquo;t web&rsquo;s, so every IP frame</td>
      </tr>
  </tbody>
</table>
<p>The difference between 20 and 27 is 4 ARP frames and 3 STP frames. They have no <code>ip.addr</code> field at all, and a
comparison on a field that isn&rsquo;t there is false, so <code>!=</code> drops them. <code>!(ip.addr == ...)</code> asks the positive
question first (<em>is web in it?</em>: no, for ARP), then turns the answer round, so they stay.</p>
<p>That&rsquo;s usually what you want when you say &ldquo;everything except web&rdquo;. The habit to build: write the filter for
the thing you want to hide, then put <code>!( )</code> round it.</p>
<hr>
<h2 id="does--bind-tighter-than-">Does &amp;&amp; bind tighter than ||?</h2>
<p>Yes, since <strong>Wireshark 4.0</strong>, whose release notes say logical AND &ldquo;now has higher precedence than logical OR,
in line with most programming languages&rdquo;. Before 4.0 it didn&rsquo;t, so a filter that mixes the two can give a
different answer on an older version.</p>
<p>It changes results. Take <em>DNS or ARP, sent by the laptop</em>:</p>
<table>
  <thead>
      <tr>
          <th>Filter</th>
          <th>Frames</th>
          <th>Read as</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>dns || arp &amp;&amp; eth.src == aa:aa:aa:11:11:11</code></td>
          <td>13</td>
          <td>DNS from anyone, or ARP from the laptop</td>
      </tr>
      <tr>
          <td><code>(dns || arp) &amp;&amp; eth.src == aa:aa:aa:11:11:11</code></td>
          <td>6</td>
          <td>what you meant</td>
      </tr>
  </tbody>
</table>
<p>Use brackets whenever you mix them. They make a filter mean the same thing in every version, and to every
person who reads it.</p>
<hr>
<h2 id="how-much-should-you-capture">How much should you capture?</h2>
<p>Capture wide, filter narrow. You can always hide a frame you saved. You can never get back a frame a capture
filter threw away, and the clue you didn&rsquo;t expect is usually in the traffic you&rsquo;d have thought irrelevant: an
ARP storm, a second DHCP server, a retransmission from a host you weren&rsquo;t watching.</p>
<p>Capture filters still have their place, for three reasons:</p>
<ul>
<li><strong>Volume.</strong> A busy link fills a disk quickly. A filter keeps the file to what you need.</li>
<li><strong>Load.</strong> Capturing every frame on a fast link can drop frames on a slow capture machine. Filtering in the
kernel is the cheapest way to keep up.</li>
<li><strong>Privacy and policy.</strong> Sometimes you&rsquo;re only allowed to keep one application&rsquo;s traffic. The safest copy of
someone else&rsquo;s data is the one you never saved.</li>
</ul>
<p>Two more settings decide how much a capture keeps.</p>
<h3 id="snap-length">Snap length</h3>
<p>The <strong>snap length</strong> is how many bytes of each frame are saved. The lab&rsquo;s files save up to 262,144 bytes per
frame, the same number as that <code>ret #262144</code> in the BPF program: in practice, all of every frame. Set it
small and you keep the headers and drop the payload. Here&rsquo;s the HTTP request from the reference capture,
cut to 54 bytes per frame with <code>editcap -s 54</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">Frame 24: Packet, 137 bytes on wire (1096 bits), 54 bytes captured (432 bits) on interface sw1:eth1, id 0
</span></span><span class="line"><span class="cl">   24 2.382133001 192.168.1.175 → 192.168.2.10 TCP 137 50124 → 80 [PSH, ACK] Seq=1 Ack=1 Win=502 Len=71[Packet size limited during capture]
</span></span></code></pre></div><p>The addresses and ports survived, the <code>GET</code> is gone, and Wireshark says so. A small snap length is a good
way to capture <em>who talked to whom</em> for a long time without keeping <em>what they said</em>. It&rsquo;s a bad idea when
you&rsquo;re troubleshooting the application.</p>
<h3 id="ring-buffers">Ring buffers</h3>
<p>A <strong>ring buffer</strong> is a set of capture files used in turn: write file 1 until it reaches a size or an age,
then file 2, and when the last one is full, overwrite the oldest. tshark&rsquo;s <code>-b</code> option does it (<code>filesize</code>,
<code>duration</code>, <code>files</code>). It&rsquo;s how you wait for an intermittent problem: leave a capture running for days with,
say, ten files of 100 MB, and when the problem happens, stop it. The last gigabyte of traffic before the
problem is on disk, and the disk never filled.</p>
<hr>
<h2 id="what-i-see-go-wrong">What I see go wrong</h2>
<p><strong>A capture filter that removed the answer.</strong> <code>host 192.168.2.10</code> looks like a sensible filter for &ldquo;web&rsquo;s
traffic&rdquo;, until the problem turns out to be the client&rsquo;s ARP or DNS, which never involved that address. If you
filter at capture time, filter on the client, not the server, and keep it loose.</p>
<p><strong><code>ip.addr != x</code> from an old guide.</strong> It&rsquo;s no longer the disaster those guides describe, but it still drops
every frame without an IP address. When you want &ldquo;everything except&rdquo;, use <code>!(field == value)</code>.</p>
<p><strong>Mixing <code>&amp;&amp;</code> and <code>||</code> without brackets.</strong> It works, in Wireshark 4.0 and later, by precedence. Someone on an
older version, or someone reading it quickly, gets a different answer.</p>
<p><strong>Treating BPF and display filters as one language.</strong> <code>port 53</code> is BPF; <code>udp.port == 53</code> is a display filter.
The filter bar rejects the first, and tcpdump rejects the second. Know which box you&rsquo;re typing in.</p>
<p><strong>Trusting an empty result.</strong> A display filter that shows nothing might mean the traffic isn&rsquo;t there, or
that you asked for a field from the wrong protocol. Before you conclude anything, take a frame you <em>know</em>
should match and check that the filter matches it.</p>
<hr>
<h2 id="try-it">Try it</h2>
<p><a href="/labs/wireshark-101/#chapter-2">Chapter 2 of the lab</a> gives you the same reference capture used here. You
write display filters that have to pick exactly the right frames, see the <code>!=</code> difference for yourself, and
then write capture filters in BPF and find out what they threw away.</p>
<h2 id="whats-next">What&rsquo;s next</h2>
<p><a href="/lessons/wireshark-101/arp/">Lesson 3</a> uses those filters on the first protocol every conversation needs:
ARP, the question a host asks before it can send anything to its neighbor.</p>
]]></content:encoded></item><item><title>What a Packet Capture Shows You (Wireshark 101, Lesson 1)</title><link>https://networkdoodles.com/lessons/wireshark-101/what-a-capture-shows/</link><pubDate>Mon, 16 Nov 2026 09:00:00 -0500</pubDate><dc:creator>David Henderson</dc:creator><guid>https://networkdoodles.com/lessons/wireshark-101/what-a-capture-shows/</guid><category>wireshark</category><category>packet-capture</category><category>networking-basics</category><category>ethernet</category><description>What a packet capture records, how Wireshark's three panes map to the layers, pcap vs pcapng, and why where you capture decides what you can see.</description><content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[<p>A packet capture is a recording of every frame that crossed one point in the network while the recording
ran: each one bit for bit, with the time it was seen. Wireshark opens that recording and decodes it, layer by
layer. The most important thing to know about a capture is that it only shows <strong>its own point</strong>. Pick the
wrong cable and you see nothing, and <em>nothing</em> looks exactly like <em>it never happened</em>.</p>
<p><a href="/lessons/networking-101/">Networking 101</a> explained what MAC addresses, IP addresses, switches and routers
are. This course shows you what they actually put on the wire. This first lesson is about the tool itself:
what&rsquo;s in a capture, how Wireshark lays it out, what the timestamps mean, the two file formats, and the
question every capture starts with: <em>where am I listening?</em></p>
<hr>
<h2 id="what-is-a-packet-capture">What is a packet capture?</h2>
<p>Everything you&rsquo;ve used to look at a network so far is a summary. <code>ip neigh</code> shows what a host <em>concluded</em>
about its neighbors. <code>show mac address-table</code> shows what a switch <em>learned</em>. A routing table shows what a
router <em>decided</em>. They&rsquo;re all useful, and they&rsquo;re all someone&rsquo;s interpretation.</p>
<p>A capture isn&rsquo;t. It&rsquo;s the frames themselves, in the order they were seen, with nothing left out and nothing
summarized. When a show command and a capture disagree, the capture is the one that&rsquo;s telling you what
happened. That&rsquo;s why engineers reach for it when everything else says the network is fine and the
application says it isn&rsquo;t.</p>
<p>Two words before we go on. A <strong>frame</strong> is a message in its layer 2 envelope: an Ethernet header with a <em>to</em>
and a <em>from</em> MAC address, and something inside. A <strong>packet</strong> is, strictly, the layer 3 thing inside it, with
IP addresses. Wireshark and everyone else say &ldquo;packet&rdquo; for both, and so will I, except where the difference
matters.</p>
<p>Each frame in a capture is stored with a few extra facts the capture added:</p>
<ul>
<li><strong>when</strong> it was seen, to the nanosecond in the lab&rsquo;s files;</li>
<li><strong>where</strong> it was seen: the name of the interface it was captured on;</li>
<li><strong>how long</strong> it was on the wire, and how many of those bytes were saved (more on that in
<a href="/lessons/wireshark-101/capture-vs-display-filters/">lesson 2</a>).</li>
</ul>
<p>Then the bytes, exactly as they were.</p>
<hr>
<h2 id="how-do-wiresharks-three-panes-map-to-the-layers">How do Wireshark&rsquo;s three panes map to the layers?</h2>
<p>Open a capture and Wireshark shows three panes, top to bottom.</p>
<table>
  <thead>
      <tr>
          <th>Pane</th>
          <th>What it shows</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><strong>Packet list</strong></td>
          <td>one line per frame: number, time, source, destination, protocol, length, a one-line summary</td>
      </tr>
      <tr>
          <td><strong>Packet details</strong></td>
          <td>the frame you clicked, one layer per line, each one opens to show its fields</td>
      </tr>
      <tr>
          <td><strong>Packet bytes</strong></td>
          <td>the same frame as raw hex. Click a field in the details and its bytes light up</td>
      </tr>
  </tbody>
</table>
<p>The packet list is for finding things. Here&rsquo;s the start of the lab&rsquo;s first capture, taken on the laptop&rsquo;s
cable while it pings the printer, as tshark (Wireshark&rsquo;s command-line twin, version 4.6.5 here) prints it:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">    3 2.177114918 aa:aa:aa:11:11:11 → Broadcast    ARP 42 Who has 192.168.1.20? Tell 192.168.1.175
</span></span><span class="line"><span class="cl">    4 2.177311459 aa:aa:aa:22:22:22 → aa:aa:aa:11:11:11 ARP 42 192.168.1.20 is at aa:aa:aa:22:22:22
</span></span><span class="line"><span class="cl">    5 2.177313709 192.168.1.175 → 192.168.1.20 ICMP 98 Echo (ping) request  id=0x0926, seq=1/256, ttl=64
</span></span><span class="line"><span class="cl">    6 2.177450834 192.168.1.20 → 192.168.1.175 ICMP 98 Echo (ping) reply    id=0x0926, seq=1/256, ttl=64 (request in 5)
</span></span></code></pre></div><p>Two things in that list are worth knowing straight away. The <strong>Protocol</strong> column is the <em>highest</em> layer
Wireshark recognized, not the only one: a ping is ICMP inside IP inside Ethernet. And the source and
destination columns show IP addresses when there&rsquo;s an IP layer, and fall back to MAC addresses when there
isn&rsquo;t, which is why the ARP lines look different.</p>
<p>The details pane is where the layers from Networking 101&rsquo;s
<a href="/lessons/networking-101/task-1-2/">OSI lesson</a> become visible. Click frame 5 and you get:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">Frame 5: Packet, 98 bytes on wire (784 bits), 98 bytes captured (784 bits) on interface laptop:eth1, id 0
</span></span><span class="line"><span class="cl">Ethernet II, Src: aa:aa:aa:11:11:11 (aa:aa:aa:11:11:11), Dst: aa:aa:aa:22:22:22 (aa:aa:aa:22:22:22)
</span></span><span class="line"><span class="cl">Internet Protocol Version 4, Src: 192.168.1.175, Dst: 192.168.1.20
</span></span><span class="line"><span class="cl">Internet Control Message Protocol
</span></span></code></pre></div><table>
  <thead>
      <tr>
          <th>Line</th>
          <th>Layer</th>
          <th>What it holds</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><strong>Frame</strong></td>
          <td>none: Wireshark&rsquo;s own notes</td>
          <td>when and where the capture saw it, its length</td>
      </tr>
      <tr>
          <td><strong>Ethernet II</strong></td>
          <td>2, data link</td>
          <td>the MAC addresses, and the type of what&rsquo;s inside</td>
      </tr>
      <tr>
          <td><strong>Internet Protocol Version 4</strong></td>
          <td>3, network</td>
          <td>the IP addresses, the TTL, the protocol inside</td>
      </tr>
      <tr>
          <td><strong>Internet Control Message Protocol</strong></td>
          <td>3, carried in IP</td>
          <td>the ping itself</td>
      </tr>
  </tbody>
</table>
<p>For a web request you&rsquo;d see TCP (layer 4) under IP, and HTTP under that. The order is the order of the
envelopes on the wire: outermost first. Reading top to bottom is unwrapping the package.</p>
<p>The <strong>Frame</strong> line is the one people misread. It isn&rsquo;t a protocol and nothing on the wire sent it. It&rsquo;s the
capture&rsquo;s own label on the frame: the arrival time, the interface, the lengths. You&rsquo;ll use it more than you&rsquo;d
think, because &ldquo;on interface laptop:eth1&rdquo; is the answer to &ldquo;where was this seen?&rdquo;.</p>
<p>The bytes pane is the honest one. The details pane is Wireshark&rsquo;s <em>interpretation</em> of those bytes, made by a
<strong>dissector</strong>: the piece of Wireshark that knows how to decode one protocol. Dissectors are very good, but
when one guesses wrong (a protocol on an unusual port, say), the bytes are still right.</p>
<hr>
<h2 id="what-do-the-timestamps-mean">What do the timestamps mean?</h2>
<p>Every frame gets a timestamp from the machine doing the capturing, at the moment the capture saw it. Not when
the sender built it, and not when the receiver read it. That distinction matters as soon as you capture at
more than one place, because two capture points have two clocks.</p>
<p>Wireshark shows the time in the packet list as <strong>seconds since the first frame</strong> by default. You can switch
it (View › Time Display Format) to the time of day, or to the time since the previous frame, which is what
you want when you&rsquo;re looking for a pause. The real time is always in the Frame line:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">Arrival Time: Nov 11, 2026 01:46:20.722588928 UTC
</span></span></code></pre></div><p>Timestamps are where a capture turns into measurements. In the capture above, the printer&rsquo;s ARP reply came
196.5 microseconds after the laptop&rsquo;s question (frame 3 to frame 4), and the first ping&rsquo;s reply came 137
microseconds after the request (frame 5 to frame 6). Those numbers are from the laptop&rsquo;s cable, so they include
the switch, the printer&rsquo;s own processing, and the switch again. A ping in the lab&rsquo;s second capture, to a web
server two routers away, took 566 microseconds, measured on the switch&rsquo;s port towards the first router. Time
between a question and its answer, measured on the asker&rsquo;s cable, is the round trip that user actually
experienced. Measured anywhere else, it&rsquo;s the round trip from that point onwards.</p>
<hr>
<h2 id="whats-the-difference-between-pcap-and-pcapng">What&rsquo;s the difference between pcap and pcapng?</h2>
<p>Captures are saved in one of two file formats.</p>
<p><strong>pcap</strong> is the original, from the tcpdump world. It&rsquo;s simple: a short file header, then one record per
frame. <strong>pcapng</strong> (<em>pcap next generation</em>) is what Wireshark saves by default today, and what the lab&rsquo;s
captures use. It&rsquo;s built from blocks, so it can say much more about the capture.</p>
<table>
  <thead>
      <tr>
          <th></th>
          <th>pcap</th>
          <th>pcapng</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>Interfaces per file</td>
          <td>one link type, no names</td>
          <td>several, each with its name</td>
      </tr>
      <tr>
          <td>Timestamp resolution</td>
          <td>microseconds (the classic format)</td>
          <td>per interface; nanoseconds in the lab&rsquo;s files</td>
      </tr>
      <tr>
          <td>Comments</td>
          <td>no</td>
          <td>on the file and on each frame</td>
      </tr>
      <tr>
          <td>Which program made it</td>
          <td>no</td>
          <td>yes</td>
      </tr>
      <tr>
          <td>Who can read it</td>
          <td>nearly everything</td>
          <td>Wireshark, tshark, tcpdump&rsquo;s libpcap and most current tools</td>
      </tr>
  </tbody>
</table>
<p>Here&rsquo;s what that means in practice. The lab&rsquo;s capture of two switch ports, converted from pcapng to pcap with
Wireshark&rsquo;s <code>editcap</code>, went from 1,524 bytes to 1,186. The bytes saved are the bad news: the interface names
<code>sw1:eth1</code> and <code>sw1:eth3</code> were gone (every frame now came from one nameless interface), and every timestamp
lost its last three digits:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">pcapng: 1794361588.669028918
</span></span><span class="line"><span class="cl">pcap:   1794361588.669028000
</span></span></code></pre></div><p>The interface name is the bit you&rsquo;ll miss. When one file holds two capture points, that name is the only way
to tell them apart. So keep captures in pcapng, and only convert when a tool demands pcap.</p>
<hr>
<h2 id="why-does-where-you-capture-decide-what-you-see">Why does where you capture decide what you see?</h2>
<p>This is the section to remember. A capture records one point in the network, and what crosses that point
depends on what&rsquo;s around it.</p>
<h3 id="a-hosts-own-port">A host&rsquo;s own port</h3>
<p>Capture on a host&rsquo;s cable, as the lab&rsquo;s first task does on the laptop, and you see everything that host sends
and everything sent to it, plus the network&rsquo;s broadcasts. That&rsquo;s the easy case, because the host is one end of
every conversation you&rsquo;re looking at.</p>
<h3 id="a-switch-port-other-peoples-unicast-isnt-there">A switch port: other people&rsquo;s unicast isn&rsquo;t there</h3>
<p>Networking 101&rsquo;s <a href="/lessons/networking-101/task-2-1/">lesson on switches</a> gave the rule. A switch sends a
frame for one MAC address out of <strong>one port</strong>, the one where it learned that address. It sends a broadcast
out of <strong>every</strong> port. It floods a frame for an address it hasn&rsquo;t learned yet out of every port too, but only
until it learns it.</p>
<p>So a capture on a switch port sees that port&rsquo;s own host, the broadcasts, and the occasional flood. Nothing
else. The lab shows it with two captures taken at once, on sw1&rsquo;s port towards r1 and its port towards the
printer, while the laptop pings a server on another network:</p>
<table>
  <thead>
      <tr>
          <th>Seen on</th>
          <th>ARP</th>
          <th>ICMP (pings and replies)</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>r1&rsquo;s port, <code>sw1:eth1</code></td>
          <td>the laptop&rsquo;s request and r1&rsquo;s reply</td>
          <td>all 6</td>
      </tr>
      <tr>
          <td>the printer&rsquo;s port, <code>sw1:eth3</code></td>
          <td>the laptop&rsquo;s request only</td>
          <td>0</td>
      </tr>
  </tbody>
</table>
<p>The printer&rsquo;s cable carried exactly one frame of that conversation: the ARP request, because it was a
broadcast. r1&rsquo;s reply was addressed to the laptop alone, and the pings were addressed to r1&rsquo;s MAC address, so
the switch never sent them anywhere near the printer. That&rsquo;s the switch doing its job.</p>
<p>It&rsquo;s also why &ldquo;I&rsquo;ll plug my laptop into a spare switch port and capture&rdquo; doesn&rsquo;t work. You&rsquo;ll see broadcasts,
your own traffic, and very little else. In the days of hubs, which repeated everything to everyone, it did
work. Switches ended that.</p>
<h3 id="mirror-ports-and-taps">Mirror ports and taps</h3>
<p>Real networks solve this in two ways.</p>
<p>A <strong>mirror port</strong> (Cisco calls it <strong>SPAN</strong>, for <em>switched port analyzer</em>) is a switch feature: you tell the
switch to send a copy of everything on one or more ports, or a whole VLAN, out of another port, where your
capture machine sits. It costs nothing but configuration. The catch is that it&rsquo;s the switch&rsquo;s best effort:
a mirror port can only send as much as its own link carries, so if you mirror more traffic than that, some
copies are dropped, and your capture is missing frames the real network delivered.</p>
<p>A <strong>tap</strong> is a small device you put in the cable itself. It passes the traffic through and copies it to a
monitoring port. It sees exactly what was on the wire, but you have to break the cable to install it, which
is why taps go in during planned work and stay there.</p>
<p>The lab does something closer to a tap: <code>containerlab capture</code> records a port from outside the node, so
nothing on the laptop or the switch knows it&rsquo;s being watched, and you can capture any cable in the network.</p>
<h3 id="inside-a-host-versus-on-the-wire">Inside a host versus on the wire</h3>
<p>There&rsquo;s one more place to capture: on the host itself, with Wireshark installed on the laptop. That&rsquo;s the
most common capture there is, and it comes with a surprise.</p>
<p>Modern network cards do work the operating system used to do. Two kinds of this <strong>offload</strong> change what a
capture on the host shows. With <em>checksum offload</em>, the card fills in the IP and TCP checksums just before
the frame leaves, so a capture of outgoing traffic, taken before the card, sees checksums that haven&rsquo;t been
calculated yet. With <em>segmentation offload</em>, the operating system hands the card one big block of data and
the card cuts it into frames, so the capture shows &ldquo;packets&rdquo; larger than the network&rsquo;s MTU, which no
real cable could have carried. Wireshark&rsquo;s own
<a href="https://wiki.wireshark.org/CaptureSetup/Offloading">page on offloading</a> explains both.</p>
<p>Neither is a fault. Both are the capture being honest about where it is: above the card, not on the wire.
If a host capture shows bad checksums on everything it sent, or 1,500-byte limits apparently ignored,
check offloads before you suspect the network.</p>
<table>
  <thead>
      <tr>
          <th>Capture point</th>
          <th>Sees</th>
          <th>Doesn&rsquo;t see</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>on the host itself</td>
          <td>what that host sent and received, as the OS saw it</td>
          <td>other hosts&rsquo; traffic; the final checksums and frame sizes</td>
      </tr>
      <tr>
          <td>a switch port</td>
          <td>that port&rsquo;s host, broadcasts, floods</td>
          <td>other hosts&rsquo; unicast</td>
      </tr>
      <tr>
          <td>a mirror (SPAN) port</td>
          <td>copies of the ports or VLAN you chose</td>
          <td>copies the switch dropped when the mirror was full</td>
      </tr>
      <tr>
          <td>a tap</td>
          <td>exactly what was on that cable</td>
          <td>anything on another cable</td>
      </tr>
  </tbody>
</table>
<hr>
<h2 id="what-i-see-go-wrong">What I see go wrong</h2>
<p><strong>Capturing in the wrong place, and believing the silence.</strong> Someone captures on the server, sees no
request, and concludes the client never sent one. Maybe it didn&rsquo;t. Or maybe it was dropped on the way, or
went to a different server. A capture can prove a frame crossed a point. Its absence only proves it didn&rsquo;t
cross <em>that</em> point. The fix is a second capture closer to the sender, and lesson 8 of this course is built
around it.</p>
<p><strong>Expecting a switch port to show everyone.</strong> It won&rsquo;t, as the table above shows. If you need someone else&rsquo;s
traffic, capture on their port, on the uplink they use, or on a mirror of it.</p>
<p><strong>Comparing timestamps from two machines.</strong> Two captures taken on two computers use two clocks. A few
milliseconds&rsquo; difference between them is often just the clocks disagreeing. Compare times <em>within</em> one capture,
or use one machine that records both points into one file, which is what the lab does.</p>
<p><strong>Reading checksum errors as damage.</strong> On a host capture, outgoing frames with &ldquo;incorrect&rdquo; checksums are
almost always checksum offload. Damaged frames on a real wire usually don&rsquo;t reach a capture at all, because
the receiving card throws them away.</p>
<p><strong>Saving in a format that loses the answer.</strong> A multi-point capture converted to pcap has lost the one field
that says which point each frame came from.</p>
<hr>
<h2 id="try-it">Try it</h2>
<p><a href="/labs/wireshark-101/#chapter-1">Chapter 1 of the lab</a> has you take your first capture on the laptop&rsquo;s cable,
open it in Wireshark in your browser, read the three panes, and then capture two switch ports at once to see
for yourself which cable carried which frames.</p>
<h2 id="whats-next">What&rsquo;s next</h2>
<p>A busy network gives you thousands of frames a second.
<a href="/lessons/wireshark-101/capture-vs-display-filters/">Lesson 2</a> is about cutting that down: capture filters,
which decide what&rsquo;s saved at all, and display filters, which decide what you look at, and the one small
operator whose meaning changed under everyone&rsquo;s feet.</p>
]]></content:encoded></item><item><title>The Shell and the Filesystem: Reading Any Path (Linux CLI 101, Lesson 1)</title><link>https://networkdoodles.com/lessons/linux-cli-101/shell-and-filesystem/</link><pubDate>Mon, 16 Nov 2026 09:00:00 -0500</pubDate><dc:creator>David Henderson</dc:creator><guid>https://networkdoodles.com/lessons/linux-cli-101/shell-and-filesystem/</guid><category>linux</category><category>networking-basics</category><category>network-automation</category><category>capeos</category><description>What a shell is, how a command line is built, and how Linux finds any file in one tree: absolute and relative paths, ~, . and .., hidden files and help.</description><content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[<p>A <strong>shell</strong> is the program that reads what you type at the prompt, works out which command you mean, and runs
it. On almost every Linux system, including this course&rsquo;s lab, the shell is <strong>Bash</strong>. Every file Linux can see
sits in <strong>one tree</strong> that starts at <code>/</code>, the root. A <strong>path</strong> is the route through that tree to one file:
<strong>absolute</strong> if it starts at <code>/</code>, <strong>relative</strong> if it starts from the folder you&rsquo;re in now.</p>
<p>Once you can read a path and know where it points, the rest of the command line gets much easier. This lesson
covers the shell, the prompt, how a command line is put together, the tree and its paths, hidden files, where
Linux keeps things, and how to get help.</p>
<hr>
<h2 id="why-does-a-network-engineer-need-linux">Why does a network engineer need Linux?</h2>
<p>Because you&rsquo;re already using it, even if the box says something else. The lab&rsquo;s router, r1, runs
<a href="/posts/capeos-lab-network-os/">CapeOS</a>, and it looks like a router from its own CLI:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">CapeOS Software (CO-BOLT), Version 0.11.2, LAB USE ONLY
</span></span><span class="line"><span class="cl">CapeOS is open-source lab software. Not for production use.
</span></span></code></pre></div><p>Ask the container underneath the same question Linux answers, and it&rsquo;s Debian:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">PRETTY_NAME=&#34;Debian GNU/Linux 12 (bookworm)&#34;
</span></span></code></pre></div><p>That&rsquo;s the pattern across the industry. Nokia&rsquo;s <a href="https://learn.srlinux.dev/">SR Linux</a>, which other labs on
this site use, runs on a standard Linux kernel. Many network operating systems run on a Linux kernel and give you a way
to a Linux shell underneath. The automation tools are Linux-first too: Ansible&rsquo;s control node, Python
scripts, containerlab, Git servers and the CI runners that test your changes all run there. The codespace
you&rsquo;ll use for every lab on this site is a Linux container, Debian 12 in this one.</p>
<p>You don&rsquo;t need to become a Linux administrator. You need to be at home in a shell: find a file, read it, run a
tool, and understand the error when it doesn&rsquo;t work.</p>
<hr>
<h2 id="what-is-the-difference-between-a-terminal-and-a-shell">What is the difference between a terminal and a shell?</h2>
<p>They&rsquo;re two programs that people often mix up.</p>
<table>
  <thead>
      <tr>
          <th></th>
          <th>What it is</th>
          <th>In the lab</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><strong>Terminal</strong></td>
          <td>the window: it draws text, takes your keystrokes and passes them on</td>
          <td>the Terminal panel in VS Code</td>
      </tr>
      <tr>
          <td><strong>Shell</strong></td>
          <td>the program behind the window that reads each line and runs it</td>
          <td>Bash</td>
      </tr>
  </tbody>
</table>
<p>The terminal doesn&rsquo;t understand commands. It hands what you type to the shell, and shows what the shell and
the commands print back. You can swap one without the other: the same Bash runs in VS Code&rsquo;s panel, in a
Mac&rsquo;s Terminal app, or over SSH from anywhere.</p>
<p>Bash is the <strong>Bourne Again SHell</strong>, from the GNU project. It&rsquo;s a newer take on the original Unix shell by
Stephen Bourne, which is where the name comes from. The versions in the lab:</p>
<table>
  <thead>
      <tr>
          <th>Machine</th>
          <th>OS</th>
          <th><code>bash --version</code></th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>the codespace</td>
          <td>Debian 12 (bookworm)</td>
          <td>5.2.15</td>
      </tr>
      <tr>
          <td>ws1, srv1</td>
          <td>Debian 13 (trixie)</td>
          <td>5.2.37</td>
      </tr>
  </tbody>
</table>
<p>Everything in this course works the same on both. The <a href="https://www.gnu.org/software/bash/manual/">GNU Bash manual</a>
is the primary source when you want the full story.</p>
<hr>
<h2 id="what-is-the-prompt-telling-you">What is the prompt telling you?</h2>
<p>The <strong>prompt</strong> is what the shell prints when it&rsquo;s ready for your next command. It&rsquo;s not decoration: it tells
you who you are, where you are, and sometimes how the last command went.</p>
<p>In the codespace, the prompt is set up by the dev container and looks like this, with your GitHub name in
front:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">@yourname ➜ /workspaces/linux-cli-101 (main) $
</span></span></code></pre></div><p>On ws1, a plain Debian machine, it&rsquo;s the Debian default:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">netops@ws1:~$
</span></span></code></pre></div><table>
  <thead>
      <tr>
          <th>Part</th>
          <th>Means</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>@yourname</code> or <code>netops</code></td>
          <td>the user you are</td>
      </tr>
      <tr>
          <td><code>@ws1</code></td>
          <td>the machine you&rsquo;re on: worth checking before you type anything risky</td>
      </tr>
      <tr>
          <td><code>/workspaces/linux-cli-101</code> or <code>~</code></td>
          <td>the folder you&rsquo;re in now</td>
      </tr>
      <tr>
          <td><code>(main)</code></td>
          <td>the Git branch, when the folder is a Git repository</td>
      </tr>
      <tr>
          <td><code>➜</code></td>
          <td>in the codespace, it turns red when the last command failed</td>
      </tr>
      <tr>
          <td><code>$</code></td>
          <td>an ordinary user; <code>#</code> means you&rsquo;re root, the all-powerful user (<a href="/lessons/linux-cli-101/users-permissions-sudo/">lesson 4</a>)</td>
      </tr>
  </tbody>
</table>
<p>When you&rsquo;re logged in to three machines in three tabs, reading the prompt is how you avoid running a command
on the wrong one.</p>
<hr>
<h2 id="how-is-a-command-line-put-together">How is a command line put together?</h2>
<p>Every line you type has the same shape: a <strong>command</strong>, then <strong>options</strong> that change how it behaves, then
<strong>arguments</strong> that say what it works on. Spaces separate them.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">ls  -l  files/configs
</span></span><span class="line"><span class="cl">│   │   └── argument: what to work on
</span></span><span class="line"><span class="cl">│   └────── option: how to do it (long listing)
</span></span><span class="line"><span class="cl">└────────── command: what to run
</span></span></code></pre></div><p>Here&rsquo;s that line in the lab, with real output:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">total 20
</span></span><span class="line"><span class="cl">-rw-r--r-- 1 vscode vscode  63 Nov 11 05:44 README.txt
</span></span><span class="line"><span class="cl">-rw-r--r-- 1 vscode vscode 384 Nov 11 05:44 chi leaf2.cfg
</span></span><span class="line"><span class="cl">-rw-r--r-- 1 vscode vscode 411 Nov 11 05:44 chi-leaf1.cfg
</span></span><span class="line"><span class="cl">-rw-r--r-- 1 vscode vscode 411 Nov 11 05:44 nyc-leaf1.cfg
</span></span><span class="line"><span class="cl">-rw-r--r-- 1 vscode vscode 384 Nov 11 05:44 nyc-leaf2.cfg
</span></span></code></pre></div><p>Options come in two spellings: <strong>short</strong>, one dash and one letter (<code>-l</code>), and <strong>long</strong>, two dashes and a word
(<code>--all</code>). Short ones can be bundled: <code>ls -la</code> is <code>ls -l -a</code>. Most GNU tools accept both, and <code>--help</code> lists
them side by side.</p>
<p>Two things trip beginners up:</p>
<ul>
<li><strong>Linux is case-sensitive.</strong> <code>ls -l</code> and <code>ls -L</code> are different options, and <code>README.txt</code> and <code>readme.txt</code>
are different files.</li>
<li><strong>A space ends an argument.</strong> <code>chi leaf2.cfg</code> above is one file with a space in its name. Typed bare, the
shell sees two arguments. <a href="/lessons/linux-cli-101/working-with-files/">Lesson 2</a> covers quoting.</li>
</ul>
<p>Not every command is a program on disk. Some are built into the shell itself. <code>type</code> tells you which:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">cd is a shell builtin
</span></span><span class="line"><span class="cl">pwd is a shell builtin
</span></span></code></pre></div><p><code>cd</code> has to be a builtin: a separate program could change its own folder, but not the shell&rsquo;s.</p>
<hr>
<h2 id="what-is-the-linux-filesystem-tree">What is the Linux filesystem tree?</h2>
<p>Windows gives each disk a letter: <code>C:\</code>, <code>D:\</code>. Linux doesn&rsquo;t. There is <strong>one tree</strong>, and everything hangs off
its root, written <code>/</code>. Extra disks, USB sticks and network shares are attached somewhere inside that tree, at a
folder called a <strong>mount point</strong>, so they look like any other folder.</p>
<p>The top of the tree in the codespace (<code>ls /</code>) looks like this:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">bin   boot  dev  etc  home  lib  media  mnt  opt  proc  root
</span></span><span class="line"><span class="cl">run   sbin  srv  sys  tmp   usr  var  workspaces
</span></span></code></pre></div><p><code>workspaces</code> is the one Codespaces adds; the rest you&rsquo;ll find on any Debian machine. The layout is set out in
the <a href="https://refspecs.linuxfoundation.org/FHS_3.0/fhs/index.html">Filesystem Hierarchy Standard</a>. A handful of
folders matter from day one:</p>
<table>
  <thead>
      <tr>
          <th>Folder</th>
          <th>What&rsquo;s in it</th>
          <th>You&rsquo;ll go there to</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>/etc</code></td>
          <td>system configuration, as text files</td>
          <td>read how a service is set up (<code>/etc/hostname</code>, <code>/etc/os-release</code>)</td>
      </tr>
      <tr>
          <td><code>/var/log</code></td>
          <td>log files</td>
          <td>find out what a service did, and when</td>
      </tr>
      <tr>
          <td><code>/home</code></td>
          <td>one folder per user (<code>/home/vscode</code>, <code>/home/netops</code>)</td>
          <td>keep your own files</td>
      </tr>
      <tr>
          <td><code>/tmp</code></td>
          <td>temporary files, often emptied at reboot</td>
          <td>try things out</td>
      </tr>
      <tr>
          <td><code>/usr/bin</code></td>
          <td>most of the programs you run (<code>/usr/bin/ls</code>)</td>
          <td>see where a command lives</td>
      </tr>
      <tr>
          <td><code>/root</code></td>
          <td>the root user&rsquo;s home, not inside <code>/home</code></td>
          <td>rarely: it&rsquo;s root&rsquo;s</td>
      </tr>
  </tbody>
</table>
<p>Linux keeps its configuration in plain text, and it&rsquo;s one of the reasons it automates so well: a tool that can
read and write a text file can configure the system.</p>
<hr>
<h2 id="what-is-the-difference-between-an-absolute-and-a-relative-path">What is the difference between an absolute and a relative path?</h2>
<p>An <strong>absolute path</strong> starts with <code>/</code> and spells out the route from the root. It means the same thing wherever
you are: <code>/workspaces/linux-cli-101/files/site/nyc/README.txt</code> is one file, from any folder, on any shell in
that machine.</p>
<p>A <strong>relative path</strong> doesn&rsquo;t start with <code>/</code>. The shell starts it from your <strong>working directory</strong>, the folder
you&rsquo;re in now (<code>pwd</code> prints it). <code>files/configs</code> only means something because the terminal is sitting in
<code>/workspaces/linux-cli-101</code>.</p>
<p>Three short names help you move around:</p>
<table>
  <thead>
      <tr>
          <th>Name</th>
          <th>Means</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>.</code></td>
          <td>this folder</td>
      </tr>
      <tr>
          <td><code>..</code></td>
          <td>the folder above this one, its <strong>parent</strong></td>
      </tr>
      <tr>
          <td><code>~</code></td>
          <td>your home folder: <code>/home/vscode</code> in the codespace, <code>/home/netops</code> on ws1</td>
      </tr>
  </tbody>
</table>
<p>Every folder has a <code>.</code> and a <code>..</code> inside it, which is how relative paths can climb. Here&rsquo;s one from the lab&rsquo;s
practice tree, with <code>realpath</code>, which turns any path into the absolute path it points to:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ pwd
</span></span><span class="line"><span class="cl">/workspaces/linux-cli-101/files/site/nyc/rack2
</span></span><span class="line"><span class="cl">$ realpath ../README.txt
</span></span><span class="line"><span class="cl">/workspaces/linux-cli-101/files/site/nyc/README.txt
</span></span></code></pre></div><p>Read it one piece at a time: start in <code>rack2</code>; <code>..</code> climbs one level to <code>nyc</code>; <code>README.txt</code> is the file in
there. Each <code>..</code> is one folder up. Chain them (<code>../../</code>) to climb further, then name folders to go back down.</p>
<p><code>~</code> is a little different. It isn&rsquo;t a folder name: the <strong>shell</strong> replaces it with your home folder&rsquo;s absolute
path before the command sees it. <code>cd</code> on its own does the same thing and takes you home.</p>
<h3 id="which-kind-should-you-use">Which kind should you use?</h3>
<table>
  <thead>
      <tr>
          <th>Use</th>
          <th>When</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>absolute</td>
          <td>in scripts, config files, cron jobs: anything that might run from a different folder</td>
      </tr>
      <tr>
          <td>relative</td>
          <td>typing at the prompt, inside a project you&rsquo;re already in</td>
      </tr>
  </tbody>
</table>
<p>Most &ldquo;file not found&rdquo; errors from a script that works at the prompt are a relative path started from the wrong
folder.</p>
<hr>
<h2 id="what-are-hidden-files">What are hidden files?</h2>
<p>A file or folder whose name starts with a dot is <strong>hidden</strong>: <code>ls</code> leaves it out unless you add <code>-a</code>. It isn&rsquo;t
secret or protected. It&rsquo;s a convention for files you don&rsquo;t need to see every day, mostly configuration.</p>
<p>New users on ws1 get their starting files from <code>/etc/skel</code>, and <code>ls</code> alone shows nothing there. <code>ls -a</code> shows
what&rsquo;s really in it:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">.
</span></span><span class="line"><span class="cl">..
</span></span><span class="line"><span class="cl">.bash_logout
</span></span><span class="line"><span class="cl">.bashrc
</span></span><span class="line"><span class="cl">.profile
</span></span></code></pre></div><p><code>.bashrc</code> sets up your shell each time it starts (<a href="/lessons/linux-cli-101/processes-environment-path/">lesson 5</a>).
Your home folder also holds <code>.ssh</code>, where SSH keeps your keys (<a href="/lessons/linux-cli-101/ssh-and-keys/">lesson 6</a>).
Hidden files are where a lot of tools keep their settings, so when something behaves oddly, <code>ls -a</code> is worth a
look.</p>
<hr>
<h2 id="how-do-you-get-help-without-leaving-the-terminal">How do you get help without leaving the terminal?</h2>
<p>Two places, and you&rsquo;ll use both.</p>
<p><strong><code>--help</code></strong> is built into most commands. It prints a short summary: the usage line and every option, one line
each. It&rsquo;s quick, and it describes the exact version you have.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">Usage: mkdir [OPTION]... DIRECTORY...
</span></span><span class="line"><span class="cl">Create the DIRECTORY(ies), if they do not already exist.
</span></span></code></pre></div><p>Read the usage line as a pattern: <code>[ ]</code> means optional, <code>...</code> means you can give more than one, and capitals
are the things you fill in.</p>
<p><strong><code>man</code></strong> opens the <strong>manual page</strong>: the full reference, with descriptions and often examples. It opens in a
pager, so you scroll with the arrow keys or Space, search with <code>/</code>, and leave with <code>q</code>.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">LS(1)                            User Commands                           LS(1)
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">NAME
</span></span><span class="line"><span class="cl">       ls - list directory contents
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">SYNOPSIS
</span></span><span class="line"><span class="cl">       ls [OPTION]... [FILE]...
</span></span></code></pre></div><p>The <code>(1)</code> is the manual&rsquo;s <strong>section</strong>. Section 1 is commands, section 5 is file formats, section 8 is admin
commands. When a name lives in more than one, ask for the section: <code>man 5 passwd</code> is the file, <code>man passwd</code>
the command. The same pages are online at <a href="https://man7.org/linux/man-pages/">man7.org</a>, which is handy when
you&rsquo;re reading on another screen.</p>
<p>Not every machine has <code>man</code>. The codespace does. ws1 is a slim container and doesn&rsquo;t (<code>bash: man: command not found</code>), so <code>--help</code> is your friend there. Builtins such as <code>cd</code> have no manual page of their own: <code>help cd</code> asks
Bash for its summary.</p>
<hr>
<h2 id="what-i-see-go-wrong">What I see go wrong</h2>
<p><strong>Working in the wrong folder.</strong> A command says <em>No such file or directory</em> for a file you can see in the
explorer. The file exists; you&rsquo;re just not where the relative path starts. Check the prompt, or <code>pwd</code>.</p>
<p><strong>Working on the wrong machine.</strong> Three tabs, three machines, one prompt that looks a lot like another. Read
<code>user@host</code> before anything that changes state.</p>
<p><strong>Reading <code>~</code> as a folder name.</strong> <code>~</code> works because the shell expands it. Inside quotes (<code>&quot;~/notes.txt&quot;</code>) it
doesn&rsquo;t, and some programs reading paths from a config file won&rsquo;t expand it either. Use the absolute path
there.</p>
<p><strong>Typing a Windows path.</strong> Linux uses <code>/</code>, not <code>\</code>, and there&rsquo;s no drive letter. A backslash means something
else to the shell entirely.</p>
<p><strong>Missing a hidden file.</strong> The config you&rsquo;re looking for &ldquo;isn&rsquo;t there&rdquo; because its name starts with a dot.
<code>ls -a</code>.</p>
<p><strong>Wrong case.</strong> <code>Files/</code> and <code>files/</code> are different folders. Tab completion avoids this: let the shell finish
names for you.</p>
<hr>
<h2 id="try-it">Try it</h2>
<p><a href="/labs/linux-cli-101/task-1-1/">Chapter 1 of the lab</a> starts in the codespace&rsquo;s terminal. You find your home
folder, the lab&rsquo;s folder and the operating system&rsquo;s name, go looking for a hidden file, and then read a small
practice tree of site and rack folders until you can reach any file in it by an absolute path and by the
shortest relative one. If the terminal itself is new, <a href="/labs/basics/">Meet the lab</a> shows the editor, the
terminal and the checker first.</p>
<h2 id="whats-next">What&rsquo;s next</h2>
<p>Now you can find a file, <a href="/lessons/linux-cli-101/working-with-files/">lesson 2</a> is about what to do with it:
reading logs and configs, editing in nano, copying, moving and deleting, and finding files by name and age.</p>
]]></content:encoded></item></channel></rss>