<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/"><channel><title>Tcp · Network Doodles</title><link>https://networkdoodles.com/tags/tcp/</link><description>Network automation, from first playbook to agentic operations. Ansible, NetBox, MCP, Python and the occasional doodle.</description><language>en-us</language><managingEditor>David Henderson</managingEditor><lastBuildDate>Mon, 16 Nov 2026 09:00:00 -0500</lastBuildDate><atom:link href="https://networkdoodles.com/tags/tcp/index.xml" rel="self" type="application/rss+xml"/><item><title>Troubleshooting a Broken Path with Wireshark: Loss, Silent Drops and MTU (Wireshark 101, Lesson 8)</title><link>https://networkdoodles.com/lessons/wireshark-101/broken-path/</link><pubDate>Mon, 16 Nov 2026 09:00:00 -0500</pubDate><dc:creator>David Henderson</dc:creator><guid>https://networkdoodles.com/lessons/wireshark-101/broken-path/</guid><category>wireshark</category><category>packet-capture</category><category>networking-basics</category><category>troubleshooting</category><category>tcp</category><description>Find where a path breaks: capture at several points, follow one packet, read loss and retransmissions, silent ACL drops, missing routes and MTU holes.</description><content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[<p>When a connection fails, the capture on the client tells you <em>that</em> something went wrong, rarely <em>where</em>. The
method that finds where is simple: capture at two or more points on the path at the same time, and follow one
packet from point to point. The fault is between the last place you see it and the first place you don&rsquo;t. Then
the shape of what you see tells you what kind of fault it is: a burst of duplicate ACKs is loss, a SYN that
vanishes is a filter, a SYN-ACK that never comes back is a route, and a connection that opens and then stalls
is very often the MTU.</p>
<p>This lesson pulls the whole course together on four faults from the
<a href="/labs/wireshark-101/">Wireshark 101 lab</a>. Each one looks different on the wire, and two of them look the same
from the laptop.</p>
<hr>
<h2 id="silence-versus-refusal">Silence versus refusal</h2>
<p><a href="/lessons/wireshark-101/tcp/">Lesson 6</a> drew the line that matters most here. A SYN answered by a RST is a
<strong>refusal</strong>: the network works and the far end said no. A SYN answered by nothing is <strong>silence</strong>: something on
the way dropped it, or dropped the answer, and nobody said a word.</p>
<p>A refusal is easy to troubleshoot, because the answer came from somewhere. Silence is hard, because the client
sees exactly the same thing whatever the cause: the same SYN, sent again and again. In the lab&rsquo;s mystery fault,
the laptop&rsquo;s capture is seven copies of one SYN over seven seconds, then nothing. Three different faults
produce that capture. You can&rsquo;t tell them apart from the laptop, however long you stare.</p>
<p>So you stop staring and add capture points.</p>
<hr>
<h2 id="capture-at-two-points-and-follow-one-packet">Capture at two points and follow one packet</h2>
<p>The lab&rsquo;s path from the laptop to the web server crosses a switch and two routers:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">laptop ──[laptop:eth1]── sw1 ── r1 ──[r1:eth2]── r2 ──[r2:eth2]── web
</span></span></code></pre></div><p>Capture all three marked points into one file, at the same time, and filter on one connection. A SYN that
appears at <code>laptop:eth1</code> and <code>r1:eth2</code>, but never at <code>r2:eth2</code>, got through r1 and died at r2. That&rsquo;s the
whole method: divide the path, and look for the boundary between seen and not seen. Wireshark&rsquo;s
<code>frame.interface_name</code> field records which point saw each frame.</p>
<p>Two rules make it work:</p>
<ul>
<li><strong>Capture at the same time.</strong> Faults come and go. Two captures taken ten minutes apart don&rsquo;t describe the
same packet.</li>
<li><strong>Take away what you&rsquo;re not testing.</strong> The lab&rsquo;s mystery traffic opens the web server by its IP address, not
its name, so a DNS problem can&rsquo;t hide the fault you&rsquo;re looking for.</li>
</ul>
<h3 id="the-multi-interface-trap">The multi-interface trap</h3>
<p>Put one packet in a file three times and Wireshark&rsquo;s TCP analysis gets confused. It doesn&rsquo;t know the copies are
the same packet seen in three places. It thinks the second and third copies were sent again:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">1  0.000000000  laptop:eth1  51606 → 80 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 ...
</span></span><span class="line"><span class="cl">2  0.000217167  r1:eth2      [TCP Retransmission] 51606 → 80 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 ...
</span></span></code></pre></div><p>Frame 2 isn&rsquo;t a retransmission. It&rsquo;s frame 1, 0.2 ms later and one router further on. Every <code>tcp.analysis</code>
flag in a multi-point capture is suspect until you deal with this.</p>
<p>The obvious fix, a display filter like <code>frame.interface_name == &quot;laptop:eth1&quot;</code>, isn&rsquo;t enough. A display filter
hides frames; it doesn&rsquo;t redo the analysis. In the lab&rsquo;s MTU capture, the SYN-ACK reached the switch&rsquo;s capture
point first, so the laptop&rsquo;s own copy is still flagged <em>TCP Retransmission</em> with everything else filtered out.
Two fixes that do work, both checked against the lab&rsquo;s captures with Wireshark 4.6:</p>
<ul>
<li><strong>Export one interface to its own file</strong> (File › Export Specified Packets, with the interface filter applied)
and open that. The analysis starts over, and the false flags are gone.</li>
<li><strong>Tell Wireshark the copies are different conversations.</strong> The preference
<code>protocols.conversation_deinterlacing_key</code> (find it under Edit › Preferences › Advanced) set to <code>.MI</code> keeps
frames from different interfaces and MACs in separate TCP streams. Each capture point becomes its own stream,
and only real retransmissions are flagged.</li>
</ul>
<hr>
<h2 id="packet-loss-duplicate-acks-and-fast-retransmission">Packet loss: duplicate ACKs and fast retransmission</h2>
<p>A link that loses a few percent of its packets doesn&rsquo;t look broken. Pages load, downloads finish, ping mostly
answers. Everything is just slower. A capture shows it plainly, because TCP has to do something about every lost
packet, and it does it on the wire.</p>
<p>In the lab, r2 drops 3% of what it sends towards r1, and the laptop downloads a 2 MB file. Here&rsquo;s one lost
segment and its repair:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">35 0.578603209 192.168.2.10 → 192.168.1.175 [TCP Previous segment not captured] 80 → 56336 [ACK] Seq=17623 ... Len=1448
</span></span><span class="line"><span class="cl">36 0.578605084 192.168.1.175 → 192.168.2.10 [TCP Dup ACK 34#1] 56336 → 80 [ACK] Seq=79 Ack=16175 ... SLE=17623 SRE=19071
</span></span><span class="line"><span class="cl">38 0.578606375 192.168.1.175 → 192.168.2.10 [TCP Dup ACK 34#2] 56336 → 80 [ACK] Seq=79 Ack=16175 ... SLE=17623 SRE=20519
</span></span><span class="line"><span class="cl">40 0.578607334 192.168.1.175 → 192.168.2.10 [TCP Dup ACK 34#3] 56336 → 80 [ACK] Seq=79 Ack=16175 ... SLE=17623 SRE=21967
</span></span><span class="line"><span class="cl">...
</span></span><span class="line"><span class="cl">58 0.578705625 192.168.2.10 → 192.168.1.175 [TCP Fast Retransmission] 80 → 56336 [ACK] Seq=16175 ... Len=1448
</span></span></code></pre></div><p>Read it with the sequence numbers from lesson 6:</p>
<ol>
<li><strong>The gap.</strong> Frame 35 starts at byte 17,623, but the laptop was expecting 16,175. One 1,448-byte segment is
missing. Wireshark marks it <em>Previous segment not captured</em>.</li>
<li><strong>Duplicate ACKs.</strong> The laptop can only acknowledge bytes up to the gap, so every segment that arrives after it
triggers the same ACK again: <code>Ack=16175</code>, &ldquo;I still need byte 16,175&rdquo;.</li>
<li><strong>SACK blocks.</strong> Each duplicate ACK also carries a <strong>SACK</strong> block (<a href="https://www.rfc-editor.org/rfc/rfc2018">RFC 2018</a>):
<code>SLE</code> and <code>SRE</code> (the left and right edges) list the bytes the laptop <em>did</em> get past the gap. The right edge
grows with every segment. The sender learns exactly what&rsquo;s missing and resends only that.</li>
<li><strong>Fast retransmission.</strong> The sender doesn&rsquo;t wait for a timeout. Duplicate ACKs are a strong hint that one
segment was lost and the ones after it arrived, so it resends the missing one straight away.
<a href="https://www.rfc-editor.org/rfc/rfc5681">RFC 5681</a> sets the trigger at three duplicate ACKs.</li>
</ol>
<p>In the lab&rsquo;s reference capture, 3% loss on one direction of one link produced 26 retransmissions (25 of them
fast) and 142 duplicate ACKs, plus 39 gaps and 18 segments marked out of order. The receiver is noisy about loss;
the sender is quiet and efficient.</p>
<h3 id="wiresharks-labels-are-heuristics">Wireshark&rsquo;s labels are heuristics</h3>
<p>Those <code>tcp.analysis</code> labels are Wireshark&rsquo;s best guess from the sequence numbers, not something TCP wrote in
the packet. <a href="https://www.wireshark.org/docs/wsug_html_chunked/ChAdvTCPAnalysis.html">Wireshark&rsquo;s TCP analysis documentation</a>
gives the rules, and they don&rsquo;t always match the RFCs. It calls a segment a <em>fast retransmission</em> after two
duplicate ACKs, not three. And it calls a filled gap <em>out-of-order</em> instead of a retransmission when it arrives
very soon after the previous segment, within the initial RTT or 3 ms. The lab&rsquo;s link loses packets but never reorders them, so
those 18 &ldquo;out-of-order&rdquo; segments are really repairs too.</p>
<table>
  <thead>
      <tr>
          <th>Flag</th>
          <th>Wireshark&rsquo;s meaning</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>tcp.analysis.lost_segment</code></td>
          <td>a gap: the sequence number jumped past what was expected</td>
      </tr>
      <tr>
          <td><code>tcp.analysis.duplicate_ack</code></td>
          <td>an ACK that repeats the previous one, with no data and the same window</td>
      </tr>
      <tr>
          <td><code>tcp.analysis.retransmission</code></td>
          <td>data the sender had already sent (fast retransmissions included)</td>
      </tr>
      <tr>
          <td><code>tcp.analysis.fast_retransmission</code></td>
          <td>a retransmission that followed duplicate ACKs</td>
      </tr>
      <tr>
          <td><code>tcp.analysis.out_of_order</code></td>
          <td>a filled gap that arrived quickly after the segment before it</td>
      </tr>
  </tbody>
</table>
<p><code>tcp.analysis.flags</code> matches them all, and <strong>Analyze › Expert Information</strong> counts them by type. That&rsquo;s the
fastest summary of a capture&rsquo;s health. Just remember the multi-interface trap above before trusting the counts.</p>
<p>To find <em>where</em> the loss happens, use the method: a segment on <code>r2:eth2</code> that never appears on <code>r1:eth2</code> was lost
between them.</p>
<hr>
<h2 id="acl-drops-silent-or-icmp-administratively-prohibited">ACL drops: silent, or ICMP administratively prohibited</h2>
<p>An <strong>access list</strong> (ACL) is a router&rsquo;s list of permit and deny rules. When it denies a packet, the router has two
choices: drop it silently, or tell the sender with an ICMP <strong>Destination Unreachable</strong> message, type 3,
code 13, <em>communication administratively prohibited</em> (<a href="https://www.rfc-editor.org/rfc/rfc1812">RFC 1812</a>).</p>
<p>The two make very different captures:</p>
<table>
  <thead>
      <tr>
          <th>Router behavior</th>
          <th>What the client sees</th>
          <th>What you can learn</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>sends ICMP type 3 code 13</td>
          <td>the SYN, then an ICMP message from the router that denied it</td>
          <td>which router, straight away: it&rsquo;s the source of the ICMP</td>
      </tr>
      <tr>
          <td>drops silently</td>
          <td>the SYN, again and again, then a timeout</td>
          <td>nothing, until you capture further along</td>
      </tr>
  </tbody>
</table>
<p>The routers in the lab run CapeOS 0.11.2, which answers an inbound ACL deny with ICMP administratively
prohibited unless the interface has <code>no ip unreachables</code>. The mystery fault sets it, so the drop is silent: the
harder case, and the one you&rsquo;ll meet most often, because many networks turn these messages off.</p>
<p>In the lab&rsquo;s reference capture the fault was on r2. The SYN shows up at <code>laptop:eth1</code> and at <code>r1:eth2</code>, seven
times each, and never once at <code>r2:eth2</code>. It crossed r1, arrived at r2, and went no further. No ICMP anywhere.</p>
<hr>
<h2 id="a-missing-return-route">A missing return route</h2>
<p>The third mystery fault looks the same from the laptop, and it&rsquo;s completely different underneath. r2 loses its
route back to the home network, 192.168.1.0/24.</p>
<p>Follow the SYN and it gets all the way to web. web answers with a SYN-ACK. The SYN-ACK reaches r2, which has no
route for 192.168.1.175, and dies there. The captures show it:</p>
<ul>
<li>the SYN at all three points;</li>
<li>a SYN-ACK at <code>r2:eth2</code> only, never at <code>r1:eth2</code> or the laptop;</li>
<li>an ICMP <em>Network unreachable</em> (type 3, code 0) from r2&rsquo;s address on that link, 192.168.2.1, <strong>to web</strong>.</li>
</ul>
<p>That last one catches people out. A router that can&rsquo;t forward a packet tells the packet&rsquo;s <em>source</em>, and the
packet it couldn&rsquo;t forward was web&rsquo;s SYN-ACK. So the ICMP message goes to web, not to the laptop. The laptop never
hears a thing. You only see that message if you capture on web&rsquo;s side of r2.</p>
<p>The general lesson: a path has two directions, and a fault in the return direction looks like a fault in the
forward direction from the client. If the SYN arrives and the answer doesn&rsquo;t come back, look at routes <em>back</em>.</p>
<hr>
<h2 id="mtu-when-small-packets-work-and-big-ones-dont">MTU: when small packets work and big ones don&rsquo;t</h2>
<p>The <strong>MTU</strong> (maximum transmission unit) is the biggest IP packet a link carries in one frame. On ordinary
Ethernet it&rsquo;s 1,500 bytes, a 1,514-byte frame with the Ethernet header. Some networks use <strong>jumbo frames</strong>, often
9,000 bytes, to move data in fewer, bigger packets.</p>
<p>At the start of a connection, each side announces its <strong>MSS</strong> (maximum segment size), worked out from its own
MTU: 1,460 for 1,500, 8,960 for 9,000. The other side then sends segments up to that size. Neither side knows
anything about the links in between.</p>
<p>In the lab, someone turned on jumbo frames on the laptop and on web, but not on the switch between the laptop
and r1. Both hosts announced <code>MSS=8960</code>. The handshake is small, so it crossed fine. Then:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">11  laptop:eth1    74  57982 → 443 [SYN] Seq=0 Win=62720 Len=0 MSS=8960 ...
</span></span><span class="line"><span class="cl">13  sw1:eth1       74  443 → 57982 [SYN, ACK] Seq=0 Ack=1 ... MSS=8960 ...
</span></span><span class="line"><span class="cl">15  laptop:eth1    66  57982 → 443 [ACK] Seq=1 Ack=1 ...
</span></span><span class="line"><span class="cl">17  laptop:eth1  1624  Client Hello (SNI=web.lab)
</span></span><span class="line"><span class="cl">18  laptop:eth1  1624  [TCP Retransmission] 57982 → 443 [PSH, ACK] Seq=1 Ack=1 ... Len=1558
</span></span><span class="line"><span class="cl">19  laptop:eth1  1624  [TCP Retransmission] 57982 → 443 [PSH, ACK] Seq=1 Ack=1 ... Len=1558
</span></span><span class="line"><span class="cl">...
</span></span></code></pre></div><p>The third column is the frame length. The TLS 1.3 Client Hello from <a href="/lessons/wireshark-101/tls-without-decryption/">lesson 7</a>,
1,558 bytes, now went as <strong>one</strong> 1,624-byte frame, because the laptop believed it had 8,960 bytes to play with.
It left the laptop 16 times (the original and 15 retransmissions), about every half second, and never once appeared on <code>sw1:eth1</code>, the switch&rsquo;s port
towards r1. Filter <code>frame.len &gt; 1514</code> and the picture is complete in two lines: big frames on one side of the
switch, none on the other.</p>
<h3 id="why-a-switch-makes-a-black-hole">Why a switch makes a black hole</h3>
<p>A router that receives a packet too big for the next link, with the IP header&rsquo;s <strong>DF</strong> (don&rsquo;t fragment) bit set,
drops it and sends back ICMP type 3, code 4, <em>fragmentation needed</em>, with the next link&rsquo;s MTU in the message.
That&rsquo;s <strong>Path MTU Discovery</strong> (<a href="https://www.rfc-editor.org/rfc/rfc1191">RFC 1191</a>): the sender learns the
smaller size and sends smaller segments. TCP on Linux sets DF on its packets for exactly this reason. The lab&rsquo;s
1,624-byte frame had it set.</p>
<p>A <strong>switch</strong> can&rsquo;t do that. It works at layer 2, forwarding frames by MAC address, and has no IP address on the
path to send ICMP from. A frame bigger than its port&rsquo;s MTU is simply dropped. Nobody is told, so the sender never
learns, and it keeps retransmitting into the hole until the application gives up. That&rsquo;s a <strong>black hole</strong>:
packets go in and nothing comes out.</p>
<p>Routers can make black holes too. If the <em>fragmentation needed</em> message is filtered on the way back, often by a
firewall that blocks &ldquo;all ICMP&rdquo;, PMTUD breaks in the same silent way.
<a href="https://www.rfc-editor.org/rfc/rfc2923">RFC 2923</a> describes the problem.</p>
<h3 id="why-tls-13-breaks-first">Why TLS 1.3 breaks first</h3>
<p>An MTU problem only hurts packets bigger than the smallest link. Ping is 98 bytes. A DNS lookup is under 100.
A TCP handshake is 74. A plain HTTP request is often under 200. On the lab&rsquo;s broken path, all of those work. The
first big packet in most HTTPS connections is now the client&rsquo;s TLS 1.3 Client Hello, and with a post-quantum key
share it&rsquo;s over 1,500 bytes on its own. So the classic symptom is: <strong>ping works, HTTP works, HTTPS hangs right
after the handshake.</strong> It used to take a large download to trip over an MTU mismatch. Now a single Client Hello
does it.</p>
<hr>
<h2 id="what-i-see-go-wrong">What I see go wrong</h2>
<p><strong>Trusting the analysis in a multi-point capture.</strong> Half the <em>Retransmission</em> and <em>Dup ACK</em> labels in a
three-point capture are the same packet seen three times. Export one interface, or set the deinterlacing
preference, before you count anything.</p>
<p><strong>Capturing at one point and guessing.</strong> The client&rsquo;s capture says &ldquo;no answer&rdquo;. It can&rsquo;t say where. Two capture
points, at the same time, usually settle it in minutes.</p>
<p><strong>Forgetting the return path.</strong> A SYN that arrives and an answer that doesn&rsquo;t come back is a return-route or
return-filter problem. The ICMP message that would explain it goes to the server, not to you.</p>
<p><strong>&ldquo;Ping works, so the network&rsquo;s fine.&rdquo;</strong> Ping is small. Test with the size that fails. When the application
hangs after the handshake, filter <code>frame.len &gt; 1514</code> and look for large frames sent over and over and never
acknowledged.</p>
<p><strong>Mixed MTUs.</strong> Jumbo frames on the hosts and not on the switches, or the other way around, is a black hole
waiting for its first big packet. Make every device on a layer 2 network agree on the MTU.</p>
<hr>
<h2 id="try-it">Try it</h2>
<p><a href="/labs/wireshark-101/#chapter-8">Chapter 8 of the lab</a> breaks the path three ways: packet loss you measure, a
fault the lab picks at random that you find with three capture points, and the jumbo-frame black hole.</p>
<h2 id="where-to-go-next">Where to go next</h2>
<p>That&rsquo;s the end of Wireshark 101. Captures are for one conversation, on demand. For watching a whole network all
the time, logs and metrics do the job, and the <span class="link-soon">observability course</span> builds them. When
those say something is wrong and can&rsquo;t say why, come back to the packets. Or start again at the
<a href="/lessons/wireshark-101/dns/">DNS lesson</a> with a capture from your own network.</p>
]]></content:encoded></item><item><title>The Network from the Command Line: ip route get, ss, curl -v, dig and tcpdump (Linux CLI 101, Lesson 8)</title><link>https://networkdoodles.com/lessons/linux-cli-101/network-commands/</link><pubDate>Mon, 16 Nov 2026 09:00:00 -0500</pubDate><dc:creator>David Henderson</dc:creator><guid>https://networkdoodles.com/lessons/linux-cli-101/network-commands/</guid><category>linux</category><category>networking-basics</category><category>troubleshooting</category><category>packet-capture</category><category>tcp</category><description>Troubleshoot from a Linux host: ip route get, dig, traceroute, what ss says about listening addresses, curl -v from DNS to HTTP, and packets with tcpdump.</description><content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[<p>When a web service works on the server but not from another host, work through the path in order. Does the
name resolve to the right address (<code>dig</code>)? Which way will the packet go (<code>ip route get</code>)? Is anything on the
way answering (<code>ping</code>, <code>traceroute</code>)? Is the service listening <strong>on an address the other host can reach</strong>
(<code>ss -tlnp</code>)? Then let <code>curl -v</code> show where the conversation stops, and <code>tcpdump</code> show what&rsquo;s on the wire.</p>
<p>A common answer to &ldquo;works locally, not remotely&rdquo; is the listening address. A program listening on
<code>127.0.0.1</code> only accepts connections from its own machine. <code>0.0.0.0</code> means every address the machine has.</p>
<p>You met most of these commands in the <a href="/labs/networking-101-lab/task-3-3/">Networking 101 lab</a>. This lesson is
about the options and output that matter when you&rsquo;re troubleshooting. Everything below ran on the lab&rsquo;s <code>ws1</code>
(Debian 13: iproute2 6.15.0, iputils 20240905, traceroute 2.1.6, curl 8.14.1, DiG 9.20.30, tcpdump 4.99.5),
except the TLS example, which ran in the codespace.</p>
<hr>
<h2 id="how-do-i-see-a-hosts-addresses-at-a-glance">How do I see a host&rsquo;s addresses at a glance?</h2>
<p><code>ip addr</code> prints a screenful per interface. <code>-br</code> (<em>brief</em>) prints one line each, and <code>-4</code> drops IPv6:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ ip -br -4 addr
</span></span><span class="line"><span class="cl">lo               UNKNOWN        127.0.0.1/8
</span></span><span class="line"><span class="cl">eth0@if76        UP             172.80.102.11/24
</span></span><span class="line"><span class="cl">eth1@if80        UP             192.168.10.10/24
</span></span></code></pre></div><p>Name, state, addresses. On ws1, <code>eth0</code> is the lab&rsquo;s management network (the way the codespace reaches it), and
<code>eth1</code> is its cable to the router, r1. <code>lo</code> is the <strong>loopback</strong> interface: a virtual interface that only ever
talks to the machine itself, with <code>127.0.0.1</code> on it. The <code>@if76</code> part says the interface is one end of a
virtual cable whose other end is interface 76 somewhere else, which is how containers are wired. Without
<code>-4</code>, <code>ip -br addr</code> also lists half a dozen tunnel interfaces that are <code>DOWN</code> and can be ignored.</p>
<hr>
<h2 id="which-way-will-a-packet-go">Which way will a packet go?</h2>
<p><code>ip route</code> shows the routing table. <code>ip route get</code> answers the question you actually have: <em>for this one
address, which route wins?</em> It asks the kernel&rsquo;s routing decision directly and sends nothing
(<a href="https://man7.org/linux/man-pages/man8/ip-route.8.html">ip-route(8)</a>).</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ ip route get 172.80.102.12
</span></span><span class="line"><span class="cl">172.80.102.12 dev eth0 src 172.80.102.11 uid 1000
</span></span><span class="line"><span class="cl">    cache
</span></span><span class="line"><span class="cl">$ ip route get 8.8.8.8
</span></span><span class="line"><span class="cl">8.8.8.8 via 192.168.10.1 dev eth1 src 192.168.10.10 uid 1000
</span></span><span class="line"><span class="cl">    cache
</span></span></code></pre></div><table>
  <thead>
      <tr>
          <th>Word</th>
          <th>Means</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>via 192.168.10.1</code></td>
          <td>send it to this router. No <code>via</code> means the address is on a directly connected network</td>
      </tr>
      <tr>
          <td><code>dev eth1</code></td>
          <td>out of this interface</td>
      </tr>
      <tr>
          <td><code>src 192.168.10.10</code></td>
          <td>with this as the source address</td>
      </tr>
      <tr>
          <td><code>uid 1000</code></td>
          <td>the user who asked. Ignore it</td>
      </tr>
  </tbody>
</table>
<p>The first address is on the management network, so it goes straight out of <code>eth0</code>. The second matches only the
default route, so it goes to r1. On a host with several interfaces, this is the quickest way to find out
whether replies leave by the interface you expect. The <code>src</code> field matters too: it&rsquo;s the address the far end
sees, and the address its firewall rules have to allow.</p>
<hr>
<h2 id="which-ping-and-traceroute-options-matter">Which ping and traceroute options matter?</h2>
<p>Linux <code>ping</code> runs until you press Ctrl-C. In anything you&rsquo;ll script or paste, give it a count:</p>
<table>
  <thead>
      <tr>
          <th>Option</th>
          <th>Does</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>ping -c 3</code></td>
          <td>send 3 and stop, with a summary</td>
      </tr>
      <tr>
          <td><code>ping -W 1</code></td>
          <td>wait at most 1 second for each reply</td>
      </tr>
      <tr>
          <td><code>ping -n</code></td>
          <td>numbers only, no reverse lookups</td>
      </tr>
      <tr>
          <td><code>traceroute -n</code></td>
          <td>numbers only, no names</td>
      </tr>
      <tr>
          <td><code>traceroute -q 1</code></td>
          <td>one probe per hop instead of three</td>
      </tr>
      <tr>
          <td><code>traceroute -w 1</code></td>
          <td>wait at most 1 second per probe</td>
      </tr>
      <tr>
          <td><code>traceroute -m 5</code></td>
          <td>give up after 5 hops instead of 30</td>
      </tr>
  </tbody>
</table>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ ping -c 3 -n srv1.lab
</span></span><span class="line"><span class="cl">PING srv1.lab (192.168.20.10) 56(84) bytes of data.
</span></span><span class="line"><span class="cl">64 bytes from 192.168.20.10: icmp_seq=1 ttl=63 time=0.074 ms
</span></span><span class="line"><span class="cl">64 bytes from 192.168.20.10: icmp_seq=2 ttl=63 time=0.252 ms
</span></span><span class="line"><span class="cl">64 bytes from 192.168.20.10: icmp_seq=3 ttl=63 time=0.075 ms
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">--- srv1.lab ping statistics ---
</span></span><span class="line"><span class="cl">3 packets transmitted, 3 received, 0% packet loss, time 2041ms
</span></span><span class="line"><span class="cl">rtt min/avg/max/mdev = 0.074/0.133/0.252/0.083 ms
</span></span></code></pre></div><p><code>ttl=63</code> is a free hop count. Linux sends with a TTL of 64, and each router on the way takes one off, so 63
means one router in between. The exit code also matters here: <code>ping</code> exits 0 if replies came back and 1 if none
did, which is what <a href="/lessons/linux-cli-101/bash-scripting-basics/">lesson 7</a>&rsquo;s scripts read.</p>
<p>Without <code>-n</code>, traceroute looks up a name for every hop. In the lab that&rsquo;s quick, because srv1&rsquo;s DNS server
knows the names. On a real network, a reverse lookup that nobody answers can add seconds per hop:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ traceroute 192.168.20.10
</span></span><span class="line"><span class="cl">traceroute to 192.168.20.10 (192.168.20.10), 30 hops max, 60 byte packets
</span></span><span class="line"><span class="cl"> 1  r1.lab (192.168.10.1)  1.073 ms  0.124 ms  0.057 ms
</span></span><span class="line"><span class="cl"> 2  srv1.lab (192.168.20.10)  4.811 ms  0.082 ms  0.045 ms
</span></span></code></pre></div><p>Stars don&rsquo;t always mean a fault. Running <code>traceroute -n</code> straight after that one gave this:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">traceroute to 192.168.20.10 (192.168.20.10), 30 hops max, 60 byte packets
</span></span><span class="line"><span class="cl"> 1  192.168.10.1  0.799 ms  0.035 ms  0.031 ms
</span></span><span class="line"><span class="cl"> 2  * * *
</span></span><span class="line"><span class="cl"> 3  * * *
</span></span><span class="line"><span class="cl"> 4  * * *
</span></span><span class="line"><span class="cl"> 5  * * *
</span></span><span class="line"><span class="cl"> 6  * * *
</span></span><span class="line"><span class="cl"> 7  * 192.168.20.10  0.461 ms  0.107 ms
</span></span></code></pre></div><p>Same two-hop path, a few seconds later, run again: back to two lines. Linux traceroute sends many probes at
once, and Linux limits how fast it sends the ICMP errors traceroute depends on (<code>net.ipv4.icmp_ratelimit</code>,
1000 ms on these hosts), so a burst of probes can get only some answers. If a count looks wrong, wait a moment
and run it again with <code>-q 1</code>.</p>
<hr>
<h2 id="what-does-a-listening-address-mean">What does a listening address mean?</h2>
<p>A server program <strong>listens</strong> on a port: it asks the operating system for every new connection to that port. It
also picks <strong>which of the machine&rsquo;s addresses</strong> to listen on. <code>ss -tlnp</code> shows both: TCP (<code>t</code>), listening
(<code>l</code>), numbers (<code>n</code>), and the process (<code>p</code>) that owns each socket
(<a href="https://man7.org/linux/man-pages/man8/ss.8.html">ss(8)</a>).</p>
<p>To see it, I started Python&rsquo;s built-in web server on ws1, listening only on loopback:
<code>python3 -m http.server --bind 127.0.0.1 9999</code>.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ ss -tln
</span></span><span class="line"><span class="cl">State  Recv-Q Send-Q Local Address:Port  Peer Address:Port
</span></span><span class="line"><span class="cl">LISTEN 0      5          127.0.0.1:9999       0.0.0.0:*
</span></span><span class="line"><span class="cl">LISTEN 0      128          0.0.0.0:22         0.0.0.0:*
</span></span><span class="line"><span class="cl">LISTEN 0      4096      127.0.0.11:35819      0.0.0.0:*
</span></span><span class="line"><span class="cl">LISTEN 0      128             [::]:22            [::]:*
</span></span></code></pre></div><p>Two programs, two very different promises. SSH on <code>0.0.0.0:22</code> takes connections on any IPv4 address ws1 has
(<code>[::]:22</code> is the same for IPv6). The web server on <code>127.0.0.1:9999</code> takes them only on loopback. The
<code>127.0.0.11</code> line is Docker&rsquo;s own DNS helper inside the container.</p>
<p><code>-p</code> adds the process, but only for sockets you own, unless you use <code>sudo</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ sudo ss -tlnp &#34;sport = :9999&#34;
</span></span><span class="line"><span class="cl">State  Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
</span></span><span class="line"><span class="cl">LISTEN 0      5          127.0.0.1:9999      0.0.0.0:*    users:((&#34;python3&#34;,pid=467,fd=3))
</span></span></code></pre></div><p>The <code>&quot;sport = :9999&quot;</code> part is a filter: only sockets whose source port is 9999. Then I tried the same server
listening three ways, and connected to it by both addresses:</p>
<table>
  <thead>
      <tr>
          <th>Listening on</th>
          <th><code>curl 127.0.0.1:9999</code></th>
          <th><code>curl 192.168.10.10:9999</code></th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>127.0.0.1</code></td>
          <td>the page</td>
          <td>refused</td>
      </tr>
      <tr>
          <td><code>192.168.10.10</code></td>
          <td>refused</td>
          <td>the page</td>
      </tr>
      <tr>
          <td><code>0.0.0.0</code></td>
          <td>the page</td>
          <td>the page</td>
      </tr>
  </tbody>
</table>
<p><code>127.0.0.1</code> is the &ldquo;works on the server, nowhere else&rdquo; case. Loopback traffic never leaves the machine, so no
other host can ever reach it, whatever the firewall says. Plenty of software listens there by default, on
purpose, until you tell it otherwise. A specific address, such as <code>192.168.10.10</code>, accepts only connections sent
to that address. <code>0.0.0.0</code> (or <code>*</code> in some tools) means all of them. When <code>ss</code> says the right port but the wrong
address, the fix is in the program&rsquo;s configuration, not the network.</p>
<hr>
<h2 id="what-does-curl--v-show">What does curl -v show?</h2>
<p><code>curl -v</code> (<em>verbose</em>) narrates the whole request: lines starting <code>*</code> are curl&rsquo;s notes, <code>&gt;</code> is what it sent,
<code>&lt;</code> is what came back. Here it is against the same test server on <code>0.0.0.0</code>, by name:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ curl -sv http://ws1.lab:9999/
</span></span><span class="line"><span class="cl">* Host ws1.lab:9999 was resolved.
</span></span><span class="line"><span class="cl">* IPv6: (none)
</span></span><span class="line"><span class="cl">* IPv4: 192.168.10.10
</span></span><span class="line"><span class="cl">*   Trying 192.168.10.10:9999...
</span></span><span class="line"><span class="cl">* Connected to ws1.lab (192.168.10.10) port 9999
</span></span><span class="line"><span class="cl">* using HTTP/1.x
</span></span><span class="line"><span class="cl">&gt; GET / HTTP/1.1
</span></span><span class="line"><span class="cl">&gt; Host: ws1.lab:9999
</span></span><span class="line"><span class="cl">&gt; User-Agent: curl/8.14.1
</span></span><span class="line"><span class="cl">&gt; Accept: */*
</span></span><span class="line"><span class="cl">&gt;
</span></span><span class="line"><span class="cl">* Request completely sent off
</span></span><span class="line"><span class="cl">* HTTP 1.0, assume close after body
</span></span><span class="line"><span class="cl">&lt; HTTP/1.0 200 OK
</span></span><span class="line"><span class="cl">&lt; Server: SimpleHTTP/0.6 Python/3.13.5
</span></span><span class="line"><span class="cl">&lt; Date: Wed, 11 Nov 2026 05:57:33 GMT
</span></span><span class="line"><span class="cl">&lt; Content-type: text/html
</span></span><span class="line"><span class="cl">&lt; Content-Length: 26
</span></span><span class="line"><span class="cl">&lt; Last-Modified: Wed, 11 Nov 2026 05:57:12 GMT
</span></span><span class="line"><span class="cl">&lt;
</span></span><span class="line"><span class="cl">{ [26 bytes data]
</span></span><span class="line"><span class="cl">* shutting down connection #0
</span></span><span class="line"><span class="cl">Hello from ws1, port 9999
</span></span></code></pre></div><p>Read it as layers, top down. The first block is <strong>DNS</strong>: the name became <code>192.168.10.10</code>. <code>Trying</code> then
<code>Connected</code> is <strong>TCP</strong>: the handshake worked. The <code>&gt;</code> lines are the <strong>HTTP</strong> request, and <code>&lt; HTTP/1.0 200 OK</code>
is the answer. (<code>-s</code> only hides the progress meter.) Wherever it stops tells you which layer failed:</p>
<table>
  <thead>
      <tr>
          <th>curl says</th>
          <th>Layer</th>
          <th>Usual cause</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>Could not resolve host: nosuch.lab</code> (exit 6)</td>
          <td>DNS</td>
          <td>wrong name, or the DNS server doesn&rsquo;t know it</td>
      </tr>
      <tr>
          <td><code>connect to ... failed: Connection refused</code> (exit 7)</td>
          <td>TCP</td>
          <td>the host answered, but nothing listens on that address and port</td>
      </tr>
      <tr>
          <td><code>connect to ... failed: No route to host</code> (exit 7)</td>
          <td>IP</td>
          <td>nobody has that address, or no route to it</td>
      </tr>
      <tr>
          <td><code>Connection timed out</code> or <code>Operation timed out</code> (exit 28)</td>
          <td>IP or firewall</td>
          <td>something dropped the packets silently</td>
      </tr>
      <tr>
          <td><code>&lt; HTTP/1.1 404 Not Found</code></td>
          <td>HTTP</td>
          <td>the server works, the path doesn&rsquo;t</td>
      </tr>
  </tbody>
</table>
<p>The two <code>connect</code> failures, from the lab: a port nobody listened on, and an address nobody had.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">* connect to 192.168.10.10 port 9999 from 192.168.10.10 port 38612 failed: Connection refused
</span></span><span class="line"><span class="cl">* connect to 192.168.20.99 port 80 from 192.168.10.10 port 39970 failed: No route to host
</span></span></code></pre></div><p>Refused came back in 5 ms. No route to host took about 3 seconds: most likely the time r1 spent asking for <code>192.168.20.99</code> with ARP
before it gave up and told ws1. <code>-m 5</code> caps the whole request at 5 seconds, which you want in a script.</p>
<p>For HTTPS, <code>-v</code> adds the <strong>TLS</strong> layer between TCP and HTTP. Trimmed, from the codespace&rsquo;s curl 7.88.1 against
this site:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">* Connected to networkdoodles.com (35.185.44.232) port 443 (#0)
</span></span><span class="line"><span class="cl">* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
</span></span><span class="line"><span class="cl">* ALPN: server accepted h2
</span></span><span class="line"><span class="cl">* Server certificate:
</span></span><span class="line"><span class="cl">*  subject: CN=networkdoodles.com
</span></span><span class="line"><span class="cl">*  expire date: Dec 11 12:42:43 2026 GMT
</span></span><span class="line"><span class="cl">*  issuer: C=US; O=Let&#39;s Encrypt; CN=YR1
</span></span><span class="line"><span class="cl">*  SSL certificate verify ok.
</span></span><span class="line"><span class="cl">&gt; GET / HTTP/2
</span></span><span class="line"><span class="cl">&lt; HTTP/2 200
</span></span></code></pre></div><p>That&rsquo;s the certificate&rsquo;s name, expiry and issuer without opening a browser. The
<a href="https://curl.se/docs/manpage.html">curl man page</a> lists every exit code.</p>
<hr>
<h2 id="when-do-i-want-digs-full-answer-and-when-short">When do I want dig&rsquo;s full answer, and when +short?</h2>
<p><code>dig +short</code> prints only the answer, which is what a script wants:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ dig +short srv1.lab
</span></span><span class="line"><span class="cl">192.168.20.10
</span></span></code></pre></div><p>The full answer is what you want when something&rsquo;s wrong:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ dig srv1.lab
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">; &lt;&lt;&gt;&gt; DiG 9.20.30-1~deb13u1-Debian &lt;&lt;&gt;&gt; srv1.lab
</span></span><span class="line"><span class="cl">;; global options: +cmd
</span></span><span class="line"><span class="cl">;; Got answer:
</span></span><span class="line"><span class="cl">;; -&gt;&gt;HEADER&lt;&lt;- opcode: QUERY, status: NOERROR, id: 51819
</span></span><span class="line"><span class="cl">;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">;; OPT PSEUDOSECTION:
</span></span><span class="line"><span class="cl">; EDNS: version: 0, flags:; udp: 1232
</span></span><span class="line"><span class="cl">;; QUESTION SECTION:
</span></span><span class="line"><span class="cl">;srv1.lab.			IN	A
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">;; ANSWER SECTION:
</span></span><span class="line"><span class="cl">srv1.lab.		60	IN	A	192.168.20.10
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">;; Query time: 2 msec
</span></span><span class="line"><span class="cl">;; SERVER: 192.168.20.10#53(192.168.20.10) (UDP)
</span></span><span class="line"><span class="cl">;; WHEN: Wed Nov 11 05:57:44 UTC 2026
</span></span><span class="line"><span class="cl">;; MSG SIZE  rcvd: 53
</span></span></code></pre></div><table>
  <thead>
      <tr>
          <th>Part</th>
          <th>Tells you</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>status: NOERROR</code></td>
          <td>the name exists. <code>NXDOMAIN</code> means it doesn&rsquo;t, <code>REFUSED</code> means this server won&rsquo;t answer</td>
      </tr>
      <tr>
          <td><code>flags: ... aa</code></td>
          <td><em>authoritative answer</em>: this server owns the name, it didn&rsquo;t ask anyone else</td>
      </tr>
      <tr>
          <td><code>60</code> in the answer</td>
          <td>the TTL: how many seconds clients may cache it</td>
      </tr>
      <tr>
          <td><code>SERVER:</code></td>
          <td>which DNS server actually answered</td>
      </tr>
  </tbody>
</table>
<p><code>+short</code> hides the status, and that matters. A name that doesn&rsquo;t exist prints <strong>nothing</strong>, and <code>dig</code> still
exits 0, because it did get an answer: <em>no such name</em>. The full output shows <code>status: NXDOMAIN</code>.</p>
<p>One more trap: <code>dig</code> ignores the <code>search lab</code> line in <code>/etc/resolv.conf</code>. <code>curl http://srv1</code> works, because it
uses the system&rsquo;s resolver, which adds <code>.lab</code>. <code>dig srv1</code> asks for exactly <code>srv1</code>, and the lab&rsquo;s DNS server
answers <code>status: REFUSED</code>. Use the full name, or <code>dig +search srv1</code>.</p>
<hr>
<h2 id="how-do-i-capture-packets-with-tcpdump">How do I capture packets with tcpdump?</h2>
<p><code>tcpdump</code> prints, or saves, the packets crossing an interface. It needs root, because seeing other programs'
traffic is a privilege:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ tcpdump -n -i eth1 -c 1
</span></span><span class="line"><span class="cl">tcpdump: eth1: You don&#39;t have permission to perform this capture on that device
</span></span><span class="line"><span class="cl">(socket: Operation not permitted)
</span></span></code></pre></div><p>With <code>sudo</code>, and a <strong>capture filter</strong> that keeps only DNS, while <code>dig +short srv1.lab</code> ran in another terminal:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ sudo tcpdump -n -i eth1 udp port 53
</span></span><span class="line"><span class="cl">tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
</span></span><span class="line"><span class="cl">listening on eth1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
</span></span><span class="line"><span class="cl">05:59:01.773866 IP 192.168.10.10.33468 &gt; 192.168.20.10.53: 35944+ [1au] A? srv1.lab. (49)
</span></span><span class="line"><span class="cl">05:59:01.782984 IP 192.168.20.10.53 &gt; 192.168.10.10.33468: 35944* 1/0/1 A 192.168.20.10 (53)
</span></span></code></pre></div><p>The question (<code>A? srv1.lab.</code>) and the answer (<code>A 192.168.20.10</code>), with each address written as
<code>address.port</code>.</p>
<table>
  <thead>
      <tr>
          <th>Option or filter</th>
          <th>Does</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>-i eth1</code></td>
          <td>capture on this interface. Pick the one <code>ip route get</code> named</td>
      </tr>
      <tr>
          <td><code>-n</code></td>
          <td>don&rsquo;t turn addresses and ports into names (and don&rsquo;t send DNS queries of its own)</td>
      </tr>
      <tr>
          <td><code>-c 10</code></td>
          <td>stop after 10 packets</td>
      </tr>
      <tr>
          <td><code>-w file.pcap</code></td>
          <td>save raw packets to a file instead of printing them</td>
      </tr>
      <tr>
          <td><code>-r file.pcap</code></td>
          <td>read a saved file back</td>
      </tr>
      <tr>
          <td><code>host 192.168.20.10</code></td>
          <td>only packets to or from this address</td>
      </tr>
      <tr>
          <td><code>port 8080</code>, <code>tcp port 22</code></td>
          <td>only this port</td>
      </tr>
      <tr>
          <td><code>and</code>, <code>or</code>, <code>not</code></td>
          <td>combine them: <code>host 192.168.20.10 and tcp port 22</code></td>
      </tr>
  </tbody>
</table>
<p>The filter language is libpcap&rsquo;s, documented in <a href="https://www.tcpdump.org/manpages/pcap-filter.7.html">pcap-filter(7)</a>,
and the options in <a href="https://www.tcpdump.org/manpages/tcpdump.1.html">tcpdump(1)</a>. A capture filter decides what&rsquo;s
kept at all: anything it drops is gone. Wireshark 101&rsquo;s lesson on
<a href="/lessons/wireshark-101/capture-vs-display-filters/">capture filters vs display filters</a> covers the difference.</p>
<p>Saving with <code>-w</code> and reading back with <code>-r</code>: a test connection from ws1 to srv1&rsquo;s SSH port, written to a file
and then read back:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">$ sudo tcpdump -n -i eth1 -w /tmp/lesson8-ssh.pcap host 192.168.20.10 and tcp port 22
</span></span><span class="line"><span class="cl">tcpdump: listening on eth1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
</span></span><span class="line"><span class="cl">6 packets captured
</span></span><span class="line"><span class="cl">$ tcpdump -n -r /tmp/lesson8-ssh.pcap
</span></span><span class="line"><span class="cl">reading from file /tmp/lesson8-ssh.pcap, link-type EN10MB (Ethernet), snapshot length 262144
</span></span><span class="line"><span class="cl">05:59:05.387230 IP 192.168.10.10.60292 &gt; 192.168.20.10.22: Flags [S], seq 3914208904, win 64240, options [mss 1460,sackOK,TS val 1308834635 ecr 0,nop,wscale 7], length 0
</span></span><span class="line"><span class="cl">05:59:05.387361 IP 192.168.20.10.22 &gt; 192.168.10.10.60292: Flags [S.], seq 1686286914, ack 3914208905, win 65160, options [mss 1460,sackOK,TS val 590197084 ecr 1308834635,nop,wscale 7], length 0
</span></span><span class="line"><span class="cl">05:59:05.387369 IP 192.168.10.10.60292 &gt; 192.168.20.10.22: Flags [.], ack 1, win 502, options [nop,nop,TS val 1308834635 ecr 590197084], length 0
</span></span><span class="line"><span class="cl">05:59:05.420529 IP 192.168.20.10.22 &gt; 192.168.10.10.60292: Flags [P.], seq 1:42, ack 1, win 510, options [nop,nop,TS val 590197117 ecr 1308834635], length 41: SSH: SSH-2.0-OpenSSH_10.0p2 Debian-7+deb13u4
</span></span><span class="line"><span class="cl">05:59:05.420549 IP 192.168.10.10.60292 &gt; 192.168.20.10.22: Flags [.], ack 42, win 502, options [nop,nop,TS val 1308834668 ecr 590197117], length 0
</span></span><span class="line"><span class="cl">05:59:05.420966 IP 192.168.10.10.60292 &gt; 192.168.20.10.22: Flags [R.], seq 1, ack 42, win 502, options [nop,nop,TS val 1308834668 ecr 590197117], length 0
</span></span></code></pre></div><p>Reading the file back needs no <code>sudo</code>. The first three lines are the TCP handshake: <code>[S]</code> (SYN), <code>[S.]</code>
(SYN-ACK) and <code>[.]</code> (ACK). Then srv1 sends its SSH banner, and my test hung up abruptly, so the last packet is a
reset, <code>[R.]</code>. <code>60292</code> is ws1&rsquo;s temporary port for this one connection. A <strong>pcap</strong> file like this opens in
Wireshark, which is the better place to read anything longer than a handful of lines:
<a href="/lessons/wireshark-101/tcp/">TCP in Wireshark</a> reads the same flags field by field.</p>
<hr>
<h2 id="what-i-see-go-wrong">What I see go wrong</h2>
<p><strong>&ldquo;It works on the server.&rdquo;</strong> <code>curl localhost</code> on the server proves the program runs. It doesn&rsquo;t prove anyone
else can reach it. Check the address in <code>ss -tlnp</code> before you look at the network.</p>
<p><strong>Reading <code>0.0.0.0</code> in the Peer column as a problem.</strong> On a <code>LISTEN</code> line, <code>0.0.0.0:*</code> under Peer Address just
means nobody&rsquo;s connected yet. The address that matters is the Local one.</p>
<p><strong><code>dig</code> and the application disagree.</strong> <code>dig</code> asks the DNS server directly and skips <code>/etc/hosts</code> and the
search list. The application uses the system resolver, which reads both. <code>getent hosts srv1</code> asks the way the
application does.</p>
<p><strong><code>dig +short</code> printing nothing.</strong> That&rsquo;s an answer: the name doesn&rsquo;t exist (or has no record of that type).
Run it without <code>+short</code> and read the status.</p>
<p><strong>Capturing on the wrong interface.</strong> ws1 has two. Traffic to srv1 leaves by <code>eth1</code>; a capture on <code>eth0</code>
shows nothing and looks like &ldquo;no traffic&rdquo;. <code>ip route get</code> says which one.</p>
<p><strong>Forgetting <code>-n</code> in tcpdump.</strong> Name lookups slow the output down, and the DNS queries tcpdump sends for them
can end up in your own capture.</p>
<p><strong>A <code>-w</code> file that&rsquo;s short or empty.</strong> tcpdump buffers what it writes. Stop it with Ctrl-C, or let <code>-c</code> stop it,
so it finishes writing the file. Killed any harder, it can leave the last packets out.</p>
<hr>
<h2 id="try-it">Try it</h2>
<p><a href="/labs/linux-cli-101/task-8-1/">Chapter 8 of the lab</a> works from ws1 towards srv1, through r1. You find the
gateway and the outgoing interface, count the hops and resolve srv1 with <code>dig</code>. Then a web service on srv1
answers when you <code>curl</code> it on srv1 itself but not from ws1, and it&rsquo;s yours to find out why and fix.
Last, you capture the conversation with <code>tcpdump -w</code>, copy the file back with <code>scp</code>, and read the handshake out
of it.</p>
<h2 id="where-to-go-next">Where to go next</h2>
<p>That&rsquo;s the end of Linux and the command line 101. You can now move around the filesystem, read and search files,
chain tools with pipes, fix permissions, manage processes, log in with keys, script a check, and troubleshoot a
service from the command line. Every lab on this site assumes those.</p>
<p>Two courses pick up from here:</p>
<ul>
<li><a href="/lessons/wireshark-101/">Wireshark 101</a> takes the pcap files <code>tcpdump</code> saves and reads them properly:
ARP, DHCP, DNS, the TCP handshake and TLS, and a broken path to troubleshoot.</li>
<li><a href="/lessons/python-101/">Python 101</a> takes over where Bash gets awkward: real data structures, parsing <code>show</code>
output, and logging in to every device from one script.</li>
</ul>
<p>If you want a refresher first, the course starts at <a href="/lessons/linux-cli-101/shell-and-filesystem/">the shell and the filesystem</a>.</p>
]]></content:encoded></item><item><title>TCP in Wireshark: Handshake, Sequence Numbers, FIN and RST (Wireshark 101, Lesson 6)</title><link>https://networkdoodles.com/lessons/wireshark-101/tcp/</link><pubDate>Mon, 16 Nov 2026 09:00:00 -0500</pubDate><dc:creator>David Henderson</dc:creator><guid>https://networkdoodles.com/lessons/wireshark-101/tcp/</guid><category>wireshark</category><category>packet-capture</category><category>networking-basics</category><category>tcp</category><description>A TCP connection in Wireshark from SYN to FIN: handshake, sequence and ACK numbers, window, MSS, streams, and a refused connection versus silence.</description><content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[<p>TCP turns a network that loses and reorders packets into a reliable stream of bytes, and it does all of that
in the open, where a capture can watch. A connection opens with three packets (SYN, SYN-ACK, ACK), numbers
every byte it sends, acknowledges what arrived, and closes each direction with a FIN. A refused connection is
two packets: a SYN and a RST. Once you can read those, you can read most of what goes wrong on a network.</p>
<p>Every protocol in this course so far (ARP, DHCP, DNS) has been one question and one answer. A web page can&rsquo;t
work like that, so this lesson follows one: <code>curl http://web.lab/</code> in the
<a href="/labs/wireshark-101/">Wireshark 101 lab</a>, captured on the laptop&rsquo;s cable.</p>
<hr>
<h2 id="what-does-a-whole-tcp-connection-look-like">What does a whole TCP connection look like?</h2>
<p>Here&rsquo;s the entire connection, from the reference capture. DNS for <code>web.lab</code> came first and isn&rsquo;t shown.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl"> 6 0.021836458 192.168.1.175 → 192.168.2.10 TCP 74 47996 → 80 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM TSval=2403693004 TSecr=0 WS=128
</span></span><span class="line"><span class="cl"> 7 0.022324792 192.168.2.10 → 192.168.1.175 TCP 74 80 → 47996 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1460 SACK_PERM TSval=4000512039 TSecr=2403693004 WS=128
</span></span><span class="line"><span class="cl"> 8 0.022400375 192.168.1.175 → 192.168.2.10 TCP 66 47996 → 80 [ACK] Seq=1 Ack=1 Win=64256 Len=0 ...
</span></span><span class="line"><span class="cl"> 9 0.022699583 192.168.1.175 → 192.168.2.10 HTTP 137 GET / HTTP/1.1
</span></span><span class="line"><span class="cl">10 0.022722250 192.168.2.10 → 192.168.1.175 TCP 66 80 → 47996 [ACK] Seq=1 Ack=72 Win=65152 Len=0 ...
</span></span><span class="line"><span class="cl">11 0.025193000 192.168.2.10 → 192.168.1.175 TCP 302 HTTP/1.1 200 OK
</span></span><span class="line"><span class="cl">12 0.025222875 192.168.1.175 → 192.168.2.10 TCP 66 47996 → 80 [ACK] Seq=72 Ack=237 Win=64128 Len=0 ...
</span></span><span class="line"><span class="cl">13 0.025324750 192.168.2.10 → 192.168.1.175 HTTP 99 HTTP/1.1 200 OK  (text/html)
</span></span><span class="line"><span class="cl">14 0.025327875 192.168.1.175 → 192.168.2.10 TCP 66 47996 → 80 [ACK] Seq=72 Ack=270 Win=64128 Len=0 ...
</span></span><span class="line"><span class="cl">15 0.025727375 192.168.1.175 → 192.168.2.10 TCP 66 47996 → 80 [FIN, ACK] Seq=72 Ack=270 Win=64128 Len=0 ...
</span></span><span class="line"><span class="cl">16 0.026296000 192.168.2.10 → 192.168.1.175 TCP 66 80 → 47996 [FIN, ACK] Seq=270 Ack=73 Win=65152 Len=0 ...
</span></span><span class="line"><span class="cl">17 0.026305292 192.168.1.175 → 192.168.2.10 TCP 66 47996 → 80 [ACK] Seq=73 Ack=271 Win=64128 Len=0 ...
</span></span></code></pre></div><p>Twelve frames, four and a half milliseconds, for a 33-byte page. Three open the connection, three carry data
(the request, and the answer in two pieces), three acknowledge that data, and three close it. Most of the packets are TCP
keeping its promises, not carrying the page.</p>
<p><code>47996 → 80</code> is the two <strong>ports</strong>. A port says which program at each end owns the connection. Port 80 is the
web server. 47996 is an <strong>ephemeral port</strong> the laptop picked for this one connection; the next connection gets
a different one. The four numbers together (two addresses, two ports) identify a connection uniquely.</p>
<hr>
<h2 id="the-three-way-handshake">The three-way handshake</h2>
<p>Before any data moves, the two ends agree to talk and tell each other where their numbering starts.</p>
<ol>
<li><strong>SYN</strong> (synchronise), from the laptop: &ldquo;I&rsquo;d like to connect, and my bytes start here.&rdquo;</li>
<li><strong>SYN, ACK</strong>, from web: &ldquo;Agreed. Mine start here, and I&rsquo;ve got your SYN.&rdquo;</li>
<li><strong>ACK</strong>, from the laptop: &ldquo;Got yours.&rdquo;</li>
</ol>
<p>After frame 8, both sides know each other&rsquo;s starting number, and either can send. The time from the SYN to the
final ACK is the <strong>initial round-trip time</strong>. Wireshark calculates it as <code>tcp.analysis.initial_rtt</code> and shows
it on every packet of the stream: here it&rsquo;s 0.56 ms. It&rsquo;s a useful number to have, because every later delay in
the connection can be measured against it.</p>
<hr>
<h2 id="sequence-numbers-raw-and-relative">Sequence numbers: raw and relative</h2>
<p>TCP numbers <strong>bytes</strong>, not packets. Every segment carries two numbers:</p>
<ul>
<li><strong>Seq</strong>: the number of the first byte of data in this segment;</li>
<li><strong>Ack</strong>: the next byte this side expects from the other one. It means &ldquo;I have everything before this&rdquo;.</li>
</ul>
<p>Each side starts at a random <strong>initial sequence number</strong> (ISN), chosen fresh for every connection so that
stray packets from an old connection, or a guess from an attacker, don&rsquo;t land inside a new one. In this capture
the laptop&rsquo;s ISN was 500774591 and web&rsquo;s was 702278227. Those are hard to read, so Wireshark subtracts the ISN
and shows <strong>relative</strong> numbers that start at 0. The raw values are still in the packet as <code>tcp.seq_raw</code> and
<code>tcp.ack_raw</code>. You need them when you compare captures from two points and something in between rewrites
sequence numbers, which some firewalls do.</p>
<p>Now follow the numbers through the connection:</p>
<table>
  <thead>
      <tr>
          <th>Frame</th>
          <th>Who</th>
          <th>Seq</th>
          <th>Ack</th>
          <th>Len</th>
          <th>What it says</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>6</td>
          <td>laptop</td>
          <td>0</td>
          <td></td>
          <td>0</td>
          <td>SYN. A SYN counts as one byte</td>
      </tr>
      <tr>
          <td>7</td>
          <td>web</td>
          <td>0</td>
          <td>1</td>
          <td>0</td>
          <td>&ldquo;your SYN arrived; next I expect byte 1&rdquo;</td>
      </tr>
      <tr>
          <td>9</td>
          <td>laptop</td>
          <td>1</td>
          <td>1</td>
          <td>71</td>
          <td>the HTTP request, bytes 1 to 71</td>
      </tr>
      <tr>
          <td>10</td>
          <td>web</td>
          <td>1</td>
          <td>72</td>
          <td>0</td>
          <td>&ldquo;got all 71; next I expect byte 72&rdquo;</td>
      </tr>
      <tr>
          <td>11</td>
          <td>web</td>
          <td>1</td>
          <td>72</td>
          <td>236</td>
          <td>HTTP headers</td>
      </tr>
      <tr>
          <td>13</td>
          <td>web</td>
          <td>237</td>
          <td>72</td>
          <td>33</td>
          <td>the page itself</td>
      </tr>
      <tr>
          <td>14</td>
          <td>laptop</td>
          <td>72</td>
          <td>270</td>
          <td>0</td>
          <td>&ldquo;got everything up to 269&rdquo;</td>
      </tr>
  </tbody>
</table>
<p><code>Ack=72</code> after a 71-byte request starting at 1 is the whole idea. You can check, packet by packet, exactly
what each side has received. When something goes missing, it shows up as an Ack that stops moving, which is
what <a href="/lessons/wireshark-101/broken-path/">lesson 8</a> is about.</p>
<p>Frames 11 and 13 also show something you&rsquo;ll see a lot: nginx sent its headers and the body in <strong>separate
segments</strong>, 236 bytes and then 33. TCP doesn&rsquo;t care where one HTTP message ends; it carries a stream. Wireshark
reassembles the pieces and shows the complete HTTP response on the last one, frame 13, which is why frame 11&rsquo;s
Protocol column says TCP and not HTTP.</p>
<hr>
<h2 id="the-window-and-the-options-in-the-syn">The window, and the options in the SYN</h2>
<p><strong>Win</strong> is the receive <strong>window</strong>: how many more bytes this side can accept right now, before it needs the
sender to wait. It&rsquo;s flow control. A receiver that&rsquo;s falling behind advertises a smaller window, and a window
of zero means &ldquo;stop&rdquo;.</p>
<p>The window field is only 16 bits, so it can&rsquo;t say more than 65,535 bytes. That&rsquo;s where the SYN&rsquo;s options come
in. A SYN is the only time each side can announce what it supports, and the lab&rsquo;s SYN has four:</p>
<table>
  <thead>
      <tr>
          <th>Option</th>
          <th>In the SYN</th>
          <th>Means</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><strong>MSS</strong></td>
          <td><code>MSS=1460</code></td>
          <td>the biggest chunk of data this side can receive in one segment: a 1,500-byte MTU minus 20 bytes of IP header and 20 of TCP</td>
      </tr>
      <tr>
          <td><strong>SACK_PERM</strong></td>
          <td><code>SACK_PERM</code></td>
          <td>&ldquo;I understand selective acknowledgements&rdquo; (<a href="https://www.rfc-editor.org/rfc/rfc2018">RFC 2018</a>). Used when packets are lost, in lesson 8</td>
      </tr>
      <tr>
          <td><strong>Timestamps</strong></td>
          <td><code>TSval=… TSecr=…</code></td>
          <td>a clock value in every segment, echoed back, for measuring round-trip time (<a href="https://www.rfc-editor.org/rfc/rfc7323">RFC 7323</a>)</td>
      </tr>
      <tr>
          <td><strong>Window scale</strong></td>
          <td><code>WS=128</code></td>
          <td>multiply every later window value by 128 (RFC 7323)</td>
      </tr>
  </tbody>
</table>
<p>Window scaling explains a number that looks wrong at first. Frame 8 says <code>Win=64256</code>, but the raw window field
in the packet is 502. Wireshark saw <code>WS=128</code> in the handshake and did the multiplication: 502 × 128 = 64,256.
It can only do that if it captured the SYNs. Start a capture halfway through a connection and Wireshark doesn&rsquo;t
know the scale factor, so the windows it shows are too small.</p>
<p>Options only count if both sides send them. If one side&rsquo;s SYN has no <code>SACK_PERM</code>, the connection doesn&rsquo;t use
SACK. Comparing the options in the SYN and the SYN-ACK is how you find out what the connection really
negotiated.</p>
<hr>
<h2 id="fin-and-rst-two-ways-to-end">FIN and RST: two ways to end</h2>
<p>A connection closes politely with <strong>FIN</strong> (finish). Each direction closes separately: a FIN means &ldquo;I&rsquo;ve
finished sending&rdquo;, not &ldquo;go away&rdquo;. In the lab, curl had the whole page, so it sent the first FIN (frame 15). web
answered with its own FIN (16), and the laptop acknowledged it (17). A FIN counts as one byte, just like a SYN,
which is why the Ack goes from 72 to 73. Whoever sends the first FIN chose to end the conversation. Here it was
the client, even though nginx had offered <code>Connection: keep-alive</code>.</p>
<p><strong>RST</strong> (reset) is the other way: &ldquo;this connection doesn&rsquo;t exist, stop&rdquo;. No goodbye, no acknowledgement. A RST
mid-connection usually means one side crashed, timed out, or decided something was badly wrong. A RST in answer
to a SYN means something much more specific.</p>
<hr>
<h2 id="refused-versus-no-answer">Refused versus no answer</h2>
<p>Here&rsquo;s the laptop trying port 8080 on web, where nothing listens:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">21 0.071235375 192.168.1.175 → 192.168.2.10 TCP 74 55482 → 8080 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM ...
</span></span><span class="line"><span class="cl">22 0.071316959 192.168.2.10 → 192.168.1.175 TCP 54 8080 → 55482 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0
</span></span></code></pre></div><p>Two packets, 0.08 ms apart. web&rsquo;s operating system got a SYN for a port no program has open and answered
straight away with a RST. curl reports <em>Connection refused</em> and stops.</p>
<p>A refusal is good news for troubleshooting. It proves the network path works in both directions, and that
something at the far end is up and answering. The problem is the service: wrong port, program not running,
program listening on a different address.</p>
<p>Compare that with a SYN that gets <strong>no answer at all</strong>. The client can&rsquo;t tell whether the SYN was lost, so it
sends it again, and again, until it gives up. In the lab&rsquo;s broken-path chapter,
the laptop sent the same SYN at 0, 1, 2, 3, 4, 5 and 7 seconds before curl&rsquo;s eight-second limit ran out. From the
application, that&rsquo;s a hang: <em>Connection timed out</em>, eventually.</p>
<table>
  <thead>
      <tr>
          <th>What you see</th>
          <th>What it means</th>
          <th>Where to look</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>SYN, then RST, ACK</td>
          <td>the host is reachable; nothing listens on that port</td>
          <td>the server: is the service running, on that port?</td>
      </tr>
      <tr>
          <td>SYN, SYN again, SYN again…</td>
          <td>something on the path dropped the SYN or the answer</td>
          <td>the network: a filter, a route, a firewall</td>
      </tr>
      <tr>
          <td>SYN, SYN-ACK, then nothing works</td>
          <td>the connection opened; the problem is later</td>
          <td>sizes, the application, lesson 8</td>
      </tr>
  </tbody>
</table>
<p>One caution: a RST proves <em>something</em> answered, not necessarily the server. Some firewalls reject a
connection by sending a RST on the server&rsquo;s behalf. If a refusal makes no sense, capture closer to the server
and see whether the SYN ever arrived.</p>
<hr>
<h2 id="streams-follow-tcp-stream-and-conversations">Streams, Follow TCP Stream and Conversations</h2>
<p>A real capture holds many connections at once. Wireshark numbers each one, in the order it first sees them,
as <code>tcp.stream</code>. In the lab&rsquo;s refused capture, stream 0 is the first page, stream 1 the refused attempt and
stream 2 the page again. The filter <code>tcp.stream == 1</code> shows one connection and nothing else, which is the
fastest way to pull one conversation out of thousands of frames.</p>
<p>Two views build on streams.</p>
<p><strong>Follow › TCP Stream</strong> strips away the packets and shows the bytes each side sent, in order, one color per
side. For plain HTTP that&rsquo;s the request and the response as text:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">GET / HTTP/1.1
</span></span><span class="line"><span class="cl">Host: web.lab
</span></span><span class="line"><span class="cl">User-Agent: curl/8.14.1
</span></span><span class="line"><span class="cl">Accept: */*
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">HTTP/1.1 200 OK
</span></span><span class="line"><span class="cl">Server: nginx/1.28.3
</span></span><span class="line"><span class="cl">Content-Type: text/html
</span></span><span class="line"><span class="cl">Content-Length: 33
</span></span><span class="line"><span class="cl">...
</span></span><span class="line"><span class="cl">Hello from web.lab, 192.168.2.10
</span></span></code></pre></div><p><strong>Statistics › Conversations › TCP</strong> lists every connection in the capture with its addresses, ports, frames,
bytes and duration. For this page: 12 frames and 1,148 bytes, seven frames from the laptop and five from web.
Sort it by bytes and the biggest transfers come to the top. Sort it by duration and the connections that hung
around do.</p>
<hr>
<h2 id="what-i-see-go-wrong">What I see go wrong</h2>
<p><strong>Starting the capture too late.</strong> Without the handshake, Wireshark doesn&rsquo;t know the window scale, the MSS or
the initial RTT, and its analysis is weaker. Start capturing before the connection opens.</p>
<p><strong>Counting packets instead of bytes.</strong> Seq and Ack count bytes. A missing packet shows up as a gap in the byte
numbers, and two segments of very different sizes can carry one HTTP message. Read the numbers, not the
number of lines.</p>
<p><strong>Calling every RST a failure.</strong> A RST answering a SYN is the clearest answer TCP has. It&rsquo;s silence that
should worry you.</p>
<p><strong>Assuming the client closed it.</strong> Whoever sent the first FIN (or the RST) ended the connection. When a
connection &ldquo;drops&rdquo;, check which side did it first. The answer often points straight at a timeout on the other
end.</p>
<hr>
<h2 id="try-it">Try it</h2>
<p><a href="/labs/wireshark-101/#chapter-6">Chapter 6 of the lab</a> has you capture a page load, find the handshake and
the FINs, then pick the refused connection out of three by its stream index.</p>
<h2 id="whats-next">What&rsquo;s next</h2>
<p>The same connection on port 443 starts with a handshake of its own, and most of it is still readable.
<a href="/lessons/wireshark-101/tls-without-decryption/">Lesson 7</a> is about what TLS shows without decrypting
anything, and how to decrypt your own when you need to. Back to <a href="/lessons/wireshark-101/dns/">lesson 5, DNS</a>.</p>
]]></content:encoded></item></channel></rss>