online posts 16
wireshark 101

task 8.3 · chapter 8 of 8: a broken path · 20 min

The black hole

objective

Someone turned on jumbo frames on the laptop and on web, but not on the network between them. Ping works and HTTPS hangs. Find the frame that never gets through, and where it disappears.

you edit, in the labanswers/8.3.yml

open a file in VS Code: Ctrl+P (Cmd+P on a Mac), type its name, Enter

given
The faultmake break X=mtu: the laptop’s and web’s MTU set to 9000
Capturemake capture AT=laptop:eth1,sw1:eth1 NAME=8.3 X=tls
Afterwardsmake fix
Your answersanswers/8.3.yml
new to this? step by stepbeginner
  1. make break X=mtu; from the laptop, ping -c 2 web.lab works, curl -m 5 https://web.lab/ doesn’t.
  2. make capture AT=laptop:eth1,sw1:eth1 NAME=8.3 X=tls.
  3. Filter frame.len > 1514: which interface has them, which doesn’t? Note the frame length.
  4. Challenge: who dropped them, and why. make fix, then make check T=8.3.

more in meet the lab

1 explain

When small packets work and big ones don’t

MTU

Every link has an MTU (maximum transmission unit): the biggest packet it can carry in one frame. On ordinary Ethernet it’s 1,500 bytes of IP packet, which makes a 1,514-byte frame once the Ethernet header is added. Some networks use jumbo frames, often 9,000 bytes, to move data with fewer packets.

At the start of a TCP connection, each side announces its MSS (maximum segment size): the biggest chunk of data it can take, worked out from its own MTU (1,460 bytes for 1,500; 8,960 for 9,000). Each side then sends segments no bigger than the other’s MSS.

The black hole

Here’s the trap. If two hosts both have jumbo frames, they each announce 8,960, and each happily sends big segments. But if a link between them only takes 1,500, the big frames don’t fit. A router would at least be able to send back an ICMP message saying too big (that’s how Path MTU Discovery works). A switch can’t send anything: it works at layer 2, has no IP address on the path, and drops the oversized frame without a word.

So small packets (the handshake, ping, DNS) all work, and the first big one disappears. The connection opens and then hangs. That’s called a black hole: packets go in and nothing comes out, and nobody’s told.

in one sentence An MTU mismatch lets small packets through and silently drops big ones: the connection opens, then hangs.

Why TLS finds it first

Remember task 7.1: the laptop’s TLS 1.3 Client Hello is 1,558 bytes, because of its post-quantum key share. On a normal 1,500-byte link TCP splits it in two, but with the laptop thinking it has 9,000 bytes to play with, it goes as one segment. HTTPS breaks before anything else does, while ping, DNS and plain HTTP requests, all small, carry on working.

2 show

Sent, sent, sent

Both capture points, TCP only:

11	laptop:eth1	74	42898 → 443 [SYN] Seq=0 Win=62720 Len=0 MSS=8960 ...
12	sw1:eth1	74	[TCP Retransmission] 42898 → 443 [SYN] Seq=0 ... MSS=8960 ...
13	sw1:eth1	74	443 → 42898 [SYN, ACK] Seq=0 Ack=1 ... MSS=8960 ...
14	laptop:eth1	74	[TCP Retransmission] 443 → 42898 [SYN, ACK] Seq=0 Ack=1 ... MSS=8960 ...
15	laptop:eth1	66	42898 → 443 [ACK] Seq=1 Ack=1 ...
16	sw1:eth1	66	[TCP Dup ACK 15#1] 42898 → 443 [ACK] Seq=1 Ack=1 ...
17	laptop:eth1	1624	Client Hello (SNI=web.lab)
18	laptop:eth1	1624	[TCP Retransmission] 42898 → 443 [PSH, ACK] Seq=1 Ack=1 ... Len=1558
19	laptop:eth1	1624	[TCP Retransmission] 42898 → 443 [PSH, ACK] Seq=1 Ack=1 ... Len=1558
...

The third column is the frame length. The handshake crosses both points (the Retransmission and Dup ACK labels on frames 12 to 16 are the same packets seen twice: task 8.2 explains why to ignore them here). Both sides announced MSS=8960. Then the Client Hello, 1,624 bytes, goes out on the laptop’s cable again and again, and never once shows up on sw1:eth1.

3 follow along

Reproduce it, then capture it

  1. Break it, in the VS Code tab’s terminal:

    run in the terminal
    make break X=mtu
    

    Then see the symptom from the laptop: click laptop on the Topology tab, and in the laptop tab:

    run in the laptop tab
    ping -c 2 web.lab
    curl -m 5 https://web.lab/
    ip link show eth1
    

    Ping answers. curl gives up after five seconds. ip link shows mtu 9000.

  2. Capture the laptop’s cable and the switch’s port towards r1 together, while the laptop tries HTTPS:

    run in the terminal
    make capture AT=laptop:eth1,sw1:eth1 NAME=8.3 X=tls
    
  3. Open /captures/8.3.pcapng and filter the frames too big for a normal cable:

    frame.len > 1514
    

    Note their length, and which interface they’re on. Then add && frame.interface_name == "sw1:eth1".

  4. Fill in the follow-along lines:

    editanswers/8.3.ymlCtrl+P → 8.3.yml
    stuck_frame_bytes: 1624
    big_frames_past_switch: 0
    
4 challenge

Who, and why

  • dropped_by: the box that dropped the frames;
  • laptop_mtu: the laptop’s MTU, the cause.

Then put it right with make fix, and try the curl again.

hint

The frames are on the laptop’s cable and not on the far side of the switch. Only one box is in between.

hint

Why didn’t anything tell the laptop? Which kinds of device can send ICMP too big?

show the solutionanswers/8.3.yml

or pull it into your lab: make solve T=8.3, then git diff --staged shows what changed

dropped_by: sw1
laptop_mtu: 9000

sw1’s ports take 1,500-byte packets. A 1,624-byte frame arrived from the laptop, didn’t fit, and was dropped. A switch has no way to say so, so the laptop kept retransmitting into the hole until curl gave up.

The fix is to make every device on a network agree on the MTU: either jumbo frames everywhere on the path, or nowhere. And the habit to keep: when ping works but the application doesn’t, filter frame.len > 1514 and look for frames that are sent over and over and never acknowledged.

5 check: run it, then prove it

$ make check T=8.3

when it's right, you'll see

╭─ task 8.3 · the black hole
│  ✓ captures/8.3.pcapng: taken on laptop:eth1, sw1:eth1, ... frames
│  ✓ the capture has the laptop's oversized frames: the fault was on
│  ✓ the size of the frame that never got an answer (follow along): ...
│  ✓ how many of them reached r1's side of the switch (follow along): 0
│  ✓ the box that dropped them (challenge): sw1
│  ✓ the laptop's MTU, the cause (challenge): 9000
╰─ 6/6 checks passed  verified