online posts 26
posts

Emulate the Question, Not the Network: Digital Twins at Scale (AutoCon 6, Part 2)

Russ White’s AutoCon 6 tutorial built a butterfly fabric in ContainerLab and found where it breaks. The lesson I took home: build one twin per question, script it by hand, and use LLMs for helpers, not the topology.

Cover: one twin per question. A terminal shows routers brought up in staged batches in ContainerLab with IS-IS adjacencies released in order, next to a chain from question to scoped twin to answer
part 2 of 11 · series AutoCon 6
  1. 1State of automation: start with your alerts
  2. 2Digital twins at scale
  3. 3Observability and blast radius
  4. 4AI evaluation
  5. 5Automation ROI
  6. 6Leading radical change
  7. 7Build vs. buy
  8. 8Wouldn't it be cool if...
  9. 9Lightning: the PB&J method
  10. 10Lightning: 3,000 routers
  11. 11Lightning: how much code is enough
series overview →
Short answer
Don’t try to build a digital twin of your whole network. Build a twin of the question you’re asking: break the network into functional pieces, emulate only the piece the question touches, and strip the containerized NOS down to what the test needs. Script the topology yourself and let LLMs write helper scripts, because a model asked to generate a 200-router topology gives you a different one every time.

I’ve built network digital twins a few different ways: graph models for path tracing, emulated labs for change testing. The AutoCon 6 talk closest to my own experience was Russ White’s tutorial on bringing up very large ContainerLab topologies, and it opened with the right principle. Don’t build a twin of the network. Build a twin of the question you’re trying to answer.

Why build one twin per question instead of one big twin?

A single all-encompassing twin isn’t realistic today. CPU is the hard limit. Russ described twinning as a continuum from two routers to thousands, and the job is finding the point on it that answers what you’re asking. Break the network into functional and topological pieces and run one twin per piece.

If you’re testing control plane behavior, use the containerized version of your NOS and strip it down to the routing stack. If you’re testing data plane, keep the full stack but run fewer nodes. You don’t need all 1,000 paths through a fabric to prove a forwarding behavior.

His three highest-value use cases:

  1. Validating orchestration and management tooling
  2. Version control and model diffing
  3. End-to-end policy verification across nodes

How do you bring up a large ContainerLab topology without crashing the host?

  • Split config by stage. Link addressing goes in the boot config. Routing and overlay get applied after boot. Keep IS-IS and BGP out of base templates.
  • One template per router type. Use placeholder characters that never appear in real config, and let a script stamp out per-pod directories. Three router types means three templates.
  • Stagger protocol bring-up. He set a unique IS-IS password per device at boot to block adjacencies, then stripped them interface by interface to control the convergence rate.
  • Set the Docker restart policy to none. Otherwise a host reboot restarts 500 containers at once and takes the box down with them.
  • Prefer IS-IS for lab scale. It’s interface-based and easier to script than OSPF or BGP.

How far does a ContainerLab topology scale?

TopologyResult on an 8 vCPU / 32 GB VM
2 pods, 128 routersComfortable on 4 vCPUs, full bring-up with IS-IS in 10-15 minutes
3 pods, 160 routersAbout 6 vCPUs
4 pods, 192 routersEvery vCPU consumed
224 routersFailed to converge 20-30% of the time, even with timer tuning

Physical cores outperformed cloud vCPUs by a wide margin, and the Linux kernel’s bridge limit becomes a hard ceiling once you split a topology across ContainerLab instances. One bright spot: an FRRouting pull request for optimized IS-IS flooding cuts CPU about 40%, which could roughly double those numbers.

My take: where do LLMs fit in building a twin?

The part I agree with most: Russ wrote the scripts by hand first, then used LLMs to build helper scripts, not the topology itself. Ask a model to generate a 200-router topology and you’ll get a different one every time. A twin that isn’t repeatable isn’t a twin. It’s the same principle I apply to agents in production: let the model help build deterministic tooling, then run the tooling.

Most enterprises don’t need 200 routers in ContainerLab. They need the 15 devices in the path of the change they’re about to make, generated from their source of truth on demand. That only works if the twin knows what the change can break, which is the blast radius argument from my observability post. And if you want a lab NOS with a familiar CLI for the small version of this, that’s why I built CapeOS.

The scale work matters because it shows you where the ceiling is. Deciding which question the twin answers is what makes it useful.

This is part 2 of my AutoCon 6 series.