3,000 Routers in 17 Hours With Tools They Already Had (AutoCon 6, Part 10)
Mark Elias’s AutoCon 6 lightning talk: how an AT&T team recombined backup, audit, config and push tools to change 3,000 routers in about 17 hours, and why the only fallout came from an undocumented deployment model.

part 10 of 11 · series AutoCon 6
- 1State of automation: start with your alerts
- 2Digital twins at scale
- 3Observability and blast radius
- 4AI evaluation
- 5Automation ROI
- 6Leading radical change
- 7Build vs. buy
- 8Wouldn't it be cool if...
- 9Lightning: the PB&J method
- 10Lightning: 3,000 routers
- 11Lightning: how much code is enough
About 12 years ago at AT&T, Mark Elias (now at Trinity Cyber) and his team had to add a specific subinterface to 3,000 routers. There were eight deployment models and no centralized source of truth. The only record of which interface to touch was the running config on each device. He told the story in a lightning talk at AutoCon 6.
How do you change 3,000 routers with no source of truth?
They already had a toolkit built around a network management cycle: query the device, parse the config, audit it and generate remediation. So they recombined it:
- The backup and audit tools found the target interface on every router.
- The remediation tool, instead of writing fix scripts, wrote a template customized per device.
- The config tool merged each template with the new data (IP addresses, VLAN IDs) into an individual config file.
- A multi-threaded delivery tool pushed 50 changes at a time over an upgraded bandwidth path.
| Measure | Result |
|---|---|
| Routers changed | 3,000 across 8 deployment models |
| Total time | About 17 hours over two days |
| Per device | About 20 seconds |
| Manual budget | Three to four months |
| Fallout | About 0.5%, 10-15 routers, all one undocumented deployment model |
My take: the tools were never the blocker
First, this is the case for small, composable tools. No platform came with an “add a subinterface to 3,000 heterogeneous routers” feature. A set of single-purpose tools that each did one job well could be put together in a new order in an afternoon. The same idea came up in Damien Garros’s build vs. buy keynote: the future is components that compose.
Second, look at where the fallout came from: the undocumented deployment model. The tooling worked. The gaps were in what the organization knew about its own network. Twelve years later the survey from part 1 of this series says the average network is about 30% automated, and the barriers are people and process, not tools. Mark’s team had good enough tools over a decade ago. What they were missing was a source of truth, and today that’s the first thing I’d stand up, starting with NetBox as the inventory.
This is part 10 of my AutoCon 6 series.


