<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/"><channel><title>AutoCon 6 · Network Doodles</title><link>https://networkdoodles.com/series/autocon6/</link><description>What I took away from AutoCon 6 in Tucson, November 2026: the state of network automation, digital twins, observability, AI evaluation, automation ROI, leading change, build vs. buy, and the lightning talks. My take on each, not a recap.</description><language>en-us</language><managingEditor>David Henderson</managingEditor><lastBuildDate>Sat, 21 Nov 2026 00:00:00 +0000</lastBuildDate><atom:link href="https://networkdoodles.com/series/autocon6/index.xml" rel="self" type="application/rss+xml"/><item><title>Your Network Automation ROI Number Is Probably Marketing (AutoCon 6, Part 5)</title><link>https://networkdoodles.com/posts/network-automation-roi/</link><pubDate>Sat, 21 Nov 2026 00:00:00 +0000</pubDate><dc:creator>David Henderson</dc:creator><guid>https://networkdoodles.com/posts/network-automation-roi/</guid><category>network-automation</category><category>automation</category><category>observability</category><category>devops</category><description>Jonathon Davis's six-step method for automation ROI: a human baseline, oversight, real costs, priced failures, and productivity kept apart from savings.</description><content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[<div class="details admonition abstract open">
        <div class="details-summary admonition-title "><i class="icon admonition-icon icon-abstract"></i>Short answer<i class="details-icon  admonition-icon admonition-icon-arrow-right"></i></div>
        <div class="details-content">
            <div class="admonition-content">Vendor time-saved dashboards guess the minutes, count only their own successful runs, treat automation as free
and annualize the best week, so finance discounts them. Measure it yourself: stopwatch a human, subtract the
oversight that stayed, charge build, recurring and per-run costs, price the failures, keep productivity apart
from hard savings, and compute payback from trailing actuals.</div>
        </div>
    </div>
<p>Jonathon Davis opened his <a href="https://networkautomation.forum/autocon6">AutoCon 6</a> talk with a script he wrote
years ago to dial 50 modems and open a ticket for any that didn&rsquo;t answer. Nobody sat down with a list anymore,
and the check went from monthly to weekly. Everyone agreed it was a success. When leadership asked what it was
worth, he had no answer. He could feel the savings, but he couldn&rsquo;t put a number on them.</p>
<p>That&rsquo;s the normal case. His argument: part of the network automation adoption ceiling is an accounting ceiling,
and accounting is something we can fix.</p>
<h2 id="why-dont-finance-teams-trust-automation-time-saved-numbers">Why don&rsquo;t finance teams trust automation time-saved numbers?</h2>
<p>Most automation value is never measured. What does get measured usually comes from the platform that did the
work, and its time-saved dashboard inflates in four ways:</p>
<ul>
<li><strong>Minutes</strong> are an estimate someone typed in, with no human oversight subtracted.</li>
<li><strong>Runs</strong> count only that platform, and only successes.</li>
<li><strong>Cost</strong> is zero. A failed run that burned compute and tokens is invisible, or counted as savings.</li>
<li><strong>Window</strong> is whatever looks biggest. A good week times 52.</li>
</ul>
<p>None of that is fraud, but it&rsquo;s a platform grading its own homework. Finance discounts it on sight, and they&rsquo;re
right to.</p>
<h2 id="how-do-you-measure-what-a-network-automation-returned">How do you measure what a network automation returned?</h2>
<ol>
<li><strong>Baseline from a human.</strong> Stopwatch the manual task. Only runs that replace a real manual run earn labor
value. Extra runs because automation made it cheap get counted but earn nothing.</li>
<li><strong>Subtract oversight.</strong> Babysitting time comes off the top.</li>
<li><strong>Charge real costs:</strong> build, recurring and per-run, including tokens. Engineering time is the biggest cost
and the one most often left out.</li>
<li><strong>Price failures.</strong> Every run pays its cost. Only successful runs earn value.</li>
<li><strong>Separate productivity from hard savings.</strong> Hours returned are productivity. Hard savings means a budget line
went down. Cost avoidance is planned spend that didn&rsquo;t happen. Calling productivity hard savings is the most
common mistake.</li>
<li><strong>Payback from trailing actuals.</strong> Choose the window first (90 days is a good start) and don&rsquo;t move it.</li>
</ol>
<p>His modem example: 190 net minutes per pass, 12 of 52 annual passes with a manual counterpart, about 38 hours a
year returned. At a $75 loaded rate that&rsquo;s about $2,850 of productivity, <em>not</em> hard savings, because no budget
line moved. Build cost about $400, payback about 1.7 months.</p>
<p>His slide for leadership was three lines: about 38 NOC hours a year, paid back in about 1.7 months, risk
reduction not included. A small number he can defend beats a big one he can&rsquo;t.</p>
<p>He open-sourced the method as the <a href="https://github.com/LumaTrack/automation-roi-starter">Automation ROI Starter Kit</a>:
MIT licensed, built on Prometheus, Grafana and one standard-library Python file, with no vendor in the data path.
The guardrails are built in. There&rsquo;s no default for manual minutes, oversight is a required field, runs without a
baseline get a count but no dollars, and productivity and hard savings are never summed. His instructions: fork
it, put your measured numbers in, report every run.</p>
<h2 id="my-take-say-what-it-isnt-and-the-rest-gets-believed">My take: say what it isn&rsquo;t, and the rest gets believed</h2>
<p>This was the most useful talk of the week for anyone who has to justify automation spend. In pre-sales I get
asked for payback on almost every engagement, and too often the only number available is a vendor&rsquo;s time-saved
estimate. The productivity versus hard savings split is the part I&rsquo;m taking home. Saying &ldquo;this returns 38 hours,
it doesn&rsquo;t cut a budget line&rdquo; up front makes every other number in the deck believable.</p>
<p>The other part I&rsquo;d underline: report failures next to successes. Reporting only successes is marketing. It also
explains the survey finding from part 1 of this series, <a href="/posts/automate-network-troubleshooting-from-alerts/">Where Do You Start Automating Network Troubleshooting?</a>:
about half of organizations don&rsquo;t measure automation at all. Nobody wants to publish a number they know won&rsquo;t
survive scrutiny.</p>
<p>Where I&rsquo;d add something is risk reduction. The formula rightly leaves it out of payback, but for read-only
automation like backups, validation and finding dead backup paths before you need them, risk reduction is most
of the value. Jonathon suggested pricing it as the cost of a past outage times the reduction in probability, and
discounting it hard. I&rsquo;d track it in its own field, never summed into payback, with the same discipline as
everything else. The outage-cost side of that is a <a href="/posts/observability-business-impact/">blast radius question</a>,
which is one more reason observability comes before automation.</p>
<h2 id="what-four-questions-should-you-ask-of-any-automation-roi-number">What four questions should you ask of any automation ROI number?</h2>
<ol>
<li>Whose minutes? Measured or guessed? Oversight subtracted?</li>
<li>Which runs? One platform or all? Successes only?</li>
<li>What did it cost? Build, recurring, per-run, failures?</li>
<li>What window? Trailing actuals, or the best week annualized?</li>
</ol>
<p>He ran the questions against a real platform&rsquo;s time-saved dashboard, and it failed all four: a typed-in estimate
per workflow with no oversight field, its own successes but not its failures, no rate or costs with failed runs
missing entirely, and a one-week total that&rsquo;s easy to annualize. Then he ran them against the kit: your measured
minutes with oversight subtracted, every run including failures, build, recurring, per-run and token costs with
failures priced, and only the runs Prometheus actually observed.</p>
<p>Run them over your own numbers before someone in finance does.</p>
<p>This is part 5 of my <a href="/series/autocon6/">AutoCon 6 series</a>.</p>
]]></content:encoded></item><item><title>Your Automation Is the Teacher Making the PB&amp;J (AutoCon 6, Part 9)</title><link>https://networkdoodles.com/posts/peanut-butter-jelly-network-automation/</link><pubDate>Sat, 21 Nov 2026 00:00:00 +0000</pubDate><dc:creator>David Henderson</dc:creator><guid>https://networkdoodles.com/posts/peanut-butter-jelly-network-automation/</guid><category>network-automation</category><category>automation</category><category>python</category><category>testing</category><description>Maximilian Francis's AutoCon 6 lightning talk: automation executes literally, so make the CLI veteran's implicit steps explicit, modular and fail-fast.</description><content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[<div class="details admonition abstract open">
        <div class="details-summary admonition-title "><i class="icon admonition-icon icon-abstract"></i>Short answer<i class="details-icon  admonition-icon admonition-icon-arrow-right"></i></div>
        <div class="details-content">
            <div class="admonition-content">Automation does exactly what you wrote and nothing else, so every step a senior engineer does from memory has to
become explicit code. Research the requirements and the undocumented steps, plan with fail-fast checks and
ordered dependencies, build small reusable functions, and test until you&rsquo;d run it in your sleep.</div>
        </div>
    </div>
<p>Maximilian Francis of Network to Code opened his <a href="https://networkautomation.forum/autocon6">AutoCon 6</a> lightning
talk with the grade-school exercise most of us remember. Write instructions for making a peanut butter and jelly
sandwich, then watch the teacher follow them exactly. Forget to say &ldquo;take the lid off the peanut butter&rdquo; and the
lid stays on. The teacher infers nothing.</p>
<p>Automation works the same way. The interpreter can&rsquo;t infer your intent, however good your docstrings are. In
networking a missing step carries a big blast radius: leave out <code>add</code> on a VLAN trunk command and four VLANs
become one.</p>
<h2 id="how-do-you-turn-a-manual-network-task-into-reliable-automation">How do you turn a manual network task into reliable automation?</h2>
<ul>
<li><strong>Research.</strong> Gather business requirements and audit the documentation. The 25-year CLI veteran on your team
has steps memorized that never made it into Confluence. Turn those implicit assumptions into explicit code.</li>
<li><strong>Plan.</strong> Fail fast. Detect a bad condition and eject before touching production. Let task dependencies set the
order.</li>
<li><strong>Build modular.</strong> Small reusable functions, promoted to shared libraries.</li>
<li><strong>Test</strong> until you&rsquo;d run it in your sleep, and turn unexpected errors into handled exceptions.</li>
</ul>
<p>His real example was an OS upgrade workflow in Nautobot: about 25 chained jobs defined in YAML, kicked off from
one intake job. Each step carries a weight, so reordering is a one-integer change. An <code>is_step_supported</code> check
skips steps a platform doesn&rsquo;t support (Cisco IOS XE has no maintenance mode, Arista EOS does), and dispatch
patterns let one backup job handle both vendors. Platform three means adding an entry, not writing a new job.</p>
<h2 id="my-take-the-pbj-problem-gets-worse-with-ai">My take: the PB&amp;J problem gets worse with AI</h2>
<p>The line that stuck with me: once that veteran&rsquo;s knowledge is in the Git repo, it&rsquo;s no longer a single point of
failure. That&rsquo;s the most underrated benefit of automation. It&rsquo;s knowledge capture, and it outlasts the person.</p>
<p>The PB&amp;J problem also gets worse with AI, not better. A literal machine skips a missing step and fails visibly. An
agent fills the gap with a guess and keeps going. Explicit runbooks, ordered dependencies and fail-fast checks
matter more when the thing executing them can improvise. That&rsquo;s why I keep agents on the proposing side and leave
execution to deterministic tooling that does exactly what it&rsquo;s told. If your runbooks start as alert responses,
<a href="/posts/automate-network-troubleshooting-from-alerts/">writing down what you do when an alert fires</a> is the same
exercise as Maximilian&rsquo;s sandwich instructions.</p>
<p>And if the steps live in YAML, <a href="/posts/understandingyaml/">read it the way the interpreter does</a>.</p>
<p>This is part 9 of my <a href="/series/autocon6/">AutoCon 6 series</a>.</p>
]]></content:encoded></item><item><title>You Can't Tell an Agent Not to Hallucinate. You Can Grade It. (AutoCon 6, Part 4)</title><link>https://networkdoodles.com/posts/ai-evaluation-network-agents/</link><pubDate>Sat, 21 Nov 2026 00:00:00 +0000</pubDate><dc:creator>David Henderson</dc:creator><guid>https://networkdoodles.com/posts/ai-evaluation-network-agents/</guid><category>ai</category><category>agents</category><category>testing</category><category>network-automation</category><category>devops</category><description>Eric Chou's AutoCon 6 talk on evaluating AI: why pilots fail in production, the offline and online eval loop, and what it takes to trust a network agent.</description><content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[<div class="details admonition abstract open">
        <div class="details-summary admonition-title "><i class="icon admonition-icon icon-abstract"></i>Short answer<i class="details-icon  admonition-icon admonition-icon-arrow-right"></i></div>
        <div class="details-content">
            <div class="admonition-content">AI agents fail in production because pilots run on curated data and nobody measures drift. The fix is an
evaluation loop: write down what a right and wrong answer look like, build a labeled set from real examples,
test prompts and models against it before rollout, then grade production traffic with a rubric and feed new edge
cases back in. Start with 10-30 pass/fail examples for one workflow.</div>
        </div>
    </div>
<p>Eric Chou opened his <a href="https://networkautomation.forum/autocon6">AutoCon 6</a> talk with a show of hands. Nearly
everyone in the room uses AI. About five people have an agent doing real work for them. That gap is the whole
talk.</p>
<h2 id="why-dont-ai-pilots-survive-production">Why don&rsquo;t AI pilots survive production?</h2>
<p>Pilots run on curated data and curated questions. Production brings edge cases and users who do unpredictable
things. Eric called shipping without a validation framework &ldquo;hope-driven development,&rdquo; and named two failure
modes: demos that don&rsquo;t reflect real-world mess, and silent drift, where model updates, new tools and
undocumented prompt changes ship with no measured impact. Both end the same way: lost trust, lost budget,
stalled project.</p>
<p>Models are probabilistic. Most operations need to be deterministic. His line: 99% sounds great until your life
depends on it.</p>
<h2 id="how-do-you-evaluate-an-ai-agent-before-and-after-rollout">How do you evaluate an AI agent before and after rollout?</h2>
<p>Start with ground truth. If you can&rsquo;t write down what &ldquo;right&rdquo; looks like, you can&rsquo;t measure it. Public
leaderboards are a starting point. They test public data, not your data, your rules or your judgment.</p>
<table>
  <thead>
      <tr>
          <th>Phase</th>
          <th>What you do</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>Offline, before production</td>
          <td>Define the workflow and its business rules, build a reference dataset (real examples preferred), agree as a team on what good looks like, and iterate on prompts and models until the team accepts the results. Prompts and model choices get checked in like code.</td>
      </tr>
      <tr>
          <td>Online, in production</td>
          <td>Humans don&rsquo;t scale, so an LLM judge scores traffic against a rubric. Log the traces that matter, classify the emerging problem buckets, and feed real edge cases back into the offline set.</td>
      </tr>
  </tbody>
</table>
<p>The case studies made it concrete. DoorDash routes high-severity responses (money, refunds) to a human in real
time and cut hallucinations 90% and severe compliance issues 99%. Alibaba Cloud ran a network troubleshooting
system in production for ten months before publishing, with 20% faster troubleshooting. A clinical note-taking
system kept recording the wrong drug when a patient corrected themselves mid-sentence, which is exactly the kind
of edge case only real traffic finds.</p>
<h2 id="my-take-my-approval-step-is-my-evaluation-and-that-doesnt-scale">My take: my approval step is my evaluation, and that doesn&rsquo;t scale</h2>
<p>This is the missing piece in most agentic ops pitches, including some I&rsquo;ve given myself.</p>
<p>At home I run my own agent harness on Claude&rsquo;s Agent SDK. It has read access to my network monitoring and
telemetry and no write path to the network at all. The only way it changes the network is by submitting a change
through my CI/CD pipeline, which I approve or reject from my phone. That&rsquo;s the same split I wrote about in
<a href="/posts/infragenie-to-odin/">From InfraGenie to ODIN</a>: agents propose, Ansible disposes. On lower-stakes systems
I give it more room. While I was in Tucson it recovered a failed server in my basement, rebuilt it on new
hardware and migrated the apps without me touching anything.</p>
<p>What I don&rsquo;t have is what Eric described. My approval step <em>is</em> my evaluation, and that doesn&rsquo;t scale past one
person. A labeled set of 20 good and bad change proposals would tell me far more than my gut does at 11 p.m.</p>
<p>That&rsquo;s also the answer to the hallway conversation afterward. Most companies don&rsquo;t stall in the operations team.
They stall with upper management and security, who are paid to be risk-averse. If securing the environment were
my job, I&rsquo;d be cautious too. You don&rsquo;t move those people with a demo. You move them with evidence, and an
evaluation set is how you produce it. Autonomy should expand one workflow at a time, based on evidence instead of
intuition. A good first workflow is the read-only one from part 1 of this series:
<a href="/posts/automate-network-troubleshooting-from-alerts/">collecting evidence when an alert fires</a>.</p>
<h2 id="what-can-you-do-on-monday">What can you do on Monday?</h2>
<p>Pick one workflow. Write 10-30 pass/fail examples of what right and wrong look like. Do it before the rollout,
not after the first incident.</p>
<p>This is part 4 of my <a href="/series/autocon6/">AutoCon 6 series</a>.</p>
]]></content:encoded></item><item><title>Wouldn't It Be Cool If...: From CLI to Agentic Delivery (AutoCon 6, Part 8)</title><link>https://networkdoodles.com/posts/network-automation-journey-agentic-delivery/</link><pubDate>Sat, 21 Nov 2026 00:00:00 +0000</pubDate><dc:creator>David Henderson</dc:creator><guid>https://networkdoodles.com/posts/network-automation-journey-agentic-delivery/</guid><category>network-automation</category><category>python</category><category>ai</category><category>agents</category><category>automation</category><description>Sean Mahoney's AutoCon 6 talk: seven years from a Netmiko script to an agent engine whose rule is that AI does its task but never creates or grades it.</description><content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[<div class="details admonition abstract open">
        <div class="details-summary admonition-title "><i class="icon admonition-icon icon-abstract"></i>Short answer<i class="details-icon  admonition-icon admonition-icon-arrow-right"></i></div>
        <div class="details-content">
            <div class="admonition-content">Most automation careers start with one tedious task and the question &ldquo;wouldn&rsquo;t it be cool if there were a better
way?&rdquo; Sean Mahoney&rsquo;s went from a rogue-switch hunt in Netmiko to an agentic delivery engine whose rule is simple:
AI does its task but doesn&rsquo;t create its task, and it writes tests but can&rsquo;t pass them itself. Proposing, approving
and executing stay separate jobs.</div>
        </div>
    </div>
<p>One of my favorite talks at <a href="https://networkautomation.forum/autocon6">AutoCon 6</a> was Sean Mahoney of US Signal
compressing seven years of his career into 30 minutes. Every step started with the same sentence: wouldn&rsquo;t it be
cool if there were a better way to do this?</p>
<h2 id="how-did-one-tedious-task-turn-into-an-automation-career">How did one tedious task turn into an automation career?</h2>
<p>It started with hunting rogue hubs and switches across about 1,000 switch stacks without turning on port
security. The manual way meant SSH into each switch, run <code>show mac address-table</code>, paste into a spreadsheet and do
some Excel magic. Netmiko made it programmatic. Regex, loops and dictionaries made it scale. The CSV module ended
the copy-paste. PySNMP and a scheduler turned it into an hourly job that emailed the team about new rogue devices.
A task that would have eaten hours of several engineers&rsquo; time just ran.</p>
<p>Then a Webex chatbot, built the day he saw the API demo at a 2020 bootcamp (he skipped the rest of the training).
It went from Node.js calling Python, to Flask, to a FastAPI gateway, with TextFSM templates replacing regex along
the way. Then local LLMs summarizing BGP trap triage, then AI-assisted coding, then an engine that chains headless
Claude Code sessions to build software from a chat message.</p>
<h2 id="my-version">My version</h2>
<p>This talk hit home because my own start looked the same. I started automating small networking tasks back in 2012,
but my first big automation project came in 2017, and it was one I decided was too big to do by hand: turning
our Azure environment and our secondary data center into full disaster
recovery failovers for production.</p>
<p>The servers were covered. The company bought Zerto to replicate them. But a server without the network around it
is useless, and none of that was in sync: no DNS, no VIPs, no firewall rules, no VLAN or IPAM plan that matched
between data centers, no routing, no public DNS and no dependency map. It all had to come up in a specific order,
and the primary data center also carried dev and QA traffic that the app dev and SAP teams treated as production.</p>
<p>So I took it one layer at a time, in Python, Netmiko and Ansible. I automated collecting each production system&rsquo;s config, then planned how each
piece would translate to the DR site: firewall rules and NATs, VIPs with their iRules, health checks and pool
members, the GSLB design and sync, the Nexus configs and routing. I met with every app owner to map frontends, app
servers and databases. And I built in testing, because a failover you haven&rsquo;t tested is a guess. In 2017 there
still wasn&rsquo;t much out there to copy for a job this size. I taught myself what I didn&rsquo;t know and pushed the tools I did
know into doing things I hadn&rsquo;t thought of.</p>
<p>In the end we replicated and tested DR failover for more than 200 apps, every moving piece included, in one change
window. Doing the first few by hand, I managed two or three per window, plus hours of collecting, parsing and
converting data from the primary data center. My one regret is that I didn&rsquo;t measure it, so the manual baseline is
memory, not a number. That&rsquo;s the mistake <a href="/posts/network-automation-roi/">part 5 of this series</a> warns about. If I
can find the old code, I&rsquo;ll sanitize it and post it, terrible mistakes included.</p>
<p>The best part came after. The automation became a formal process: new business-critical apps were built with
automation that replicated and converted them to the other data centers at creation, and failover testing happened
at turn-up instead of after the fact.</p>
<p>Once it worked, I kept going. I added monitoring, then I built an IPAM. None of it was planned as a program. Each
piece solved the next annoying thing.</p>
<p>That&rsquo;s how most real automation careers start, and it&rsquo;s why the survey&rsquo;s &ldquo;half of organizations have no dedicated
automation engineer&rdquo; number, from <a href="/posts/automate-network-troubleshooting-from-alerts/">part 1 of this series</a>,
bothers me. Curiosity builds the first tool. Someone has to own the tenth.</p>
<h2 id="what-rules-keep-an-ai-agent-from-grading-its-own-work">What rules keep an AI agent from grading its own work?</h2>
<p>The best list I saw all week was his set of design principles for his current agentic delivery engine:</p>
<ul>
<li>GitHub (PRs and issues) is the source of truth, not handoff documents</li>
<li>One builder, one job, in a sealed, network-isolated container</li>
<li>Tests are written and locked before the build starts, and the builder can&rsquo;t modify them</li>
<li>Grading happens outside the builder&rsquo;s context</li>
<li>A proxy gates every external call and logs it</li>
<li>The builder works on a copy, never live code</li>
<li>Every builder gets a token and time budget</li>
<li>The engine writes its intent before acting, so it can resume after a crash</li>
<li>Shipped apps include a maintenance agent that can open PRs but can&rsquo;t assign itself work</li>
</ul>
<p>His summary: AI does its task, but doesn&rsquo;t create its task. It writes tests, but can&rsquo;t pass them itself.</p>
<p>That&rsquo;s the same separation I argue for with network agents: proposing, approving and executing are different jobs,
and the agent only gets the first one. I also liked his answer on LLM determinism for alarm handling. The LLM
wasn&rsquo;t troubleshooting. Scripts did the triage, and the model only summarized their output. That&rsquo;s the right first
job for an LLM in operations, and it&rsquo;s the pattern I described in
<a href="/posts/automate-network-troubleshooting-from-alerts/">automating troubleshooting from your alerts</a>.</p>
<p>One result worth noting: he wired up a script to generate 15 app ideas a day and got 30 apps in a weekend. About
10% worked. Building is no longer the bottleneck. Verification is, which is the whole argument of
<a href="/posts/ai-evaluation-network-agents/">Eric Chou&rsquo;s talk in part 4</a>.</p>
<h2 id="how-does-a-mid-level-engineer-find-time-to-learn-automation">How does a mid-level engineer find time to learn automation?</h2>
<p>Someone asked exactly that. His answer: volunteer for every tedious project nobody wants, then use the expected
timeline as cover to experiment. I did a version of that myself, and I&rsquo;d give the same advice today.</p>
<p>This is part 8 of my <a href="/series/autocon6/">AutoCon 6 series</a>.</p>
]]></content:encoded></item><item><title>How Much Code Do Network Engineers Need in the AI Era? (AutoCon 6, Part 11)</title><link>https://networkdoodles.com/posts/ai-coding-literacy-network-engineers/</link><pubDate>Sat, 21 Nov 2026 00:00:00 +0000</pubDate><dc:creator>David Henderson</dc:creator><guid>https://networkdoodles.com/posts/ai-coding-literacy-network-engineers/</guid><category>ai</category><category>python</category><category>network-automation</category><category>testing</category><description>Erika Dietrick's AutoCon 6 lightning talk: when to write code yourself, when AI should teach, when it can write, and why you can't review what you can't write.</description><content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[<div class="details admonition abstract open">
        <div class="details-summary admonition-title "><i class="icon admonition-icon icon-abstract"></i>Short answer<i class="details-icon  admonition-icon admonition-icon-arrow-right"></i></div>
        <div class="details-content">
            <div class="admonition-content">Network engineers need enough code to think programmatically about the network. If you can&rsquo;t write working
automation yet, write it yourself. If you can but lack real-world experience, use AI as a teacher. Only experienced
automators should let AI write the code, and only code they already understand, because you can&rsquo;t review what you
couldn&rsquo;t have written.</div>
        </div>
    </div>
<p>Erika Dietrick skipped the &ldquo;is coding dead?&rdquo; debate in her <a href="https://networkautomation.forum/autocon6">AutoCon 6</a>
lightning talk and asked a better question: how much do network engineers need to know? Her answer is enough to
think programmatically about the network. She lays it out as 11 foundational topics taught in network terms, from
developer mindset and environment setup through data structures, troubleshooting, libraries, APIs and version
control.</p>
<h2 id="when-should-a-network-engineer-let-ai-write-the-code">When should a network engineer let AI write the code?</h2>
<table>
  <thead>
      <tr>
          <th>Where you are</th>
          <th>What to do with AI</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>You can&rsquo;t write working automation yet</td>
          <td>Write all of it yourself. No AI.</td>
      </tr>
      <tr>
          <td>You can write it but lack real-world experience</td>
          <td>Use AI as a teacher, with structured prompts.</td>
      </tr>
      <tr>
          <td>You&rsquo;re an experienced automator</td>
          <td>Let AI generate code you already understand.</td>
      </tr>
  </tbody>
</table>
<p>Then she went through the objections one at a time. What about vibe coding? Not reviewing the code isn&rsquo;t safe for
infrastructure. AI agents, agent loops, a spec written by a software engineer? You still have to know what you&rsquo;re
doing. Skills and rules files? Useful, but in her experiment, telling a high-end model to look for specific issues
gave it tunnel vision and it missed others.</p>
<p>The line of the week for me: an AI that passes 100% of the tests it wrote itself is a student grading its own
homework.</p>
<h2 id="my-take-you-cant-review-what-you-couldnt-have-written">My take: you can&rsquo;t review what you couldn&rsquo;t have written</h2>
<p>I agree with all of it. Her test is the one I&rsquo;d use for hiring and staffing: you can&rsquo;t review what you couldn&rsquo;t
have written. If a team member can&rsquo;t read the playbook an agent produced, they can&rsquo;t approve it, and approval is
the only control that matters.</p>
<p>The self-graded-tests point ties to the best engineering pattern I saw at the conference, from
<a href="/posts/network-automation-journey-agentic-delivery/">Sean Mahoney&rsquo;s agentic delivery talk</a>: tests are written and
locked before the build, and grading happens outside the builder&rsquo;s context. Separate the thing that produces from
the thing that judges. <a href="/posts/ai-evaluation-network-agents/">Eric Chou&rsquo;s evaluation talk</a> makes the same argument
for agents in production.</p>
<p>Her closing point is the one I&rsquo;d tape to the wall. We&rsquo;re engineers. We&rsquo;re responsible for the network, not the AI.
That&rsquo;s the whole reason I keep execution deterministic and keep a human on the merge. Agents propose. Ansible
disposes. People own the outcome.</p>
<p>Her iDoIt program (Level 1 is 12 weeks of video, exercises and a mentor conversation; Level 2 is self-paced with a
prompt guide) is free. Worth sending to anyone on your team who&rsquo;s starting out. If they&rsquo;re starting from zero, the
fundamentals in <a href="/posts/understandingyaml/">YAML for network engineers</a> and
<a href="/posts/donthardcodepasswords/">keeping passwords out of playbooks</a> are where I&rsquo;d point them first.</p>
<p>This is part 11 of my <a href="/series/autocon6/">AutoCon 6 series</a>.</p>
]]></content:encoded></item><item><title>Everything Is Working Exactly as Designed. That's the Problem. (AutoCon 6, Part 6)</title><link>https://networkdoodles.com/posts/leading-radical-change-network-automation/</link><pubDate>Sat, 21 Nov 2026 00:00:00 +0000</pubDate><dc:creator>David Henderson</dc:creator><guid>https://networkdoodles.com/posts/leading-radical-change-network-automation/</guid><category>network-automation</category><category>automation</category><category>devops</category><category>architecture</category><description>Anna Claiborne's AutoCon 6 talk as an automation playbook: governance built to say not yet, sacred cows, real vs. habit risk, and metrics that show change.</description><content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[<div class="details admonition abstract open">
        <div class="details-summary admonition-title "><i class="icon admonition-icon icon-abstract"></i>Short answer<i class="details-icon  admonition-icon admonition-icon-arrow-right"></i></div>
        <div class="details-content">
            <div class="admonition-content">Network automation programs rarely stall on technology. They stall in change boards, approval gates and risk
reviews that are doing exactly what they were built to do. Anna Claiborne&rsquo;s answer: ask whether each risk is real
or habit, delete the habits, govern real risk by reversibility with automated controls and one owner, and
measure whether behavior is actually changing.</div>
        </div>
    </div>
<p>The least technical talk I saw at <a href="https://networkautomation.forum/autocon6">AutoCon 6</a> might be the one most
relevant to why network automation stalls. Anna Claiborne of Salesforce has led change as a startup CTO and
inside large enterprises, and she opened with this: enterprises don&rsquo;t fail to change because something is broken.
They fail because everything works exactly as designed.</p>
<h2 id="why-do-enterprises-resist-change-when-nothing-is-broken">Why do enterprises resist change when nothing is broken?</h2>
<p>Governance, process and risk management exist to keep the ship steady, and in good times they do it well.
Approval gates catch bad bets. Standard workflows make outcomes repeatable. Risk management protects what already
works.</p>
<table>
  <thead>
      <tr>
          <th>System</th>
          <th>In good times</th>
          <th>In radical change</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>Governance</td>
          <td>Approval gates and steering forums catch bad bets</td>
          <td>Every new idea waits for a forum designed to say &ldquo;not yet,&rdquo; and decisions move at meeting cadence</td>
      </tr>
      <tr>
          <td>Process</td>
          <td>Standard workflows make outcomes repeatable</td>
          <td>The new way of working doesn&rsquo;t fit the workflow, so it looks like non-compliance</td>
      </tr>
      <tr>
          <td>Risk management</td>
          <td>Limits the downside and protects what works</td>
          <td>Measures the risk of acting, rarely the risk of standing still</td>
      </tr>
  </tbody>
</table>
<p>None of that is broken. It&rsquo;s doing what it was built to do.</p>
<h2 id="is-universal-buy-in-a-realistic-goal">Is universal buy-in a realistic goal?</h2>
<p>Universal buy-in is mostly a comforting story we use to put off hard choices. Some of today&rsquo;s top performers
won&rsquo;t be top performers in the new model, and some of them are in the room.</p>
<p>Her &ldquo;Survivor: Enterprise Edition&rdquo; slide sorted people by how they handle change. The ones who thrive are
builders, fast learners comfortable saying &ldquo;I don&rsquo;t know yet,&rdquo; mission-first people who care more about the
outcome than their title, cross-functional translators, data-driven realists willing to admit the data
contradicts them, and leaders who share credit and take the heat. The ones who struggle are guardians of the old
model, micromanagers, process-first operators who use policy and risk as a default no, chronic consensus-seekers
who reopen every decision, status-dependent leaders, and leaders who can&rsquo;t handle dissent.</p>
<h2 id="what-are-sacred-cows-and-how-do-you-break-them">What are sacred cows, and how do you break them?</h2>
<p>A sacred cow is a product, customer, process or value that is never questioned, is protected by powerful
stakeholders, and has no clearly stated value. The obvious ones are doomed legacy products and unprofitable
&ldquo;strategic&rdquo; customers. The subtle ones are values:</p>
<ul>
<li>Stability becomes &ldquo;my job is the process.&rdquo;</li>
<li>Efficiency refuses a short dip in performance for a long-term gain.</li>
<li>Consensus becomes a steering committee with no decision-maker.</li>
<li>Even modern values get bent: psychological safety read as never being uncomfortable, work-life balance as no
urgent work ever, a flat hierarchy as a way to avoid hard decisions.</li>
</ul>
<p>Her tools for breaking them: a hard sunset with a public feature freeze date for a doomed product, a small funded
skunkworks team that works outside a resistant business unit, and decommission bonuses tied to retiring legacy
systems on a timeline.</p>
<h2 id="how-do-you-move-fast-in-a-risk-averse-enterprise">How do you move fast in a risk-averse enterprise?</h2>
<p>How fast you can move depends on trust. High-trust organizations are collaborative, decentralized, focused on
learning, and treat risk as opportunity. Low-trust ones are territorial, hierarchical, focused on blame, and treat
risk as a problem.</p>
<p>For a risk-averse enterprise she offered four steps:</p>
<ol>
<li>Ask whether the risk is real or habit.</li>
<li>If it&rsquo;s habit, delete the rule and put a cost on standing still.</li>
<li>If it&rsquo;s real, change how it&rsquo;s governed. Decide by reversibility, build risk into the team, automate the
controls, and name one owner who can accept the risk.</li>
<li>If the system won&rsquo;t change, go around it with fast lanes, skunkworks or time-limited exceptions. As a last
resort, spend trust deliberately.</li>
</ol>
<p>Before spending trust, check whether the thing is breakable: is it a critical system, is it in the path of
revenue, and what would happen if it went away tomorrow? Her rule of thumb: the more trust you&rsquo;re about to break,
the more communication and agreement-building you owe first.</p>
<h2 id="how-do-you-measure-whether-an-organization-is-actually-changing">How do you measure whether an organization is actually changing?</h2>
<p>In radical change, the question shifts from &ldquo;how are we performing?&rdquo; to &ldquo;are we actually changing?&rdquo; She measures
in three layers: behavior and adoption, the enablers (tech, process, skills), and business outcomes last.</p>
<p>Her how-to for leading indicators was the most practical slide of the talk:</p>
<ul>
<li>Tag systems and services as legacy or next-gen in your inventory.</li>
<li>Track movement away from the old, not just use of the new.</li>
<li>Pair every adoption metric with a quality or friction counter-metric.</li>
<li>Pick metrics that move weekly, and aim for trend improvement.</li>
<li>For each indicator, write down in advance what happens if it misses.</li>
</ul>
<p>On the process side: lead time from idea to production, percentage of releases automated versus manual, time
from issue raised to decision made, and percentage of experiments killed within 90 days. Lagging indicators like
revenue and incident rates become guardrails, with an acceptable dip agreed up front.</p>
<h2 id="my-take-her-step-3-is-a-network-automation-design-doc">My take: her step 3 is a network automation design doc</h2>
<p>The survey earlier in the week, covered in <a href="/posts/automate-network-troubleshooting-from-alerts/">part 1 of this series</a>,
said people problems outnumber technical problems four to one as the barrier to automation. This talk explained
why.</p>
<p>Her step 3 reads like a network automation design doc: decide by reversibility, automate the controls, name one
owner who accepts the risk. Read-only automation (backups, state collection, validation) is reversible. It
shouldn&rsquo;t wait in the same change advisory board queue as a core routing change, but in a lot of organizations it
does, and the automation program dies there. And &ldquo;automate the controls&rdquo; is what a good pipeline is. Pre-checks,
validation and rollback built into the workflow can replace a meeting as the control.</p>
<p>Her metrics translate almost word for word. &ldquo;Releases automated versus manual&rdquo; becomes the percentage of network
changes that go through the pipeline instead of the CLI. &ldquo;Track movement away from the old&rdquo; means watching manual
CLI changes go down, not just playbook runs go up. The counter-metric is change success rate or rollback rate, so
speed doesn&rsquo;t hide breakage. And tagging legacy versus next-gen in your inventory is exactly what your source of
truth is for. If NetBox is already your inventory, <a href="/posts/netboxasinventory/">it&rsquo;s already where those tags belong</a>.</p>
<p>Identity threat is the cultural barrier I see most. Engineers who spent 20 years getting fast at the CLI hear
&ldquo;automation&rdquo; as &ldquo;you&rsquo;re less valuable.&rdquo; The network version of &ldquo;my job is the process&rdquo; is the engineer whose
value is knowing the MOP by heart. The answer isn&rsquo;t a pep talk. It&rsquo;s being specific, early, about what changes in
someone&rsquo;s job and what doesn&rsquo;t, and making the people who capture their knowledge in code the visible winners.</p>
<p>The advice for leading change without authority matched what works in my experience: build peer pressure across
teams, run a proof of concept that produces data, and go over a resistant manager&rsquo;s head only as a last resort.
Combine that with <a href="/posts/network-automation-roi/">the ROI method from part 5</a> and you have a pitch: a small,
measured win with honest numbers. That moves budgets.</p>
<p>One warning sign is worth repeating to any automation lead: if nobody pushes back and everyone is waiting for
something to break, the change isn&rsquo;t working. Make dissent safe. Run pre-mortems. Rotate who plays devil&rsquo;s
advocate.</p>
<h2 id="final-jeopardy">Final Jeopardy</h2>
<p>Anna closed with four questions on a Final Jeopardy slide:</p>
<ul>
<li>Which sacred cow do you need to question?</li>
<li>Where are you choosing comfort over necessary change?</li>
<li>Would you rather be loved (the nice leader) or remembered as the one who saved the company?</li>
<li>If this company fails in 5 years, what will you wish you&rsquo;d done differently right now?</li>
</ul>
<p>The second one followed me into the next morning&rsquo;s keynote on build vs. buy, where I wrote &ldquo;Am I just playing it
safe?&rdquo; in my notes. That became <a href="/posts/build-vs-buy-network-automation-ai/">part 7 of this series</a>. Comfort and
judgment look identical from the inside. The only way I know to tell them apart is to keep running experiments
past the point I&rsquo;d recommend to anyone else.</p>
<p>For network teams, the sacred cow is usually the process that has always worked: the hand-built MOP, the change
window everyone works around, the CLI veteran nobody questions. Those aren&rsquo;t bad. They were built to protect the
network, which is exactly why they resist the change automation needs.</p>
<p>The last question is the one I&rsquo;d put in front of an automation program. If your network operations look the same
in five years, with the same manual changes and the same 2 a.m. troubleshooting, what will you wish you&rsquo;d started
this quarter? Write the answer down, then pick the smallest reversible version of it and do that first.</p>
<p>This is part 6 of my <a href="/series/autocon6/">AutoCon 6 series</a>.</p>
]]></content:encoded></item><item><title>Emulate the Question, Not the Network: Digital Twins at Scale (AutoCon 6, Part 2)</title><link>https://networkdoodles.com/posts/digital-twin-scale-containerlab/</link><pubDate>Sat, 21 Nov 2026 00:00:00 +0000</pubDate><dc:creator>David Henderson</dc:creator><guid>https://networkdoodles.com/posts/digital-twin-scale-containerlab/</guid><category>digital-twin</category><category>containerlab</category><category>testing</category><category>network-automation</category><category>ai</category><description>Russ White's AutoCon 6 tutorial on big ContainerLab twins: one twin per question, staged IS-IS bring-up, real scale limits, and where LLMs belong.</description><content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[<div class="details admonition abstract open">
        <div class="details-summary admonition-title "><i class="icon admonition-icon icon-abstract"></i>Short answer<i class="details-icon  admonition-icon admonition-icon-arrow-right"></i></div>
        <div class="details-content">
            <div class="admonition-content">Don&rsquo;t try to build a digital twin of your whole network. Build a twin of the question you&rsquo;re asking: break the
network into functional pieces, emulate only the piece the question touches, and strip the containerized NOS
down to what the test needs. Script the topology yourself and let LLMs write helper scripts, because a model
asked to generate a 200-router topology gives you a different one every time.</div>
        </div>
    </div>
<p>I&rsquo;ve built network digital twins a few different ways: graph models for path tracing, emulated labs for change
testing. The <a href="https://networkautomation.forum/autocon6">AutoCon 6</a> talk closest to my own experience was Russ
White&rsquo;s tutorial on bringing up very large ContainerLab topologies, and it opened with the right principle. Don&rsquo;t
build a twin of the network. Build a twin of the question you&rsquo;re trying to answer.</p>
<h2 id="why-build-one-twin-per-question-instead-of-one-big-twin">Why build one twin per question instead of one big twin?</h2>
<p>A single all-encompassing twin isn&rsquo;t realistic today. CPU is the hard limit. Russ described twinning as a
continuum from two routers to thousands, and the job is finding the point on it that answers what you&rsquo;re asking.
Break the network into functional and topological pieces and run one twin per piece.</p>
<p>If you&rsquo;re testing control plane behavior, use the containerized version of your NOS and strip it down to the
routing stack. If you&rsquo;re testing data plane, keep the full stack but run fewer nodes. You don&rsquo;t need all 1,000
paths through a fabric to prove a forwarding behavior.</p>
<p>His three highest-value use cases:</p>
<ol>
<li>Validating orchestration and management tooling</li>
<li>Version control and model diffing</li>
<li>End-to-end policy verification across nodes</li>
</ol>
<h2 id="how-do-you-bring-up-a-large-containerlab-topology-without-crashing-the-host">How do you bring up a large ContainerLab topology without crashing the host?</h2>
<ul>
<li><strong>Split config by stage.</strong> Link addressing goes in the boot config. Routing and overlay get applied after boot.
Keep IS-IS and BGP out of base templates.</li>
<li><strong>One template per router type.</strong> Use placeholder characters that never appear in real config, and let a
script stamp out per-pod directories. Three router types means three templates.</li>
<li><strong>Stagger protocol bring-up.</strong> He set a unique IS-IS password per device at boot to block adjacencies, then
stripped them interface by interface to control the convergence rate.</li>
<li><strong>Set the Docker restart policy to none.</strong> Otherwise a host reboot restarts 500 containers at once and takes
the box down with them.</li>
<li><strong>Prefer IS-IS for lab scale.</strong> It&rsquo;s interface-based and easier to script than OSPF or BGP.</li>
</ul>
<h2 id="how-far-does-a-containerlab-topology-scale">How far does a ContainerLab topology scale?</h2>
<table>
  <thead>
      <tr>
          <th>Topology</th>
          <th>Result on an 8 vCPU / 32 GB VM</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>2 pods, 128 routers</td>
          <td>Comfortable on 4 vCPUs, full bring-up with IS-IS in 10-15 minutes</td>
      </tr>
      <tr>
          <td>3 pods, 160 routers</td>
          <td>About 6 vCPUs</td>
      </tr>
      <tr>
          <td>4 pods, 192 routers</td>
          <td>Every vCPU consumed</td>
      </tr>
      <tr>
          <td>224 routers</td>
          <td>Failed to converge 20-30% of the time, even with timer tuning</td>
      </tr>
  </tbody>
</table>
<p>Physical cores outperformed cloud vCPUs by a wide margin, and the Linux kernel&rsquo;s bridge limit becomes a hard
ceiling once you split a topology across ContainerLab instances. One bright spot: an FRRouting pull request for
optimized IS-IS flooding cuts CPU about 40%, which could roughly double those numbers.</p>
<h2 id="my-take-where-do-llms-fit-in-building-a-twin">My take: where do LLMs fit in building a twin?</h2>
<p>The part I agree with most: Russ wrote the scripts by hand first, then used LLMs to build helper scripts, not the
topology itself. Ask a model to generate a 200-router topology and you&rsquo;ll get a different one every time. A twin
that isn&rsquo;t repeatable isn&rsquo;t a twin. It&rsquo;s the same principle I apply to agents in production: let the model help
build deterministic tooling, then run the tooling.</p>
<p>Most enterprises don&rsquo;t need 200 routers in ContainerLab. They need the 15 devices in the path of the change
they&rsquo;re about to make, generated from their source of truth on demand. That only works if the twin knows what
the change can break, which is the <a href="/posts/observability-business-impact/">blast radius argument from my observability post</a>.
And if you want a lab NOS with a familiar CLI for the small version of this, that&rsquo;s why I built
<a href="/posts/capeos-lab-network-os/">CapeOS</a>.</p>
<p>The scale work matters because it shows you where the ceiling is. Deciding which question the twin answers is
what makes it useful.</p>
<p>This is part 2 of my <a href="/series/autocon6/">AutoCon 6 series</a>.</p>
]]></content:encoded></item><item><title>Am I Just Playing It Safe? Build vs. Buy When AI Can Build Anything (AutoCon 6, Part 7)</title><link>https://networkdoodles.com/posts/build-vs-buy-network-automation-ai/</link><pubDate>Sat, 21 Nov 2026 00:00:00 +0000</pubDate><dc:creator>David Henderson</dc:creator><guid>https://networkdoodles.com/posts/build-vs-buy-network-automation-ai/</guid><category>ai</category><category>agents</category><category>network-automation</category><category>architecture</category><category>devops</category><description>Damien Garros's AutoCon 6 keynote, the Dinesh Dutt and John Capobianco debate, and where I land on building, buying and owning network automation.</description><content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[<div class="details admonition abstract open">
        <div class="details-summary admonition-title "><i class="icon admonition-icon icon-abstract"></i>Short answer<i class="details-icon  admonition-icon admonition-icon-arrow-right"></i></div>
        <div class="details-content">
            <div class="admonition-content">AI lowers the cost of building software but not the cost of owning it, so the build-vs-buy question is now &ldquo;do we
want to own this?&rdquo; My answer for production networks: buy the platforms, build the glue, and keep agents behind
deterministic execution and human approval. The test for building anything else: two years from now, after the
person who built it leaves, can the team still own it?</div>
        </div>
    </div>
<p>I wrote three lines in my notes during Damien Garros&rsquo;s closing keynote at
<a href="https://networkautomation.forum/autocon6">AutoCon 6</a>: <em>Am I just playing it safe? Hanging in the middle. Do we
think we are the exception?</em> This post is me trying to answer them.</p>
<h2 id="what-should-you-build-when-ai-lets-you-build-anything">What should you build when AI lets you build anything?</h2>
<p>Damien, CEO and co-founder of OpsMill, reframed build vs. buy. The question isn&rsquo;t &ldquo;can we build it?&rdquo; anymore,
because with AI the answer is usually yes. The question is &ldquo;do we want to own it?&rdquo; AI lowers the cost of
building. It doesn&rsquo;t lower the cost of owning: funding it, understanding its decisions and maintaining it after
the person who built it moves on. Vendors are adopting AI just as fast, with more people and more domain depth, so
the builder&rsquo;s edge is smaller than it feels. And there&rsquo;s always a line. Nobody builds their own NAS or their own
routing protocol.</p>
<p>His four predictions for software: quality goes up (bugs fixed in hours), value goes up (more for the same
price), volume goes up (software for problems nobody solved before), and shape changes, with every component
becoming a platform that agents extend and compose.</p>
<p>On working with AI as a coworker: delegate implementation, not trade-offs, goals or architecture. Review
everything, because AI output often looks finished and has no door in the house. Match oversight to consequences.
And don&rsquo;t give agents direct access to network devices; route changes through backend platforms and version
control.</p>
<h2 id="dinesh-dutt-vs-john-capobianco-how-fast-should-we-move">Dinesh Dutt vs. John Capobianco: how fast should we move?</h2>
<p>The Q&amp;A turned into a debate between two of the most respected people in this field.</p>
<p>Dinesh Dutt took the skeptic&rsquo;s side. Dinesh created SuzieQ, was a Cisco Fellow and chief scientist at Cumulus,
co-authored the VXLAN and TRILL standards, and wrote <em>Cloud Native Data Center Networking</em>. His argument was that
even OpenAI and the other labs couldn&rsquo;t define clean boundaries for their agents, so claiming we can is wishful
thinking. He reminded the room that networking has chased myths before, from standard interfaces to SD-WAN, and
that the volume of AI-generated PRs, the outages and the immature tooling all argue for caution.</p>
<p>John Capobianco countered. John is Head of AI and Developer Relations at Itential, built the open-source NetClaw
network agent, and has probably done more than anyone to bring MCP to network engineers. His answer was results:
20 video games, 90,000 lines of code and 1,600 tests rebuilt in two hours. The work is real, and it&rsquo;s happening
now.</p>
<p>Damien landed in the middle. Move as fast as you can sustain while still owning the outcome. If you can&rsquo;t own the
result, you&rsquo;re going too fast.</p>
<h2 id="am-i-playing-it-safe">Am I playing it safe?</h2>
<p>My own position sits in the same middle. Buy the platforms, build the glue, and keep agents behind deterministic
execution and human approval. Agents propose, Ansible disposes. I&rsquo;ve written about building the glue myself, like
<a href="/posts/buildyourownmcp/">an MCP server for network automation</a>, and none of it replaces the platforms underneath.</p>
<p>So the fair question is whether the middle is judgment or comfort. Honestly, some of both. The middle is easy to
defend in a customer meeting. Nobody gets fired for recommending human-in-the-loop.</p>
<p>Watching Dinesh and John go at it helped. Both are right, and that&rsquo;s the point. Dinesh has spent a career watching
the industry overpromise, and his caution comes from designing the protocols and data center fabrics the rest of
us run on. John has spent the last two years shipping agents and MCP servers that actually work, and his
impatience comes from seeing what&rsquo;s possible. If I only listened to Dinesh, I&rsquo;d never move. If I only listened to
John, I&rsquo;d move faster than my customers can own. The middle isn&rsquo;t a compromise between them. It&rsquo;s taking both
seriously at the same time.</p>
<p>I think it&rsquo;s still the right answer for production networks, for a reason the talk named: blast radius. A game
prototype and a hospital network don&rsquo;t deserve the same guardrails. But the middle is only judgment if I keep
testing the edges. At home I run an agent with far more autonomy than I&rsquo;d ever recommend to a customer, and that&rsquo;s
where I find out where the line actually is. If I stopped doing that, the middle would become a habit. The
question I took from <a href="/posts/leading-radical-change-network-automation/">Anna Claiborne&rsquo;s talk the day before</a>
applies here too: where am I choosing comfort over necessary change?</p>
<h2 id="do-we-think-were-the-exception">Do we think we&rsquo;re the exception?</h2>
<p>Every team building its own source of truth, its own agent framework or its own observability stack believes it&rsquo;s
the exception. Most aren&rsquo;t. The test I&rsquo;m going to use: two years from now, after the person who built it leaves,
can the team still own it? If yes, and it&rsquo;s where you differentiate, build. If not, buy, and put your build energy
into the integration that makes the bought thing yours.</p>
<p>Curiosity came up as the core skill for keeping up, and Damien argued that new grads who think AI-native can
outpace experienced engineers. Maybe. But they&rsquo;re also missing the institutional knowledge of where the bodies are
buried. The best teams I see pair those two on purpose.</p>
<p>This is part 7 of my <a href="/series/autocon6/">AutoCon 6 series</a>.</p>
]]></content:encoded></item><item><title>3,000 Routers in 17 Hours With Tools They Already Had (AutoCon 6, Part 10)</title><link>https://networkdoodles.com/posts/network-automation-microservices-3000-routers/</link><pubDate>Sat, 21 Nov 2026 00:00:00 +0000</pubDate><dc:creator>David Henderson</dc:creator><guid>https://networkdoodles.com/posts/network-automation-microservices-3000-routers/</guid><category>network-automation</category><category>automation</category><category>python</category><category>netbox</category><description>Mark Elias's AutoCon 6 lightning talk: recombining audit, template and push tools to change 3,000 routers, and why the only fallout was the undocumented part.</description><content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[<div class="details admonition abstract open">
        <div class="details-summary admonition-title "><i class="icon admonition-icon icon-abstract"></i>Short answer<i class="details-icon  admonition-icon admonition-icon-arrow-right"></i></div>
        <div class="details-content">
            <div class="admonition-content">Mark Elias&rsquo;s team at AT&amp;T changed 3,000 routers across eight deployment models in about 17 hours, roughly 20
seconds per device, against a manual budget of three to four months. They didn&rsquo;t buy a platform. They recombined
single-purpose backup, audit, config and push tools. The only fallout, about 0.5%, came from a deployment model
nobody had documented.</div>
        </div>
    </div>
<p>About 12 years ago at AT&amp;T, Mark Elias (now at Trinity Cyber) and his team had to add a specific subinterface to
3,000 routers. There were eight deployment models and no centralized source of truth. The only record of which
interface to touch was the running config on each device. He told the story in a lightning talk at
<a href="https://networkautomation.forum/autocon6">AutoCon 6</a>.</p>
<h2 id="how-do-you-change-3000-routers-with-no-source-of-truth">How do you change 3,000 routers with no source of truth?</h2>
<p>They already had a toolkit built around a network management cycle: query the device, parse the config, audit it
and generate remediation. So they recombined it:</p>
<ol>
<li>The backup and audit tools found the target interface on every router.</li>
<li>The remediation tool, instead of writing fix scripts, wrote a template customized per device.</li>
<li>The config tool merged each template with the new data (IP addresses, VLAN IDs) into an individual config file.</li>
<li>A multi-threaded delivery tool pushed 50 changes at a time over an upgraded bandwidth path.</li>
</ol>
<table>
  <thead>
      <tr>
          <th>Measure</th>
          <th>Result</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>Routers changed</td>
          <td>3,000 across 8 deployment models</td>
      </tr>
      <tr>
          <td>Total time</td>
          <td>About 17 hours over two days</td>
      </tr>
      <tr>
          <td>Per device</td>
          <td>About 20 seconds</td>
      </tr>
      <tr>
          <td>Manual budget</td>
          <td>Three to four months</td>
      </tr>
      <tr>
          <td>Fallout</td>
          <td>About 0.5%, 10-15 routers, all one undocumented deployment model</td>
      </tr>
  </tbody>
</table>
<h2 id="my-take-the-tools-were-never-the-blocker">My take: the tools were never the blocker</h2>
<p>First, this is the case for small, composable tools. No platform came with an &ldquo;add a subinterface to 3,000
heterogeneous routers&rdquo; feature. A set of single-purpose tools that each did one job well could be put together in
a new order in an afternoon. The same idea came up in
<a href="/posts/build-vs-buy-network-automation-ai/">Damien Garros&rsquo;s build vs. buy keynote</a>: the future is components that
compose.</p>
<p>Second, look at where the fallout came from: the undocumented deployment model. The tooling worked. The gaps were
in what the organization knew about its own network. Twelve years later the survey from
<a href="/posts/automate-network-troubleshooting-from-alerts/">part 1 of this series</a> says the average network is about
30% automated, and the barriers are people and process, not tools. Mark&rsquo;s team had good enough tools over a decade
ago. What they were missing was a source of truth, and today that&rsquo;s the first thing I&rsquo;d stand up, starting with
<a href="/posts/netboxasinventory/">NetBox as the inventory</a>.</p>
<p>This is part 10 of my <a href="/series/autocon6/">AutoCon 6 series</a>.</p>
]]></content:encoded></item><item><title>Where Do You Start Automating Network Troubleshooting? Your Alerts</title><link>https://networkdoodles.com/posts/automate-network-troubleshooting-from-alerts/</link><pubDate>Fri, 20 Nov 2026 00:00:00 +0000</pubDate><dc:creator>David Henderson</dc:creator><guid>https://networkdoodles.com/posts/automate-network-troubleshooting-from-alerts/</guid><category>network-automation</category><category>automation</category><category>troubleshooting</category><category>netbox</category><category>ai</category><description>Start troubleshooting automation with your monitoring: ask what you do when each alert fires, then automate the read-only first ten minutes of investigation.</description><content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[<div class="details admonition abstract open">
        <div class="details-summary admonition-title "><i class="icon admonition-icon icon-abstract"></i>Short answer<i class="details-icon  admonition-icon admonition-icon-arrow-right"></i></div>
        <div class="details-content">
            <div class="admonition-content">Start troubleshooting automation with your monitoring, not with every possible failure. Each alert is a failure
someone already decided was worth a page. For each one, ask &ldquo;when this fires, what&rsquo;s the first thing you do?&rdquo;,
write the answer down, and automate the read-only data collection. The engineer opens a ticket that already has
the evidence instead of a login prompt.</div>
        </div>
    </div>
<h2 id="why-dont-more-teams-automate-troubleshooting">Why don&rsquo;t more teams automate troubleshooting?</h2>
<p>I was in the room at <a href="https://networkautomation.forum/autocon6">AutoCon 6</a> in Tucson in November 2026 when Chris
Grundemann of the Network Automation Forum presented the 2026 State of Network Automation Survey results. Troubleshooting is the biggest
gap in that survey, and it has been the biggest gap in all four years it has run. Among the 344 respondents
doing some automation, 66.3% say troubleshooting should be automated and 13.4% have it fully automated.</p>
<p><img src="/posts/automate-network-troubleshooting-from-alerts/images/troubleshooting-gap-2026_hu_5442549934a67847.webp" srcset="/posts/automate-network-troubleshooting-from-alerts/images/troubleshooting-gap-2026_hu_42db6d9532c6f639.webp 800w, /posts/automate-network-troubleshooting-from-alerts/images/troubleshooting-gap-2026_hu_5442549934a67847.webp 1600w" sizes="(max-width: 860px) 100vw, 780px" width="1600" height="900" alt="Dot chart of the six largest gaps in the 2026 survey between tasks respondents say should be automated and tasks they have fully automated. Troubleshooting leads at 66.3% versus 13.4%, a 52.9-point gap, followed by capacity planning, firewall rules, traffic engineering, service provisioning and firmware upgrades" loading="lazy" decoding="async"></p>
<p>That gap has grown every year, from 38.9 points in 2023 to 52.9 points in 2026.</p>
<p><img src="/posts/automate-network-troubleshooting-from-alerts/images/troubleshooting-gap-trend_hu_18c5d64c5d6230e5.webp" srcset="/posts/automate-network-troubleshooting-from-alerts/images/troubleshooting-gap-trend_hu_80e074ca5d6bd235.webp 800w, /posts/automate-network-troubleshooting-from-alerts/images/troubleshooting-gap-trend_hu_18c5d64c5d6230e5.webp 1600w" sizes="(max-width: 860px) 100vw, 780px" width="1600" height="800" alt="Bar chart of the troubleshooting gap by survey year: 38.9 points in 2023, 43.9 in 2024, 52.3 in 2025 and 52.9 in 2026" loading="lazy" decoding="async"></p>
<table>
  <thead>
      <tr>
          <th>Task (2026)</th>
          <th>Should be automated</th>
          <th>Fully automated</th>
          <th>Gap (points)</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>Troubleshooting</td>
          <td>66.3%</td>
          <td>13.4%</td>
          <td>+52.9</td>
      </tr>
      <tr>
          <td>Capacity planning</td>
          <td>55.8%</td>
          <td>13.1%</td>
          <td>+42.7</td>
      </tr>
      <tr>
          <td>Firewall rules</td>
          <td>55.8%</td>
          <td>16.9%</td>
          <td>+39.0</td>
      </tr>
      <tr>
          <td>Traffic engineering</td>
          <td>42.2%</td>
          <td>7.6%</td>
          <td>+34.6</td>
      </tr>
      <tr>
          <td>Service provisioning</td>
          <td>59.0%</td>
          <td>26.5%</td>
          <td>+32.6</td>
      </tr>
      <tr>
          <td>Firmware and software upgrades</td>
          <td>59.6%</td>
          <td>27.0%</td>
          <td>+32.6</td>
      </tr>
  </tbody>
</table>
<p>Survey data from Chris&rsquo;s slides: 344 respondents doing some automation in 2026, gaps computed from unrounded
data. The 2023 and 2024 samples (77 and 105) are small, so read the early years as direction.</p>
<p>Chris&rsquo;s explanation of that gap is what stuck with me. He didn&rsquo;t think people were ignoring troubleshooting.
&ldquo;I think the troubleshooting is hard,&rdquo; he said. Fully automating it means understanding a lot of unknown
unknowns: &ldquo;you&rsquo;re not going to know what breaks ahead of time.&rdquo; And then: &ldquo;a lot of people are automating
troubleshooting, but it&rsquo;s not getting fully automated. Maybe they won&rsquo;t ever.&rdquo; You can&rsquo;t write a playbook for a
problem you haven&rsquo;t seen yet, so people don&rsquo;t start.</p>
<p>I don&rsquo;t think full automation is the goal anyway. The win is the part in between.</p>
<p>The unknowns are a problem if you think of troubleshooting as &ldquo;every possible failure in the network.&rdquo; Nobody can automate
that. But it&rsquo;s the wrong starting point, because most teams already have a list of the failures they care
about. It&rsquo;s sitting in the monitoring system.</p>
<h2 id="if-you-monitor-it-does-it-matter">If you monitor it, does it matter?</h2>
<p>Yes. Every alert in your NMS exists because someone decided that condition was worth waking a person up for. A
BGP neighbor dropping. An interface taking CRC errors. A WAN circuit crossing 90% utilization. A power supply
failing. Someone thought about each of these, set a threshold and wired it to a pager or a ticket queue.</p>
<p>That&rsquo;s a requirements document for troubleshooting automation. It&rsquo;s scoped, it&rsquo;s prioritized, and it already
has a trigger attached.</p>
<p>So the question stops being &ldquo;how do I automate troubleshooting?&rdquo; and becomes much smaller:</p>
<p><strong>When this alert fires, what&rsquo;s the first thing you do?</strong></p>
<p>Ask that of your senior engineer for one alert. They&rsquo;ll answer in about thirty seconds. &ldquo;I log into the box,
check the neighbor state, look at the interface the session rides on, check the logs for flaps, and see if the
peer is reachable.&rdquo; That&rsquo;s a runbook. Write it down, then automate it.</p>
<h2 id="which-alerts-should-you-automate-first">Which alerts should you automate first?</h2>
<p>Don&rsquo;t start with your scariest alert. Start with the one that costs the most human time.</p>
<p>Export 90 days of alert history and rank each alert type by:</p>
<table>
  <thead>
      <tr>
          <th>Factor</th>
          <th>Why it matters</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>How often it fires</td>
          <td>The interface-errors alert that fires 40 times a week beats the core-switch-down alert that fired once last year</td>
      </tr>
      <tr>
          <td>Time to the first useful fact</td>
          <td>If every instance means SSH to three devices and eight commands before anyone knows anything, that&rsquo;s your target</td>
      </tr>
      <tr>
          <td>How often the answer is the same</td>
          <td>If 80% of the time it&rsquo;s the same root cause, the automation pays off fast</td>
      </tr>
  </tbody>
</table>
<p>Frequency times triage time gives you a rough ranking. Take the top three and ignore the rest for now.</p>
<h2 id="what-does-the-automation-actually-do">What does the automation actually do?</h2>
<p>The goal isn&rsquo;t to automate the fix. It&rsquo;s to automate the first ten minutes of investigation, the part where an
engineer gathers context before they can think.</p>
<p>To find out what those ten minutes look like:</p>
<ul>
<li>Sit with the engineer who handles that alert best and watch them work it. Write down every command.</li>
<li>Read the notes on the last 20 tickets for that alert. The same commands and checks show up again and again.</li>
<li>Look at shell history or session logs if you keep them.</li>
</ul>
<p>You&rsquo;ll end up with something like this for a BGP neighbor down alert:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">alert</span><span class="p">:</span><span class="w"> </span><span class="l">bgp_neighbor_down</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">collect</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="l">show bgp neighbor {peer_ip}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="l">show interface {local_interface}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="l">show logging | include {peer_ip}|{local_interface}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="l">ping {peer_ip} source {local_ip} repeat 5</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">context</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">source_of_truth</span><span class="p">:</span><span class="w"> </span><span class="l">circuit, provider, site, criticality</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">recent_changes</span><span class="p">:</span><span class="w"> </span><span class="l">last 24h on this device and its peer</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">checks</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">interface_down</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;physical problem, check circuit with {provider}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">interface_up_ping_fails</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;L2/L3 path issue between peers&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">ping_ok_session_down</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;BGP config, auth, or peer-side policy&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span>- <span class="nt">flapping</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;count transitions in last hour, check for errors on interface&#34;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">output</span><span class="p">:</span><span class="w"> </span><span class="l">post findings to ticket, tag on-call</span><span class="w">
</span></span></span></code></pre></div><p>That&rsquo;s not clever, and it doesn&rsquo;t have to be. It turns a page that says &ldquo;BGP DOWN&rdquo; with a peer address into a
ticket that says &ldquo;Gi0/0/1 is down, the circuit is Provider X at the Denver site, no config changes in 24 hours,
here are the last 15 log lines.&rdquo; The engineer starts at the answer.</p>
<h2 id="why-does-running-at-alert-time-matter-so-much">Why does running at alert time matter so much?</h2>
<p>Plenty of network problems are transient. A link flaps, a CPU spikes, a buffer fills, and by the time a human
acknowledges the page and logs in, the counters have rolled and the state is gone. Automation that runs the
moment the alert fires captures the network as it was during the problem, not five or fifteen minutes later.
For intermittent issues that alone can be the difference between a root cause and &ldquo;couldn&rsquo;t reproduce,
closing.&rdquo;</p>
<h2 id="is-it-safe-to-let-automation-touch-production">Is it safe to let automation touch production?</h2>
<p>Collecting data and posting it to a ticket is about as safe as automation gets. <code>show</code> commands can&rsquo;t take
anything down. That matters for buy-in: the ops team that would never let a script bounce an interface will
happily let one run read-only commands and save them time.</p>
<p>The survey backs this up. Of respondents who are 10-20% automated, 55.3% describe their automation as read-only:
backups, documentation, monitoring. From the stage, Chris called read-only a clear place to start: &ldquo;You
build your skills, you build team skills, you build trust with management.&rdquo; His slides add that troubleshooting can start read-only too. And the gap is about full automation: 47.4% already
have troubleshooting at least partially automated, so many teams have pieces of this already.</p>
<p>Once that&rsquo;s trusted, there&rsquo;s a natural progression:</p>
<ol>
<li><strong>Enrich.</strong> Add context from the source of truth and attach command output.</li>
<li><strong>Diagnose.</strong> Run the decision tree and suggest a likely cause.</li>
<li><strong>Recommend.</strong> Propose the fix and let a person approve it.</li>
<li><strong>Remediate.</strong> Auto-fix the low-risk, well-understood cases, such as an err-disabled port with a known cause.</li>
</ol>
<p>Most teams will get plenty of value staying at step one or two for a long time. I went further into the
&ldquo;agents propose, a person approves&rdquo; end of this in <a href="/posts/infragenie-to-odin/">From InfraGenie to ODIN</a>, and
the hands-on version, a webhook receiver that gathers evidence and proposes a change, is
<span class="link-soon">lesson 8 of the observability course</span>.</p>
<h2 id="what-if-nobody-knows-what-to-do-when-an-alert-fires">What if nobody knows what to do when an alert fires?</h2>
<p>When you ask &ldquo;what do you do when this fires?&rdquo;, sometimes the honest answer is &ldquo;nothing, we ignore it&rdquo; or &ldquo;I
don&rsquo;t know.&rdquo; That&rsquo;s not an automation problem. That alert is noise, and noise trains people to ignore the
pager. Either define what the response should be or delete the alert. This exercise cleans up your monitoring
as a side effect, and it&rsquo;s worth doing for that reason alone.</p>
<p>It works in the other direction too. If an engineer&rsquo;s first step for an alert is always checking something your
NMS doesn&rsquo;t watch, like optic light levels or a specific log message, that check should probably be its own
alert. Every postmortem that ends with &ldquo;we didn&rsquo;t know until a user called&rdquo; is a new entry for the monitoring
list, which makes it a new entry for the automation list.</p>
<h2 id="where-does-the-source-of-truth-fit">Where does the source of truth fit?</h2>
<p>Raw command output only goes so far. What makes an enriched alert useful is context: which site, which circuit
ID, which provider and their support number, how critical the device is, what&rsquo;s downstream. If you have NetBox
or Nautobot populated, the automation can pull that in, the same way it already does for
<a href="/posts/netboxasinventory/">NetBox as your Ansible inventory</a>. It&rsquo;s the difference between &ldquo;interface down&rdquo;
and &ldquo;the only uplink for the Denver clinic is down, call Provider X with this circuit ID.&rdquo;</p>
<p>The same inventory lets you check upstream before blaming the device. One failure often throws many alerts: an
upstream link fails and twenty downstream devices go unreachable. A good first step is to check whether a
parent device or link is already in alarm and, if so, link the tickets instead of sending twenty separate
investigations to twenty people. You don&rsquo;t need a full correlation engine for that. A topology lookup and a
check of open alerts gets you most of the way, and it&rsquo;s a first step toward the
<a href="/posts/observability-business-impact/">blast-radius view of observability</a>.</p>
<h2 id="where-does-ai-fit">Where does AI fit?</h2>
<p>LLMs are good at reading a wall of <code>show</code> output and writing a three-sentence summary. They&rsquo;re less reliable at
deciding on their own which commands to run against production gear. The pattern that works: let deterministic
automation collect the data your runbook defines, then hand that output to a model to summarize and suggest a
likely cause.</p>
<p>If you&rsquo;re building agents with MCP tools, the runbook becomes the agent&rsquo;s instructions and the read-only
collectors become its tools. See <a href="/posts/buildyourownmcp/">build your own MCP server</a> for that side. The runbook
still has to come from someone who knows the network, which brings you back to the alert list.</p>
<h2 id="how-do-you-show-its-working">How do you show it&rsquo;s working?</h2>
<p>In the same survey, 46.2% of respondents doing automation have no formal way to measure what it delivers. Teams
with at least one dedicated automation engineer are more automated (45.0% vs 33.0%), more likely to validate in
production (42.9% vs 16.0%), and less likely to skip measurement (32.7% vs 59.2%). Chris was careful to call
those associations, not causes. His point in the room was simpler: without measurement, &ldquo;it&rsquo;s gonna be hard to
build trust, to build budget, to gain staff&rdquo; for more automation. Measuring is cheap to start.</p>
<p>Mean time to repair is noisy and depends on things automation doesn&rsquo;t control, like waiting on a carrier.
Measure the time from alert to first useful fact instead. If an engineer used to spend twelve minutes gathering
data and now opens a ticket that already has it, that&rsquo;s a number you can show every month. Also ask the on-call
engineers whether the enrichment helped. If they ignore it, find out why.</p>
<h2 id="isnt-this-just-replacing-engineers">Isn&rsquo;t this just replacing engineers?</h2>
<p>No. Every team has one or two people who know how to work a particular alert, and when they&rsquo;re on vacation it
takes three times as long. Writing their first ten minutes into code makes what&rsquo;s in their head available at
2 AM to whoever&rsquo;s on call. The junior engineer on night shift starts where the senior engineer would.</p>
<h2 id="how-do-you-start-next-week">How do you start next week?</h2>
<p>You don&rsquo;t need a platform decision or a budget:</p>
<ol>
<li>Export 90 days of alerts and rank them by frequency times triage time.</li>
<li>Pick the top one.</li>
<li>Ask the person who handles it best what they do first. Write it down.</li>
<li>Automate only the data collection, and post the results to the ticket.</li>
<li>Ask the on-call engineers if it helped. Adjust, then pick the next alert.</li>
</ol>
<p>The unknowns are real, but you don&rsquo;t have to start with them. Your monitoring already told you what&rsquo;s
important. Start there.</p>
]]></content:encoded></item><item><title>Observability Isn't Uptime: It's Knowing What Breaks the Business</title><link>https://networkdoodles.com/posts/observability-business-impact/</link><pubDate>Sun, 15 Nov 2026 00:00:00 +0000</pubDate><dc:creator>David Henderson</dc:creator><guid>https://networkdoodles.com/posts/observability-business-impact/</guid><category>network-automation</category><category>automation</category><category>netbox</category><category>testing</category><category>ai</category><category>devops</category><description>Observability isn't uptime or monitoring. It ties signals, inventory and business flows together so you know a failure's blast radius before you change a thing.</description><content:encoded xmlns:content="http://purl.org/rss/1.0/modules/content/"><![CDATA[<div class="details admonition abstract open">
        <div class="details-summary admonition-title "><i class="icon admonition-icon icon-abstract"></i>Short answer<i class="details-icon  admonition-icon admonition-icon-arrow-right"></i></div>
        <div class="details-content">
            <div class="admonition-content">Observability isn&rsquo;t uptime, monitoring or telemetry. It&rsquo;s all of those, plus your inventory and your business
flows, tied together so you can see end to end. The question it answers isn&rsquo;t &ldquo;is this switch up?&rdquo; It&rsquo;s &ldquo;if this
switch, server, site or circuit goes down, which business flow is impacted?&rdquo; That&rsquo;s blast radius. A digital twin
that can&rsquo;t answer that question isn&rsquo;t a valid test, and automation or AI that can&rsquo;t see your real environment
has nothing to act on.</div>
        </div>
    </div>
<p>I hear &ldquo;observability&rdquo; used for a lot of things. A dashboard full of green interfaces. A syslog server. A
streaming telemetry pipeline. An uptime report that says 99.99%. Each of those is a piece of it. None of them is
observability.</p>
<h2 id="what-did-a-room-full-of-network-engineers-say-observability-was">What did a room full of network engineers say observability was?</h2>
<p>At the October 14 meeting of PA NUG, the PA Network Users Group, I was on the discussion panel, and we turned some questions around on the
audience. What&rsquo;s your company&rsquo;s observability strategy? What&rsquo;s the observability strategy inside your networking
group? Or, simplest of all: can you define what observability means?</p>
<p>The answers were shocking. They had no idea what their company&rsquo;s observability strategy was. Most of them
couldn&rsquo;t define observability at all. When they tried, they mostly tied it to up/down status and tools like
SolarWinds. And when we asked about the strategy for just the networking piece, it came down to monitoring, not
observability: collecting telemetry, without defining what any of it means for the business.</p>
<p>That isn&rsquo;t a knock on the people in that room. When a word gets used for every product on the market, nobody
stops to say what the goal is.</p>
<p>Here&rsquo;s my definition, and the rest of this post argues for it: <strong>observability is understanding how anything in
your environment can impact your business flows.</strong> That means seeing end to end, across the network, the
servers, the applications, and the circuits and providers underneath them, with an inventory that says how they
all connect.</p>
<hr>
<h2 id="what-isnt-observability">What isn&rsquo;t observability?</h2>
<p>Observability isn&rsquo;t uptime. It isn&rsquo;t monitoring. It isn&rsquo;t telemetry. It&rsquo;s all of those things, plus your
inventory and your business flows. The first three are inputs. On their own, each one answers a narrower
question than the one the business is asking.</p>
<table>
  <thead>
      <tr>
          <th>What you have</th>
          <th>The question it answers</th>
          <th>What it can&rsquo;t tell you</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><strong>Uptime</strong></td>
          <td>Was it up, and for what share of the month?</td>
          <td>Whether anything that mattered was affected while it was down</td>
      </tr>
      <tr>
          <td><strong>Monitoring</strong></td>
          <td>Is this device, link or service up or down right now?</td>
          <td>What depends on it</td>
      </tr>
      <tr>
          <td><strong>Telemetry</strong></td>
          <td>What&rsquo;s the CPU, the error count, the latency, the interface rate?</td>
          <td>Whether those numbers mean a business problem</td>
      </tr>
      <tr>
          <td><strong>Inventory</strong></td>
          <td>What do we have, where is it, and how is it connected?</td>
          <td>Whether it&rsquo;s healthy right now</td>
      </tr>
      <tr>
          <td><strong>Business flows</strong></td>
          <td>What does the business actually do, and what does each flow need?</td>
          <td>Nothing, unless it&rsquo;s mapped to everything above</td>
      </tr>
  </tbody>
</table>
<p>Read down the right-hand column. Every piece leaves a gap, and the gaps are where outages turn into long nights.
A switch can be up while the application behind it is failing. A circuit can be green while it&rsquo;s dropping enough
packets to time out a file transfer. A server can be &ldquo;healthy&rdquo; while it&rsquo;s the one database three business flows
can&rsquo;t live without.</p>
<h2 id="what-is-observability-then">What is observability, then?</h2>
<p>Observability is being able to see end to end. Not just &ldquo;is a switch up or down&rdquo;, &ldquo;is a circuit up or down&rdquo; or
&ldquo;is a site up or down&rdquo;, but &ldquo;if that site or that server is down, which business flow is impacted?&rdquo;</p>
<p>A <strong>business flow</strong> is something the business does that people would notice if it stopped: taking an order,
running payroll, admitting a patient, shipping a package. Each flow runs on applications. Each application runs
on servers. Those servers sit on network infrastructure, and that infrastructure depends on circuits and service
providers. That&rsquo;s a chain, and observability means you can see the whole chain at once.</p>
<p>So observability has four parts, and you need all of them:</p>
<ol>
<li><strong>Signals</strong>: the up/down state, metrics, logs and telemetry from every layer.</li>
<li><strong>Inventory</strong>: what exists, where it is, and what it&rsquo;s connected to. This is your source of truth.</li>
<li><strong>Dependencies</strong>: which application runs on which servers, which servers hang off which switches, which site
rides which circuit from which provider.</li>
<li><strong>Business flows</strong>: the top of the chain, mapped down to the applications they use.</li>
</ol>
<p>The signals tell you something changed. The other three tell you what it means.</p>
<h2 id="what-is-blast-radius">What is blast radius?</h2>
<p><strong>Blast radius</strong> is everything that&rsquo;s affected when one piece fails. Not the device that alarmed: everything
upstream of it that stops working, all the way up to the business flows. The same failure can have a tiny blast
radius or a huge one, and you can&rsquo;t tell which from the alarm.</p>
<p>Here&rsquo;s a hypothetical example. Say the payroll app runs on an app server and a database server in your data
center. Each pay period it sends a payment file to the bank, out through the WAN router and a provider circuit. A
second flow, timesheets, uses a different app server but the same database. Step through it and watch what each
failure takes out.</p>

<figure class="netsim" data-netsim tabindex="0" aria-label="A dependency chain: two business flows, payroll and timesheets, depend on two applications, three servers, a data center switch, a WAN router and a provider circuit. When the circuit fails only payroll is impacted; when the shared database is taken down in a twin both flows are. Use the buttons, or the left and right arrow keys, to step through it." data-autoplay="false">
  <svg class="ns-svg" role="img" aria-label="A dependency chain: two business flows, payroll and timesheets, depend on two applications, three servers, a data center switch, a WAN router and a provider circuit. When the circuit fails only payroll is impacted; when the shared database is taken down in a twin both flows are"></svg>
  <div class="ns-ctl">
    <button class="icon-btn" type="button" data-ns="restart" aria-label="Back to the first step" title="Start again"><svg class="ico" viewBox="0 0 24 24" width="18" height="18" fill="none" stroke="currentColor" stroke-width="1.7" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M6 5v14M19 5l-9 7 9 7z"/></svg>
</button>
    <button class="icon-btn" type="button" data-ns="prev" aria-label="Previous step" title="Previous step"><svg class="ico" viewBox="0 0 24 24" width="18" height="18" fill="none" stroke="currentColor" stroke-width="1.7" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M19 12H5M11 6l-6 6 6 6"/></svg>
</button>
    <button class="btn sm primary ns-play" type="button" data-ns="play" aria-pressed="false"><span data-ns-play>Play</span></button>
    <button class="icon-btn" type="button" data-ns="next" aria-label="Next step" title="Next step"><svg class="ico" viewBox="0 0 24 24" width="18" height="18" fill="none" stroke="currentColor" stroke-width="1.7" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M5 12h14M13 6l6 6-6 6"/></svg>
</button>
    <button class="icon-btn" type="button" data-ns="again" aria-label="Play this step again" title="Play this step again"><svg class="ico" viewBox="0 0 24 24" width="18" height="18" fill="none" stroke="currentColor" stroke-width="1.7" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M4 12a8 8 0 1 0 2.5-5.8M4 4v4h4"/></svg>
</button>
    <span class="ns-count mono" data-ns-count></span>
    <button class="btn sm ns-speed" type="button" data-ns="speed" aria-pressed="false">Slow: off</button>
  </div>
  <div class="ns-dots" data-ns-dots role="group" aria-label="Jump to a step"></div>
  <figcaption class="ns-caption" data-ns-caption aria-live="polite"></figcaption>
  <div class="ns-panels">
    <div class="ns-codebox" data-ns-code hidden></div>
    <div class="ns-frame" data-ns-frame hidden></div>
    <div class="ns-tablebox" data-ns-table hidden></div>
  </div><p class="ns-note">A hypothetical environment. The point is the chain, not the devices.</p>
  <script type="application/json">{"h":400,"links":[["f1","a1"],["f2","a2"],["a1","s1"],["a1","db"],["a2","db"],["a2","s2"],["s1","sw"],["db","sw"],["s2","sw"],["sw","rtr"],["rtr","isp"]],"narrow":{"at":{"a1":[95,175],"a2":[265,175],"db":[180,305],"f1":[95,55],"f2":[265,55],"isp":[180,625],"rtr":[180,520],"s1":[55,305],"s2":[305,305],"sw":[180,420]},"h":680,"w":360},"nodes":{"a1":{"kind":"gear","label":"Payroll app","x":235,"y":110},"a2":{"kind":"gear","label":"Time app","x":235,"y":290},"db":{"kind":"db","label":"Database","x":400,"y":200},"f1":{"kind":"person","label":"Payroll run","sub":"business flow","x":80,"y":110},"f2":{"kind":"person","label":"Timesheets","sub":"business flow","x":80,"y":290},"isp":{"kind":"cloud","label":"Circuit","sub":"provider","x":815,"y":200},"rtr":{"kind":"router","label":"WAN router","x":700,"y":200},"s1":{"kind":"server","label":"App server 1","x":400,"y":70},"s2":{"kind":"server","label":"App server 2","x":400,"y":330},"sw":{"kind":"switch","label":"DC switch","x":565,"y":200}},"steps":[{"hot":["f1","f2"],"text":"Two things the business does: run **payroll** each pay period, and collect **timesheets** every day. This is the layer people notice when it breaks.","title":"Start with the business flows."},{"hot":["a1","a2"],"links":[["f1","a1"],["f2","a2"]],"text":"Payroll runs on the payroll app. Timesheets run on the time app.","title":"Each flow runs on an application."},{"hot":["s1","db","s2"],"links":[["a1","s1"],["a1","db"],["a2","db"],["a2","s2"]],"text":"Each app has its own app server, and **both share one database server**. That shared piece matters later.","title":"Applications run on servers."},{"hot":["sw","rtr"],"links":[["s1","sw"],["db","sw"],["s2","sw"],["sw","rtr"]],"text":"All three servers connect to the data center switch, which connects to the WAN router.","title":"Servers sit on the network."},{"hot":["isp"],"packets":[{"c":"amber","label":"payment file","path":["f1","a1","s1","sw","rtr","isp"]}],"text":"Payroll sends its payment file to the bank over this circuit. Timesheets never leave the data center.","title":"The network rides a provider circuit."},{"bad":["isp"],"cut":[["rtr","isp"]],"say":{"isp":"circuit down"},"text":"An alarm fires: **circuit down**. That's true, and it's all a monitoring tool knows. It doesn't say who's hurt.","title":"Monitoring sees a circuit go down."},{"bad":["isp","a1","f1"],"cut":[["rtr","isp"]],"good":["f2","a2"],"packets":[{"c":"red","end":"drop","label":"payment file","path":["f1","a1","s1","sw","rtr","isp"]}],"table":{"bad":[0],"cols":["business flow","impact"],"hl":1,"rows":[["Payroll run","can't send payment file"],["Timesheets","none"]],"title":"Impact: circuit down"},"text":"The payment file can't reach the bank, so the **payroll run is impacted**. Timesheets don't use the circuit, so they're fine. Same alarm, one flow down.","title":"Blast radius: walk up the chain."},{"bad":["db","a1","a2","f1","f2"],"cut":[["a1","db"],["a2","db"],["db","sw"]],"say":{"db":"planned reboot"},"table":{"bad":[0,1],"cols":["business flow","impact"],"rows":[["Payroll run","down"],["Timesheets","down"]],"title":"Impact: database down (simulated)"},"text":"Before patching the database server, ask the twin: *if this goes down, what breaks?* The answer isn't \"one server.\" It's **both business flows**. That's the number the change board needs.","title":"Now test a change in a twin."}],"w":880}</script>
</figure>
<script src="/js/netsim.c40a98d103487ae5866edfa12abb4afc1d3aa7ad2d12fed4fe925fe00b588106.js" integrity="sha256-xAqY0QNIeuWGbt&#43;hKrtK/B06p60tEv7U/pJf4AtYgQY=" defer></script>

<p>The circuit alarm and the database reboot are both &ldquo;one thing down.&rdquo; Their blast radius is completely
different. If all you can see is device state, you can&rsquo;t tell those two apart, and that&rsquo;s the difference between
a quiet note to the payroll team and a call to every manager in the company.</p>
<h2 id="why-cant-the-network-servers-and-apps-stay-in-silos">Why can&rsquo;t the network, servers and apps stay in silos?</h2>
<p>Observability has to measure the entire environment: the network, the servers and the applications. And not in
silos. They need to talk to each other.</p>
<p>In a lot of shops each team has its own tool. The network team watches interfaces and routing. The server team
watches CPU, memory and disk. The application team watches response times and errors. Someone else holds the
circuit IDs and the provider contacts in a spreadsheet. Every one of those tools can be excellent, and you can
still be blind, because none of them knows what&rsquo;s connected to what.</p>
<p>Applications are connected to the servers they run on, the network infrastructure they depend on, and the
circuits and service providers they depend on. If those relationships only live in people&rsquo;s heads, every
incident starts with a conference bridge where each team proves its own piece is green. Meanwhile the business
flow is still down.</p>
<table>
  <thead>
      <tr>
          <th>Siloed tools</th>
          <th>Connected observability</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>&ldquo;Switch 4 is down&rdquo;</td>
          <td>&ldquo;Switch 4 is down, and it carries the payroll database&rdquo;</td>
      </tr>
      <tr>
          <td>Each team checks its own dashboard</td>
          <td>One view walks the chain from the failure up to the flows</td>
      </tr>
      <tr>
          <td>Impact found by asking around on a bridge call</td>
          <td>Impact known from the dependency map before the bridge starts</td>
      </tr>
      <tr>
          <td>Circuits tracked in a spreadsheet</td>
          <td>Circuits and providers in the inventory, tied to the sites and apps that use them</td>
      </tr>
      <tr>
          <td>Alerts ranked by device</td>
          <td>Alerts ranked by which business flow they touch</td>
      </tr>
  </tbody>
</table>
<p>The inventory is the glue. A source of truth like NetBox already knows your sites, devices, cables, circuits and
providers. I&rsquo;ve written before about <a href="/posts/netboxasinventory/">using NetBox as your Ansible inventory</a>. The same
data is what lets an alert on a circuit become &ldquo;these sites, these apps, these flows.&rdquo; What&rsquo;s usually missing is
the top of the chain: applications and business flows mapped onto that infrastructure.</p>
<h2 id="why-is-a-digital-twin-almost-useless-without-observability">Why is a digital twin almost useless without observability?</h2>
<p>You can build a digital twin or you can buy one. A <strong>digital twin</strong> is a model of your environment that you can
change and test without touching production. The idea is good. The problem is what it can tell you.</p>
<p>If the twin isn&rsquo;t attached to full observability, where you understand the actual impact to the business, it&rsquo;s
almost useless. It will tell you &ldquo;I put this in, this goes down.&rdquo; But what does that actually mean? If you can&rsquo;t
answer that question, it&rsquo;s not a valid test.</p>
<p>A twin built on network data alone reports network state: routes withdrawn, interfaces down, a port blocked.
The change board wants to know whether payroll still runs on Friday. A twin that can&rsquo;t say hands that question
back to a person, who answers it from memory, which is the same guess you&rsquo;d make without a twin. Attach it to
observability and the same simulation gives a different answer:</p>
<table>
  <thead>
      <tr>
          <th>Twin without observability</th>
          <th>Twin with observability</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>&ldquo;These interfaces go down&rdquo;</td>
          <td>&ldquo;These interfaces go down, so the payroll and timesheet flows lose their database&rdquo;</td>
      </tr>
      <tr>
          <td>&ldquo;Route to this prefix withdrawn&rdquo;</td>
          <td>&ldquo;That prefix is the bank&rsquo;s file transfer endpoint, used only by payroll&rdquo;</td>
      </tr>
      <tr>
          <td>Pass or fail on network state</td>
          <td>Pass or fail on business impact</td>
      </tr>
      <tr>
          <td>Someone still has to interpret it</td>
          <td>The result is the impact statement</td>
      </tr>
  </tbody>
</table>
<p>Then a real digital-twin analysis of blast radius gives real data. You can understand the actual business
impact of a potential change before you make it, and you can schedule it, staff it and communicate it based on
that.</p>

<h2 id="why-is-observability-the-foundation-for-automation-and-ai">Why is observability the foundation for automation and AI?</h2>
<p>You can&rsquo;t automate what you can&rsquo;t see. You can&rsquo;t predict what you can&rsquo;t see. And AI can&rsquo;t act if it doesn&rsquo;t have
context for what your actual environment is.</p>
<p><strong>Automation</strong> needs to know what it&rsquo;s touching. A playbook that reboots a switch is easy to write. Knowing
whether it&rsquo;s safe to run at 2 p.m. on a Tuesday is the hard part, and that&rsquo;s an observability question. Without
it, automation runs exactly as fast as your riskiest guess. Your alerts are also the best place to start
automating troubleshooting: see <a href="/posts/automate-network-troubleshooting-from-alerts/">where do you start automating network troubleshooting?</a></p>
<p><strong>Prediction</strong> needs history and context together. You can trend a circuit&rsquo;s errors or a server&rsquo;s disk all day,
but a forecast only matters when you know what depends on the thing you&rsquo;re forecasting. &ldquo;This disk fills in 9
days&rdquo; is a ticket. &ldquo;The payroll database fills in 9 days, before the next pay run&rdquo; is a priority.</p>
<p><strong>Agentic operations</strong>, where an AI agent investigates and proposes changes, depend on it even more. An agent
reasons over whatever context you give it. Give it interface counters and it will reason about interface
counters. Give it the inventory, the dependencies, the telemetry and the business flows, and it can reason about
impact. When we took the InfraGenie hackathon agent to production as ODIN, the piece we added in front of every
change was exactly this: observability, telemetry, correlation and context, so the agent finds the root cause
first. I wrote that up in <a href="/posts/infragenie-to-odin/">From InfraGenie to ODIN</a>.</p>
<p>Here&rsquo;s how I think about the order:</p>
<ol>
<li><strong>See it</strong>: signals, inventory, dependencies and business flows, connected.</li>
<li><strong>Understand it</strong>: blast radius for any piece that fails or changes.</li>
<li><strong>Test it</strong>: a digital twin that reports business impact, not just network state.</li>
<li><strong>Act on it</strong>: automation and agents that work from that context, with humans approving what matters.</li>
</ol>
<p>Skip step 1 and every step after it is built on a guess.</p>
<h2 id="heres-what-i-see-go-wrong">Here&rsquo;s what I see go wrong</h2>
<p>These are the patterns I run into over and over.</p>
<ul>
<li><strong>Calling a dashboard observability.</strong> A wall of green interfaces looks like control. It tells you the devices
are up, not that the business is working.</li>
<li><strong>Measuring uptime instead of impact.</strong> A device can hit 99.99% and still have been down during the one hour
that mattered. Uptime without context can&rsquo;t tell you which hour that was.</li>
<li><strong>Keeping the inventory separate from the signals.</strong> The source of truth lives in one tool, the alerts in
another, and nothing joins them. Every alert then needs a human to look up what it means.</li>
<li><strong>Leaving circuits and providers out.</strong> Circuit IDs, providers and contacts in a spreadsheet nobody updates.
The circuit is often the piece with the biggest blast radius and the least visibility.</li>
<li><strong>Never mapping the business flows.</strong> The infrastructure is documented down to the cable, and nobody has
written down which applications payroll actually uses. The top of the chain is missing, so impact can&rsquo;t be
calculated.</li>
<li><strong>Buying a twin first.</strong> A twin shows network state changes nicely. Without the dependency map it has nothing
to say about the business, and it gets used for a few demos.</li>
<li><strong>Pointing automation or AI at a blind environment.</strong> Automating changes, or handing an agent tools, before it
can see dependencies means it acts fast on incomplete information.</li>
<li><strong>Trying to map everything at once.</strong> That project never ends. One flow, done end to end, teaches you more.</li>
</ul>
<h2 id="where-do-i-start">Where do I start?</h2>
<p>Start small, and start from the top of the chain, not the bottom.</p>
<ol>
<li><strong>Pick one critical business flow.</strong> Ask the business which one would hurt most if it stopped. Write it down
in plain words.</li>
<li><strong>Map it end to end.</strong> List every application it uses, every server those apps run on, the network
infrastructure those servers depend on, and every circuit and provider in the path. Expect surprises: a shared
database, a single circuit, a server nobody owns.</li>
<li><strong>Tie it to your inventory and source of truth.</strong> Put those dependencies where your inventory lives, so the
map is data and not a diagram that goes stale. Sites, devices, circuits and providers are probably there
already. Add the applications and the flow.</li>
<li><strong>Add signals to every piece in the chain.</strong> Up/down, metrics and logs for each layer, from the circuit to the
application, so a failure anywhere shows up against the flow it affects.</li>
<li><strong>Ask the blast-radius question.</strong> For each piece in the chain: if this goes down, what happens to the flow?
If you can answer it from data, you have observability for that flow. Then pick the next one.</li>
</ol>
<p>Once one flow works like that, a digital twin has something real to test against, and automation has something
real to act on.</p>

<p>Change testing, automation and AI all sit on top of observability. Build that first.</p>
]]></content:encoded></item></channel></rss>