online posts 26
posts

You Can't Tell an Agent Not to Hallucinate. You Can Grade It. (AutoCon 6, Part 4)

Eric Chou’s AutoCon 6 talk on AI evaluation: why pilots don’t survive production, the offline and online evaluation loop, and what it would take to trust my own home network agent with more.

Cover: grade the agent. A terminal scores an agent's proposed changes against a labeled ground-truth set with pass and fail results, next to a chain from ground truth to offline evaluation to production judge
part 4 of 11 · series AutoCon 6
  1. 1State of automation: start with your alerts
  2. 2Digital twins at scale
  3. 3Observability and blast radius
  4. 4AI evaluation
  5. 5Automation ROI
  6. 6Leading radical change
  7. 7Build vs. buy
  8. 8Wouldn't it be cool if...
  9. 9Lightning: the PB&J method
  10. 10Lightning: 3,000 routers
  11. 11Lightning: how much code is enough
series overview →
Short answer
AI agents fail in production because pilots run on curated data and nobody measures drift. The fix is an evaluation loop: write down what a right and wrong answer look like, build a labeled set from real examples, test prompts and models against it before rollout, then grade production traffic with a rubric and feed new edge cases back in. Start with 10-30 pass/fail examples for one workflow.

Eric Chou opened his AutoCon 6 talk with a show of hands. Nearly everyone in the room uses AI. About five people have an agent doing real work for them. That gap is the whole talk.

Why don’t AI pilots survive production?

Pilots run on curated data and curated questions. Production brings edge cases and users who do unpredictable things. Eric called shipping without a validation framework “hope-driven development,” and named two failure modes: demos that don’t reflect real-world mess, and silent drift, where model updates, new tools and undocumented prompt changes ship with no measured impact. Both end the same way: lost trust, lost budget, stalled project.

Models are probabilistic. Most operations need to be deterministic. His line: 99% sounds great until your life depends on it.

How do you evaluate an AI agent before and after rollout?

Start with ground truth. If you can’t write down what “right” looks like, you can’t measure it. Public leaderboards are a starting point. They test public data, not your data, your rules or your judgment.

PhaseWhat you do
Offline, before productionDefine the workflow and its business rules, build a reference dataset (real examples preferred), agree as a team on what good looks like, and iterate on prompts and models until the team accepts the results. Prompts and model choices get checked in like code.
Online, in productionHumans don’t scale, so an LLM judge scores traffic against a rubric. Log the traces that matter, classify the emerging problem buckets, and feed real edge cases back into the offline set.

The case studies made it concrete. DoorDash routes high-severity responses (money, refunds) to a human in real time and cut hallucinations 90% and severe compliance issues 99%. Alibaba Cloud ran a network troubleshooting system in production for ten months before publishing, with 20% faster troubleshooting. A clinical note-taking system kept recording the wrong drug when a patient corrected themselves mid-sentence, which is exactly the kind of edge case only real traffic finds.

My take: my approval step is my evaluation, and that doesn’t scale

This is the missing piece in most agentic ops pitches, including some I’ve given myself.

At home I run my own agent harness on Claude’s Agent SDK. It has read access to my network monitoring and telemetry and no write path to the network at all. The only way it changes the network is by submitting a change through my CI/CD pipeline, which I approve or reject from my phone. That’s the same split I wrote about in From InfraGenie to ODIN: agents propose, Ansible disposes. On lower-stakes systems I give it more room. While I was in Tucson it recovered a failed server in my basement, rebuilt it on new hardware and migrated the apps without me touching anything.

What I don’t have is what Eric described. My approval step is my evaluation, and that doesn’t scale past one person. A labeled set of 20 good and bad change proposals would tell me far more than my gut does at 11 p.m.

That’s also the answer to the hallway conversation afterward. Most companies don’t stall in the operations team. They stall with upper management and security, who are paid to be risk-averse. If securing the environment were my job, I’d be cautious too. You don’t move those people with a demo. You move them with evidence, and an evaluation set is how you produce it. Autonomy should expand one workflow at a time, based on evidence instead of intuition. A good first workflow is the read-only one from part 1 of this series: collecting evidence when an alert fires.

What can you do on Monday?

Pick one workflow. Write 10-30 pass/fail examples of what right and wrong look like. Do it before the rollout, not after the first incident.

This is part 4 of my AutoCon 6 series.