The Moment I Knew Something Was Wrong

The 60/70 Problem

What Makes Agents Different (It's Not What You Think)

1. Four Layers Change Simultaneously

2. You Can't Unit Test Non-Determinism

3. Cost Is a Deployment Variable

4. Rollback Is Structurally Harder

The Missing Layer

What the Right Solution Looks Like

Introducing AgentRoll

What's Next I've spent 9 years deploying services on Kubernetes. Last year, my team started deploying AI agents to production. That's when I realized everything I knew about safe deployments was suddenly inadequate. It was a Thursday afternoon. We pushed a "minor update" to a customer-facing AI agent — a two-word change in the system prompt. No code changes. No dependency updates. The container image tag didn't even change. Within an hour, the agent started hallucinating product features that didn't exist. Customer complaints spiked. We rolled back, but "rolling back" a prompt change in our setup meant SSH-ing into a config repo, reverting a YAML file, and waiting for the CI pipeline to redeploy — a 15-minute scramble that felt like an eternity. The worst part? Our monitoring didn't catch it. Grafana showed green across the board — HTTP 200s, latency within SLA, CPU and memory nominal. Every signal we trusted for years told us everything was fine, while the agent was confidently lying to customers. That's when it hit me: we were deploying AI agents like microservices, and it was fundamentally wrong. I started talking to other platform engineers. The pattern was eerily consistent. At a recent meetup, I asked: "How do you deploy your agents?" The most common answers were "GitHub Actions to K8s" and "docker compose is enough." A ZenML survey confirmed this — 60-70% of teams deploy agents like any other service. Then I asked: "How do you know if a new version is better than the old one?" Silence. Nervous laughter. "We... watch Slack for complaints?" This isn't a tooling gap. It's a conceptual gap. We're applying a deployment paradigm designed for deterministic, stateless services to something fundamentally different — and the cracks are showing. A Cleanlab survey of 1,837 practitioners found that 70% of regulated enterprises rebuild their agent stack every three months. Not iterate — rebuild. When I first encountered this problem, I assumed it was about AI being "harder" in some vague sense. But after months of digging, I found the differences are structural and specific. There are exactly four things that break your existing deployment playbook. A microservice has one version identity: the container image tag. Change the code, bump the tag, deploy. Simple. An agent's behavior depends on four interdependent layers: Change any one of these, and behavior shifts unpredictably. Change two simultaneously, and you have a combinatorial explosion of possible behaviors that no amount of pre-deployment testing can cover. Here's the kicker: your Git history won't tell you what changed. A prompt update is a config change, not a code change. A model version swap is an environment variable change. A tool API upgrade happens upstream, outside your control entirely. Your deployment pipeline sees "nothing changed" while the agent's behavior has transformed. Practitioners report that tool versioning causes 60% of production agent failures and model drift causes 40%. These are failure modes that don't exist in traditional software deployment. If I deploy a REST API, I can assert: "Given input X, output should be Y." If the test passes, I have reasonable confidence the service works. With an agent, the same input can produce different outputs every time. The agent might call different tools, follow different reasoning paths, or generate slightly different phrasings — and all of these might be "correct." Traditional pass/fail assertions are meaningless. Anthropic's engineering team put it clearly in their agent evaluation guide: you need multiple trials per task, with code-based, model-based, and human graders. It's a fundamentally different testing paradigm — closer to A/B testing in product development than to unit testing in software engineering. But here's the operational problem: if your evaluation takes 20 minutes to set up, developers won't test. The AWS DevOps Agent team discovered this firsthand. You need evaluation infrastructure that's as easy as kubectl apply — integrated into the deployment pipeline, not bolted on after the fact. When I deploy a microservice, I know roughly what it costs per request — CPU time is predictable, memory is bounded. I can capacity plan. An agent's cost per request varies wildly. A simple query might consume 500 tokens. A complex multi-step task might trigger 15 tool calls and consume 50,000 tokens — a 100x difference. One startup spent €5,000 in compute to determine optimal email send times — a task solvable with 50 lines of business logic. This means a "successful" deployment — one where the agent works correctly — can still bankrupt you if the new version's reasoning patterns are more token-hungry. Cost needs to be a first-class rollback signal, not an afterthought in your monthly cloud bill review. Rolling back a microservice means pointing traffic to the previous container. Done. Rolling back a stateful agent means undoing actions it already took. If the agent sent emails, created Jira tickets, updated a CRM, or modified a database — those side effects persist. You can revert the container, but you can't unsend the email. This makes the prevention of bad deployments — catching problems during canary before they reach 100% of traffic — far more critical for agents than for traditional services. Here's the thing: excellent tools exist for every other part of the agent lifecycle. Building agents? LangGraph, CrewAI, OpenAI Agents SDK — mature frameworks with active communities. Serving models? vLLM, Ray Serve, BentoML — production-hardened inference engines. Observing agents? Langfuse, Arize Phoenix, LangSmith — comprehensive tracing and evaluation. Running on Kubernetes? Kagent is bringing agent CRDs to the CNCF ecosystem. But between "I built an agent" and "it's reliably running in production with safe, continuous updates" — there's nothing. No open-source tool treats agents as first-class deployable units with evaluation-gated progressive delivery. The agent framework authors know deployment is hard. They explicitly punt on it — LangGraph requires a commercial license for production deployment, CrewAI offers a managed cloud, OpenAI's SDK has no hosting story at all. They assume you'll solve deployment yourself. Meanwhile, Argo Rollouts provides beautiful progressive delivery for Kubernetes workloads — canary deployments, blue-green switches, automated analysis. But it doesn't understand agent health. It can check HTTP error rates, but it can't check hallucination rates. It can measure latency, but not cost-per-task. It has no concept of prompt versions or model versions. After living with this problem for months, I believe the right approach has three principles: 1. Composite version tracking. An agent's version isn't just a container tag — it's the combination of prompt + model + tools + code. The deployment system needs to track all four layers as a single versioned entity. When you do a canary rollout, you need to know: "stable is running prompt v1 with Claude 3.5, canary is running prompt v2 with Claude 4." 2. Evaluation-gated progressive delivery. Instead of checking HTTP 200 rates during canary, check agent-specific quality signals: hallucination rate, tool call success rate, cost-per-task, task completion rate. If quality degrades at 5% traffic, auto-rollback before it hits 100%. 3. Framework-agnostic, Kubernetes-native. It shouldn't matter if your agent is built with LangGraph, CrewAI, or custom Python. Once it's a container, the deployment infrastructure should work the same. And it should be built on top of proven Kubernetes primitives — not another proprietary platform. This is why I started building AgentRoll — an open-source Kubernetes operator that brings evaluation-gated progressive delivery to AI agent deployments. Instead of writing raw Argo Rollouts YAML and figuring out AnalysisTemplates from scratch, you declare an AgentDeployment: Under the hood, AgentRoll: It's early — we're in alpha. But the core loop works: AgentDeployment → Argo Rollout with canary strategy → evaluation-gated promotion → automatic rollback on failure. The roadmap is straightforward: If you're a platform engineer deploying AI agents and feeling the pain I described — I'd love to hear your story. The gap between "agents work in my notebook" and "agents are reliably serving production traffic" is real, and I believe it's solvable with the right infrastructure primitives. Star the repo if this resonates. Open an issue if you have a use case I haven't considered. The best infrastructure is built by the people who feel the pain. Max Li is a software infrastructure engineer with 9 years of experience in Kubernetes, Helm, Terraform, and cloud-native systems. He's building AgentRoll because he believes AI agents deserve the same deployment rigor as microservices. Templates let you quickly answer FAQs or store snippets for re-use. Hide child comments as well For further actions, you may consider blocking this person and/or reporting abuse