Skip to content
Course outline

Learn it

Practice

Interview ready

The AI Architect Roadmap

Bounding a runaway agent loop, from first principles

Why an agent can pass every health check while quietly burning a six-figure annual run rate, the five conditions that defeat a loop's exit, and the two-instrument alarm pattern that catches it in minutes instead of months. Quiz and cue cards included.

Verified as of 2026-09-22. Full write-up: Every dashboard was green while the agent burned six figures a year.

In this guide, you'll learn to:

  • The five conditions that defeat a model-judged agent loop exit, and which one lives outside the loop
  • How to bound a runaway agent loop with more than one tripwire, and why a single cap is a false-confidence trap
  • Why standard uptime, latency and error-rate dashboards are structurally blind to a cost runaway
  • The two-instrument pattern, per-session and fleet, for catching both the acute loop and the diffuse regression
Why a green dashboard can still be wrong

Traditional monitoring answers one question: is it up? HTTP 200 on every call, p99 latency normal, CPU and memory fine, error rate zero. None of those metrics has an opinion on the question that actually matters: is the agent quietly doing something insane? The signals that were moving, tokens per turn, dollars per hour, tool calls per request, weren't on any dashboard.

One real agent ran at a rate that annualised into six figures for days, every dashboard green the whole time, until someone happened to read the bill. It was luck that it was caught on day nine and not day ninety, and luck is not a control.

Why agents run away

Agents terminate fine, almost always. A runaway needs a specific condition to defeat the exit, and it isn't "someone forgot an exit": the exit was there, it was a judgement call, and a rare input defeated it. Five conditions keep showing up.

Tool results, often thousands of characters each, pile up until the context is so large the model loses the thread and keeps calling tools without noticing it already has the answer. This is the one teams hit most.

Bound the loop

Any competent engineer can add a max-iterations cap, which is exactly why people ship one and stop. A single cap breeds false confidence: it doesn't touch a poison-pill re-drive, and it does nothing about an agent that stays inside every limit while quietly getting more expensive per answer. Cap more than one thing, and trip on whichever comes first.

Cap at roughly 12 per turn, set from your own traffic, above the heaviest legitimate turn. Catches a tool-calling loop that never checks it already has the answer.

Watch the spend two ways

Stopping the loop is the easy half. Seeing the next one coming is the half that matters, and it takes two instruments, because there are two different failures to catch, and each is blind to the other.

Tool calls and tokens per session against a p99, emitted every cycle from inside the event loop (not per turn: a runaway turn may never call end_turn). One stuck session is invisible in the fleet total but a klaxon against one normal session. This is the signal that catches the acute runaway in minutes instead of days.

Full write-up

The real numbers, the EMF instrumentation code, and why the alarm threshold is fragile on purpose: Every dashboard was green while the agent burned six figures a year.

Six questions, no pass mark, unlimited attempts. This is a self-check, not a gate.

Check your understanding

0 of 6 answered

  1. 1. An agent is passing every health check, 100% uptime, zero errors, yet trending toward a six-figure annual cost. Which signal would have caught it in minutes instead of days?

  2. 2. Which of the five runaway triggers lives outside the agent loop entirely?

  3. 3. The loop guard caps three separate things. Which combination?

  4. 4. Why emit the per-session counters from inside the event loop body, on every cycle, rather than once at the end of a turn?

  5. 5. Should a cost alarm trigger on raw token counts or on dollars converted from them?

  6. 6. What does the fleet-level token-rate alarm catch that the per-session alarm is blind to?

Cue cards

Try to answer before you flip each one.

How this comes up in an interview

"An agent has 100% uptime and zero errors, yet is burning a six-figure annual run rate. How is that possible, and what would you instrument to catch it in minutes instead of months?"

Traditional monitoring answers one question: is it up? HTTP 200, p99 latency, CPU, error rate, all fine, because the agent is doing exactly what it was told, it just never decided it was satisfied. The signals that actually move, tokens and dollars per hour, tool calls per request, aren't on a default dashboard. I'd instrument a per-session alarm (tool calls and tokens per session against a p99, emitted every cycle from inside the event loop) so one stuck session stands out immediately against a single normal session, instead of being buried in a fleet total.

"Name the five conditions that defeat a model-judged loop exit, and which one lives outside the agent's own loop entirely?"

Context bloat (tool results balloon the context until the model loses the thread), an unsatisfiable goal (the model keeps trying to reach a stop condition it can't), model regeneration (malformed output gets re-invoked with the same reasoning), a broken or oscillating tool, and a poison-pill re-drive. The re-drive is the odd one out: it isn't the agent misbehaving at all, it's an external queue, scheduler, or retry re-invoking work that crashes before reaching a terminal state, so it gets picked up and re-billed forever.

"How would you bound a runaway agent loop, and why is a single max-iterations cap not enough?"

Cap more than one thing and trip on whichever comes first: tool calls (~12/turn), reasoning restarts (~6/turn), and wall-clock (~120s/turn), set from your own traffic above the heaviest legitimate turn. A single cap is a false-confidence trap: any competent engineer can add one and stop there, but it doesn't touch a poison-pill re-drive living outside the loop, and it does nothing about an agent that stays inside every limit while quietly getting more expensive per answer across the whole fleet.

"Why can't you rely on Bedrock's stock CloudWatch metrics to catch a single runaway session?"

Bedrock's metrics are dimensioned by model, not by session, so there's no built-in per-session number to alarm on. You have to emit it yourself, from inside the event loop on every cycle (as EMF, a structured log line, not PutMetricData, which throttles under a high-frequency runaway exactly when you need the signal), beside the loop guard's own counter. Anywhere higher in the call stack and you're back to a dashboard that stays green.

"Should a cost alarm trigger on dollars or on raw token counts? Why?"

Raw counts. Converting tokens to dollars is actively misleading in the case you're trying to catch: prompt caching means a re-drive loop is mostly cheap cache reads, so pricing it at full rate overstates the rate, and it's meaningless under provisioned throughput where the per-token price isn't the real cost driver at all. Counts are the un-launderable signal; keep the dollar conversion for the write-up and the CFO.

"What's the difference between what a per-session alarm catches and what a fleet-level alarm catches?"

Per-session, emitted every cycle, catches the acute single-session loop the moment one session runs hot, because it's compared against the profile of one normal session, not buried in a fleet total. Fleet-level token-rate catches what no single session reveals: a slow bleed where every session stays inside its cap but the whole fleet is quietly getting more expensive per answer, and a poison-pill re-drive smeared thin across many cheap sessions that each individually look normal. You need both; each is structurally blind to the failure the other catches.