Testing an agent's control flow without a model

There is a category of agent bug that has nothing to do with the model, and almost nobody tests for it. The gate that can never be reached because the predicate above it is always false. The retry budget that decrements on the wrong branch and therefore never runs out. The loop whose exit edge was drawn but never wired. The phase that reports success because the failure it was watching for writes to a field nothing reads.

None of those are judgement calls. They are control flow, and control flow is ordinary software that can be tested the ordinary way — except that most agent harnesses tangle it so tightly with the model calls that the only way to exercise a path is to pay for the whole run. So the tests get written once, cost four dollars each, take nine minutes, go flaky within a fortnight, and are skipped in CI by the end of the month.

The fix is a separation, not a framework.

#Transport and judgement are different systems

A harness does two kinds of work and it is worth naming them apart. Transport is everything mechanical: pick up the work, classify it, isolate it, move it from node to node, count attempts, enforce a budget, record what happened, report back. Judgement is the part that needs a model: is this specification good enough, is this code correct, may this release proceed.

Once those are separate, the interesting property falls out: transport is deterministic. Given the same inputs and the same recorded verdicts, the flow takes the same path every time. And a deterministic system with a stubbed boundary can be executed exhaustively, offline, in milliseconds.

The delivery system I run leans on this hard. Its flows are scripts with fixed topology, bounded loops and a declared set of human stops, and every model call goes through one seam. A separate command executes any flow against a named scenario with that seam stubbed — every verdict supplied by the scenario rather than earned. No network, no keys, no spend. Control-flow verification for a seven-phase flow with three nested retry loops finishes before you have moved your hand off the keyboard.

#What a scenario actually contains

A scenario is not a mock of the model. It is a list of the verdicts a run encounters, in order, plus the state it starts from. That distinction matters: you are not asserting anything about what the model would say, only about what the machinery does once it has been told.

{
  "name": "gate-1-repairs-then-passes",
  "ticket": { "kind": "feature", "surfaces": ["api"] },
  "verdicts": [
    { "node": "spec-review", "result": "NO-GO", "blockers": 2 },
    { "node": "spec-review", "result": "GO" },
    { "node": "test", "result": "FAIL" },
    { "node": "test", "result": "PASS" },
    { "node": "panel", "result": "PASS" }
  ],
  "expect": {
    "reached": ["scout", "spec", "gate-1", "build", "gate-3", "approval"],
    "attempts": { "gate-1": 2, "build": 2 },
    "haltedAt": "approval"
  }
}

The assertions are the interesting half. reached catches an unreachable gate — the failure that is invisible in production because the run completes, just without ever having been checked. attempts catches a budget that does not decrement, which is the same bug as an infinite loop with better manners. haltedAt catches a flow that has quietly stopped stopping, which is how an unattended system ends up merging something nobody approved.

#The self-check suite that is not a test suite

Underneath the scenarios sits a different kind of harness, and it is the one I would build first in a new project. It answers questions about the system's own shape rather than about any particular run: does every declared agent exist, is every agent's prompt under its line budget, does every gate have at least one inbound edge and one path to a terminal state, does every enumerated guarded path still exist on disk, does every referenced scenario parse.

Fifty-odd named checks, no lint, no typecheck, no model. It runs in a couple of seconds and it catches the entire class of failure where a rename in one file silently orphans a reference in another. That class is common in agent projects specifically, because so much of the wiring lives in prompts and configuration rather than in code a compiler reads.

#The guard that watches for the cheapest route to green

Everything above assumes the tests are honest. There is one attack on that which is worth guarding explicitly, because it is what any optimiser — human or otherwise — will find first: when a test fails, the cheapest way to make it pass is to loosen the assertion.

So a hook watches for it. If a run modifies a test file in a way that weakens an existing assertion, that is surfaced as a finding rather than allowed through, and in the parallel fix race any candidate that took that route is disqualified outright rather than merely marked down. The same rule applies to visual baselines: a failing screenshot comparison is never resolved by accepting the new screenshot, because that is the identical move wearing a different hat.

A related rule: a change claiming to be behaviour-neutral must pass the suite without any test being modified at all. If a test had to change, it was not behaviour-neutral, and the claim was the thing that was wrong.

#Skipped is a verdict

The last piece is honesty about coverage. When a project has no interface to drive, or no driver configured, the verification step returns SKIPPED — not PASS.

This looks like a small thing and it is not. A fabricated pass is worse than a missing check, because a missing check is visible and a fabricated one is reassuring. Every gate that reports PASS on work it did not do trains the people reading the output to stop reading it, and once that has happened the whole panel is decoration.

The same principle runs through how the numbers are reported. Where nothing measured a figure, it renders as a dash rather than a zero — a panel of confident zeroes over unmeasured runs presents an outage as a healthy, empty system. Where a figure was inferred rather than recorded, it says so beside itself, because a guess printed next to a measurement reads as a measurement.

#What it costs to set up

Less than the first flaky end-to-end test costs to debug. The requirements are narrow:

1. One seam for model calls. Every call goes through a single function that a scenario can replace. If calls are scattered through the flow, this is the only real work in the list, and it is worth doing regardless.
2. Flow topology as data. The path has to be inspectable without executing it — a script or a declared graph, not control flow smeared across a dozen conditionals. This is the same property that makes a system drawable.
3. State written through one writer per file. Otherwise the assertions have nothing stable to read, and you end up asserting on log output, which is prose and changes.
4. Scenarios in version control beside the flows. They are the executable part of the specification, and they rot immediately if they live anywhere else.

Everything after that is writing scenarios, which is cheap enough that you write one for each bug rather than promising to.

#The part this does not cover

Control-flow verification tells you the machinery is sound. It says nothing about whether the reviewer catches real defects, whether the specification was any good, or whether the output is correct — those need real runs against a golden set, a judge validated against human labels, and a regression gate. That work is genuinely expensive and there is no way around it.

What this buys is that the expensive suite stops being where you discover that a gate was never wired. Those failures get caught in two seconds, offline, by something you can run on every commit without thinking about it — which is the only kind of check that survives contact with a deadline.

More on the surrounding disciplines in agentic engineering: harness, loop and graph, and on what happens when a budget runs out in what a bounded loop actually costs.