A support triage suite running its ninety cases after a prompt change
Every workflow we have shipped since 2023 started the same way: someone writes ninety historical cases with known-correct answers, and then we write the thing that has to pass them. It sounds backwards. It is the only reason we can put an agent into a client account and sleep.
Why the test comes first
A prompt written before the tests is a prompt written against imagined inputs. You end up with something that handles the cases you thought of at 11pm on a Tuesday, which are rarely the cases that arrive on a Monday morning.
Writing the suite first forces the argument to happen early, with the client in the room. What counts as correct here? What must never happen? The answers are usually more specific than anyone expected, and they are much cheaper to discover before the build than after.
What goes in the suite
Three groups, in this order:
- The ordinary. Sixty or so cases that represent the bulk of what arrives. If these break, the agent is worthless.
- The edges. Twenty cases that are unusual but real, pulled from the last year of history. These decide where the handoff rule sits.
- The forbidden. Ten cases the agent must refuse. Billing changes, anything touching a legal complaint, anything where being wrong costs more than being slow.
If the suite passes on the first run, it is not a good suite. It means the cases were chosen to flatter the prompt rather than to break it.
If the suite passes on the first run, it is not a good suite.
What it costs
Writing ninety cases takes about three days of one person’s time, most of it spent reading history rather than writing code. Against a six week build that is a rounding error, and it is the part clients keep once the engagement ends.
The one rule
No agent reaches a client account without a failing test written first and a costed baseline beside it. That is the whole method. Everything else is detail.
002
/ Keep reading
From the journal