Eval harness for agent tool-calling
Caught 23 tool-calling regressions before release across a year of weekly deploys.
- Regressions caught
- 23Pre-release, over 52 weekly deploys
- Eval suite runtime
- 41 min → 6 minAfter sharding and response caching
- Trace coverage
- 94%
- Role
- Lead engineer, team of 3
- Period
- 2024–2025
- Stack
- Python
- pytest
- OpenTelemetry
- DuckDB
The problem
We shipped an agent that called eleven internal tools, and we shipped it weekly. Every release, something in the tool-calling behaviour would shift — an argument silently coerced to the wrong type, a retry loop that had been three attempts becoming unbounded, a tool the model had reliably picked for two months quietly falling out of favour after a prompt edit.
None of it broke a test. The unit tests mocked the model, so they tested our plumbing and nothing about the behaviour that actually varied. We found these regressions in production, from support tickets, usually four to six days after the deploy that caused them.
Approach
The core decision was to treat a tool call as the unit of assertion rather
than the final response. Grading the text an agent produces is expensive,
noisy, and requires a judge model that itself drifts. Grading whether it called
search_invoices with a date range and not list_customers is cheap,
deterministic, and catches the failures that actually reach users.
Every run emits OpenTelemetry spans, one per tool call, and the harness asserts over the resulting trace tree: which tools were called, in what order, with what argument shapes, and how many times. Assertions are written as ordinary pytest functions, because the team already knew pytest and I did not want a DSL that only I could maintain.
I rejected LLM-as-judge for the primary suite. It is in there, but only as a secondary tier on the 40 cases where the final prose genuinely matters. On the other 300, it added cost and variance without catching anything the trace assertions missed.
The runtime problem was solved by caching model responses keyed on the full request, so a rerun after an unrelated code change replays instead of calling out. That is what took the suite from 41 minutes to 6 and is the reason people actually run it locally.
Results
23 regressions caught pre-release over 52 weekly deploys. Each one is a real entry in the CI log where a previously passing trace assertion failed and the deploy was held. That is the number I trust most in this write-up because it is just a count of events, not an estimate.
The runtime figure is a median over the last 100 CI runs. A cold run with no cache is still around 30 minutes, and that is what the nightly does.
Trace coverage — 94% — is the share of tool-call paths exercised by at least one case, measured by instrumenting the tool registry. The missing 6% is two admin tools nobody could construct a realistic scenario for.
What I’d do differently
The cache key includes the full prompt, so any prompt edit invalidates everything and the next run costs 41 minutes. A structural key that ignored formatting-only changes would have saved the team a great deal of waiting, and I never got around to building it.
I would also have made the harness emit a diff of trace trees rather than a pytest assertion error. When a case failed, the message told you an assertion was false, and then you spent ten minutes reading spans to find out what the agent had done instead. That is a small piece of work with an outsized effect on whether people trust a failing test or just rerun it.