2026-10-04 — views
ThinkingBox Benchmarks AI Agent Consistency via Database State
Read this because AI agents often pass tool-call checks but fail to update databases correctly. This benchmark exposes the gap between successful API calls and actual data consistency.
Microsoft and Hugging Face release ThinkingBox, a benchmark grading AI agents on terminal backend state and side effects across 507 workflows.
The Gap Between Tool Calls and Outcomes
A recent joint blog by Microsoft and Hugging Face introduces ThinkingBox, a benchmark that evaluates AI agents based on the records they leave behind rather than the sentences they generate. The system grades agents on terminal backend state and side effects across 507 stateful business workflows. Each task is executed 20 times against various Large Language Model (LLM) models to test consistency.
The authors highlight a critical failure mode where an agent performs nine valid tool calls, correctly checks tracking, and closes a ticket as resolved, yet fails to update the carrier exception status in the database. While an AI grader checking tool calls would see nine well-formed actions, a grader checking the database would see the required end state was never achieved. This discrepancy is what ThinkingBox measures.
Reliability vs. Capability
In a common-set ablation covering 121,680 valid trials across 12 LLM models, 79,853 attempts failed executable checks. Notably, 67.24% of these failures still terminated cleanly, invoked a state-changing tool, and reported no final tool error. Executable checks found wrong field values in 77.61% of these failures, unintended extra effects in 43.30%, and missing required effects in 25.36%.
The benchmark reports three metrics: pass@1, which measures single-attempt success; pass@20, which measures if a task is solved at least once in 20 tries; and Breadth, Observed 20/20, which counts tasks that passed all 20 recorded attempts. This approach reveals that one success does not equal reliability. An agent that processes a refund correctly once but mishandles it four times is not a working agent.
Model Performance and Consistency
Claude Opus 5.5 leads the overall pass@1 score at 67.16%, followed closely by Claude Opus 5 at 66.50%. Among open-weight models, Kimi-K3 is the strongest, achieving 57.37% overall. However, consistency varies wildly. Kimi-K3 solves 93.89% of tasks at least once but only succeeds in all 20 attempts for 13.41% of tasks. In contrast, Claude Opus 5 solves fewer tasks initially but completes 47.53% of the benchmark on every single attempt.
Claude Opus 5.5 scores higher on headline accuracy than Opus 5 but passes the exact same number of tasks (241) on all 20 attempts. This indicates that a half-point gain in accuracy bought no additional dependability. GPT-6 Astra retains 78% of its single-attempt rate across repeats, while models like GLM-5.1 and DeepSeek-V4-Pro retain only about 8%.
Practitioner note
Developers should prioritize the Observed 20/20 metric over single-attempt scores when selecting models for production environments that touch real records. Relying on leaderboards that only publish pass@1 can lead to deploying agents that appear capable but fail consistently in stateful workflows.