What happened
- On October 22, 2026, Microsoft and Hugging Face published ThinkingBox-Training on the Hugging Face blog, a method for training agents with rewards drawn from the final state of a database.
- The benchmark behind it, ThinkingBox-Bench, has 507 tasks. The agent gets a real backend and MCP tools, does its work, and then the database is checked for what actually changed.
- The team used those checks as rewards on tasks outside the benchmark and fine-tuned Qwen3.8-27B with 50 GRPO steps. The code is published on GitHub.
- Average success went from 51.70% to 60.78%. Tasks solved in all 20 attempts rose from 38 to 107, while tasks solved at least once fell from 453 to 446.
Why it matters
- An agent that says “done” without finishing is one of the central problems of AI automation. Measuring the outcome in the database, not in the answer, is a way of demanding proof.
- For companies testing agents on bookings or payments, the blog’s example is concrete: before fine-tuning, the agent changed a booking without the required approval in 15 of 20 attempts; afterwards, in none.
- The gain in repeatability had a cost: seven tasks that used to succeed at least once stopped succeeding.
The number
From 38 to 107. Tasks the agent solved in all 20 attempts, before and after fine-tuning, out of 507.
Context
The method picks practice tasks according to how hard they are for the current model, using a fixed estimate made before training. Tasks the model always solves, or never solves, teach it nothing, the authors explain.
What’s next
- The training code is available for other teams to replicate.
- No deadlines announced for new models or tasks.
Bottom line
In September, a study found that agents declare themselves done up to 38 points more often than the grader approves. And in email, agents execute 99.7% of their calls but solve 33.5% of the tasks.
Sources
- Hugging Face Blog — Turning Database Grading into RL Rewards: Post-Training Agents with ThinkingBox (October 22, 2026)
- Cover image: Aaron Hall, CC BY-SA 2.0, via Wikimedia Commons
Edited by Rodrigo Cornejo. How we select and verify: who writes these notes.
