2026-09-09 — views
ExecCritic: Separating Test and Repair Agents to Improve Coding Performance
Read this because Separating test generation from code repair prevents error agreement and false confidence, significantly improving agent reliability.
ExecCritic uses separate test and repair agents with reinforcement learning to boost coding agent success rates on SWE-bench Verified.
The Problem with Joint Trajectories
Execution feedback is a powerful tool for guiding coding agents toward correct repository repairs. However, this guidance is only effective when the tests accurately capture the behavior requested by the issue. A significant challenge arises when agent-generated tests encode incomplete or incorrect behavioral targets. When the same trajectory writes both the patch and the test, their errors can agree, creating a dangerous sense of false confidence. This agreement masks underlying issues, leading the agent to believe it has solved the problem when it has not.
The ExecCritic Scaffold
To address this, researchers introduced ExecCritic, a framework that combines a test-verify-revise scaffold with a role-specific reinforcement learning recipe. The core innovation is the separation of test construction from source-code repair. This architecture employs two distinct agents, both using Qwen-3.5-35B-A3B as the backbone, which are trained separately.
The first role is the Test agent, which independently generates repository-native tests. These tests are then passed through a fail-closed harness that qualifies and freezes them. The second role is the Repair agent, which revises the source code based on the execution feedback from the frozen tests. Crucially, the Repair agent does not change the tests, ensuring that the evaluation criteria remain stable during the repair process.
Training and Results
The framework utilizes two distinct training phases. In the Learn to Test phase, the Test agent learns to produce behaviorally valid tests that can effectively distinguish between correct and incorrect patches. In the Test to Improve phase, the Repair agent learns both direct task resolution and feedback-guided revision.
Evaluations on SWE-bench Verified highlight the importance of test quality. When holding the base Repair agent fixed, tests from the base Test agent actually reduced the resolved rate from a no-test baseline of 61.2% to 57.3%. In contrast, tests generated by GPT-5.6-sol raised the rate to 65.3%. This demonstrates that poor test quality can actively harm performance.
Role-specific post-training yielded significant improvements. The post-trained Qwen Test agent raised its Base-to-Gold success rate from 22.2% to 62.2%. When composing the two post-trained Qwen agents, the system reached a 72.6% success rate. This represents an 11.4-point gain over the original no-test baseline, achieved without using stronger models or Oracle feedback at evaluation time. The code for ExecCritic is publicly available on GitHub.
Practitioner note
Developers should consider decoupling test generation from code repair in their agent workflows to prevent error agreement. Investing in high-quality, behaviorally valid tests is critical, as poor tests can actively degrade agent performance rather than improve it.