2026-09-24 — views
SWE-Flux Benchmark Reveals LLM Struggles with Runtime Reasoning
Read this because LLMs remain weak at runtime behavior. SWE-Flux provides a rigorous, automated benchmark to measure this gap across real Python repositories.
New repository-level benchmark SWE-Flux shows top LLMs achieve only 37% accuracy on dynamic execution reasoning tasks.
The Challenge of Runtime Reasoning
Large language models are increasingly integrated into coding workflows, yet their capacity to reason about code execution remains an open question. While existing repository-level question-answering benchmarks primarily assess static code understanding, they often depend on LLM-based evaluation methods that may not be reliable. Furthermore, benchmarks focused on execution reasoning are typically restricted to small code snippets or individual functions, lacking the complexity of full software projects.
Introducing SWE-Flux
To address these limitations, researchers have introduced SWE-Flux, a new repository-level benchmark designed specifically for dynamic execution reasoning. This benchmark comprises 480 execution-grounded instances distributed across 12 real Python repositories. A key innovation of SWE-Flux is its methodology for generating ground truth. Instead of relying on manual writing or subjective LLM judgments, the gold answers are automatically harvested from instrumented test executions. This approach ensures higher fidelity and objectivity in evaluating model performance.
Benchmark Scope and Evaluation
SWE-Flux covers a wide range of dynamic programming concepts, including single-test and multi-test questions. The tasks evaluate understanding of control flow, loops, program state, dataflow, exceptions, and program invariants. The study evaluated five different large language models on this benchmark to gauge their capabilities. The results indicate that reasoning about runtime behavior remains a significant challenge for current AI systems.
Performance Results
The evaluation revealed that the best-performing model achieved only 37% accuracy. This low score highlights the difficulty of the task. The analysis of model performance showed distinct patterns in success and failure. Models performed relatively better on localized behaviors, such as understanding invariants, intra-procedural control flow, exceptions, and simple loops. However, they struggled significantly with more complex tasks. These difficult areas included dataflow analysis, inter-procedural execution, precise state reasoning, and suite-level aggregation.
Generating Fresh Variants
Beyond evaluation, the study demonstrated the utility of the oracle-harvesting pipeline used to create the benchmark. This pipeline can generate fresh benchmark variants through input perturbation. The researchers found that this method successfully harvested valid variants for almost 90% of the selected instances. Importantly, these newly generated variants were substantially more challenging for the evaluated models, suggesting that the benchmark can be expanded to test model limits further.
Practitioner note
Developers should recognize that current LLMs are not yet reliable for reasoning about complex runtime behaviors or inter-procedural state changes. Relying on these models for code that involves intricate dataflow or suite-level logic requires rigorous manual verification and testing.