2026-10-20 — views
OpenVLHarness Boosts MLLM Performance via Multimodal Memory and Tool Use
Read this because This framework solves the limitation of internal reasoning alone by grounding models in external evidence, offering a significant performance boost with lower token costs.
OpenVLHarness improves multimodal model scores by up to 12.6 points using persistent memory and heterogeneous tool integration.
The Limits of Internal Reasoning
Multimodal large language models (MLLMs) have demonstrated rapid progress in the field of visual reasoning. However, a controlled study conducted across 22 benchmarks reveals a critical bottleneck: many tasks require evidence that cannot be obtained through internal reasoning alone. The research indicates that simply increasing the inference budget of the base model yields only modest gains in performance. To address these complex tasks, systems need grounded perception, access to external knowledge, and the ability to perform numerical measurements using heterogeneous tools. Crucially, the outputs from these tools must be interpretable to the model and retained across various reasoning steps.
Introducing OpenVLHarness
To solve these challenges, researchers have introduced OpenVLHarness, a multimodal reasoning harness designed around two core components: a capability layer and persistent multimodal memory. The capability layer integrates task-level tool interfaces with evidence rendering. This structure provides consistent access to heterogeneous backends and converts their outputs into observations that the model can readily read. Simultaneously, the persistent multimodal memory retains corresponding images, structured data, and web references as addressable artifacts throughout each query. This ensures that evidence is not lost between steps.
Mechanisms and Evaluation
A key feature of OpenVLHarness is its coding interface, which executes generated programs over stored data. This enables computations that combine results from multiple tools, supporting an iterative process of acquiring, inspecting, and composing multimodal evidence. The system was evaluated on 22 benchmarks using eight open and proprietary backbones. The results showed that OpenVLHarness improves average scores by 9.7 to 12.6 points over matched no-tool baselines. Specific improvements included an 11.2-point increase with Qwen3-VL-8B, a 9.7-point increase with Kimi K3, and a 9.9-point increase with GPT-6 Astra.
Furthermore, when using GPT-6 Luna at medium reasoning effort, OpenVLHarness outperformed the base model, Codex, and Visual Sketchpad at maximum effort by 7.7 to 8.5 points. Notably, this superior performance was achieved while incurring at most 37% of the model-token cost required by the competing methods. This demonstrates that the harness not only enhances accuracy but also improves efficiency by reducing the computational overhead associated with high-effort reasoning.
Practitioner note
Developers should integrate OpenVLHarness to enable their models to access external tools and retain multimodal context, which significantly boosts performance on complex visual tasks. Watch for opportunities to leverage the persistent memory layer to reduce token costs while maintaining or exceeding the accuracy of high-effort baseline models.