2026-10-09 — views
FastBench Evaluates High-Dynamic Perception in Streaming VLMs
Read this because Streaming VLMs often fail in high-dynamic scenarios. FastBench exposes these limitations through rigorous trajectory-grounded evaluation, guiding future model development.
FastBench introduces a benchmark for high-dynamic video streams, revealing that current VLMs struggle with fast events and temporal granularity.
The Challenge of High-Dynamic Streaming
Streaming Video Large Language Models (VLMs) are designed to enable continuous video understanding. However, existing benchmarks primarily focus on low-dynamic scenarios, which do not reflect the complexities of real-world video streams. In high-dynamic environments, models face a critical trade-off under bounded context budgets: they must balance temporal history, spatial resolution, and temporal granularity. Sparse sampling rates of one to two frames per second (FPS) frequently miss fast-moving events, leading to significant perception errors. To address this gap, researchers have introduced FastBench, a new benchmark specifically designed to evaluate high-dynamic perception in real-world video streams.
Methodology and Dataset Construction
FastBench employs a trajectory-grounded pipeline to ensure the quality and relevance of its evaluation data. The process begins with question generation derived from high-FPS clips. These questions are then filtered to identify those that are answerable at a lower 2 FPS rate, ensuring a fair comparison of temporal perception capabilities. Answer verification is conducted using SAM3 and CoTracker3 trajectories to ground the correctness of responses. The dataset undergoes three rounds of human inspection to maintain high standards. The resulting FastBench dataset contains 306 question-answer pairs spanning eight domains and six capabilities. It covers forward, instant, and backward temporal scopes and includes human-annotated evidence intervals to support rigorous analysis.
Performance of Current Models
The study also presents ProactiveFrame, a training-free baseline that adjusts incoming frame rates through text tokens. This method uses a dual-tier sliding window to retain recent high-FPS observations while downsampling older ones into sparse history. Experiments on FastBench reveal substantial limitations in current VLM capabilities. The strongest model evaluated, Gemini-3.5-Flash, achieved a score of only 50.7%. Denser sampling improves performance; for instance, Qwen3-VL-8B increased its score from 32.9% at 2 FPS to 44.6% at 24 FPS. However, these gains saturate as the temporal history is compressed. ProactiveFrame outperforms sparse uniform sampling by 5.4 and 1.5 percentage points but remains well below oracle-guided focusing. This indicates that current VLMs struggle to determine from the stream alone when finer temporal perception is needed.
Practitioner note
Developers building streaming video applications should recognize that standard sparse sampling is insufficient for high-dynamic content. Implementing adaptive frame rate strategies, such as those explored in ProactiveFrame, may offer marginal improvements, but significant gains require models that can better identify when high temporal resolution is critical.