Methodology

Vision Arena exists because the published numbers stopped predicting the work. A model can sit near the top of every video benchmark and still fail the first real task it is handed: find the moment the load shifted, count how many crates left the bay, say whether the person who came back is the person who left.

Math-in-an-image is a useful probe of diagram parsing. It is not a substitute for video intelligence. The hard problem is keeping a world coherent while it moves, then retrieving the moment that answers the question.

So this site does two things. It collects published results under a protocol strict enough that a comparison means something, and it says where every figure came from. Where a number cannot be sourced, the cell stays empty — it is never filled with a zero, and never averaged away.

The most common use case of Vision Language Models (VLMs) is for answering queries or extracting information from images. The other use case is for video analysis, but analyzing video is an extremely complex and compute-intensive task.

Over the years, frontier labs have released benchmarks to evaluate the ability of these vision models to understand video. However, despite improvements in benchmark scores, a clear disparity between practical results and evaluation performance has grown, indicating that the available benchmarks do not effectively replicate real video analysis workflows.

To come anywhere close to human-level analysis, models must possess the baseline set of capabilities outlined here.

Video understanding benchmarks typically assess a handful of these capabilities. Real vision tasks require all of them at once.

Vision Arena assesses models across a range of benchmarks to find which of them are capable of true video understanding rather than of one column of it.

"Visual question answering" is a format, not a skill. The same VQA wrapper can ask for a count, an identity, a spatial relation, a causal order, or a timestamp. Collapsing those into one accuracy hides the actual puzzle. We separate tasks along a few axes.

Identification

What is present, who is involved, which instance the query refers to.

Counting

How many, without double-counting across time or viewpoint. Traditional, and still unsolved once the camera moves.

Spatial reasoning

Where things are relative to each other, and whether that layout still holds after the camera or the object moves.

Temporal reasoning

Order, duration, frequency, causality. The answer is a relation in time, not an object name.

Long video understanding

The evidence is not in the last few seconds. Identity and state have to survive minutes to an hour.

Moment / timestamp retrieval

The output is a location in the tape. A correct paragraph about the clip with the wrong second is still a miss.

Traditional tasks are not "easy" and should not be treated as solved warm-ups. They are the perceptual primitives the temporal and spatial puzzles sit on. A model that cannot count returns, or cannot tell two similar objects apart, will look clever on a language exam and fail in footage.

The Visual Intelligence Index is the headline number on this site: one score per model, across every public video benchmark we track. It is computed, not judged.

What goes in

The six capability indexes, not the benchmarks directly. Each of those is a weighted mean over the suites in the directory — currently 7 — using the official split and the metric the benchmark names as primary.

How it combines

0.20 Perception + 0.15 Spatial + 0.20 Temporal + 0.15 Action + 0.20 Causal + 0.10 Information Retrieval. Those six numbers are a judgement, not a derivation: reasoning about what happened and why is weighted above locating a moment. Capabilities a model has no results for are omitted and the rest renormalised, so a gap lowers confidence rather than the score.

How a capability combines

A weighted mean across every suite with items tagged for it, where a suite’s weight is √centrality × n/(n+K) — how central the capability is to that suite, damped by how many items back it up. A suite still contributes its whole score, scaled: see the limits below.

Who gets ranked

A model needs scores on at least 3 of the six capabilities. Below that it still appears, but unranked — one spectacular number and four blanks is not a standing.

Version

v1-provisional. The weighting is provisional. When a weighting lands the version changes with it, so scores published before and after stay distinguishable.

This is a crude index, not a latent skill estimate. It is a summary of a table, and the table is the real artifact.

That is what the capability indexes are for. Long video, moment retrieval and identification are scored separately — filter the leaderboard by one and the ranks recompute for it, so a strength in one column cannot paper over a hole in another.

A useful video benchmark isolates a failure mode. Long-context suites ask whether evidence separated by minutes can still be combined. Temporal suites ask whether order, frequency and magnitude survive fluent language. Retrieval suites ask whether the model can return a timestamp, not a paragraph. Broad perception suites exist to stop the table collapsing onto one dataset's quirks.

We include a suite when it is well constructed, commonly cited for video understanding, large enough for a difference between models to mean something, and adds a length or a capability the rest of the set does not already cover.

A benchmark stops being helpful when it can be gamed by the input setting — more frames, subtitles, an extra retrieval tool — while the leaderboard presents those runs as comparable. Every published number here keeps its setting next to it.

See the benchmark directory for what each suite measures, and how each one is grouped into a capability index.

Evaluation is a protocol, not a screenshot of someone else's table. The comparable question is: same split, same metric, same input budget, no hidden transcript, no extra retrieval tool unless the column says so. If one paper reports 256 frames and another reports 1 fps, those are different tests. We record the setting and refuse to average it away.

• Prefer the official split and the metric the benchmark names as primary.
• Prefer frames-only, no-subtitle settings where the suite publishes them, so language shortcuts do not masquerade as seeing.
• Never mix closed-ended accuracy, open-ended LLM judges and retrieval R@1 into one unlabeled average. Indexes group related metrics; they do not pretend they share a unit.
• Link every score to a source page. If we cannot point at the number, it does not enter the table.
• Treat sample items as illustrations of a skill, never as a private test set models were trained against.

This snapshot is a working dataset, not a finished measurement, and the site is built to show that rather than hide it.

Most cells are not sourced yet

0 of 61 benchmark scores in this snapshot are placeholders — invented to exercise the interface and labelled as such on every model page. Only figures marked with a source are published results, and nothing marked placeholder may be cited.

We have not run these ourselves

Scores here are collected from published evaluations. Where we later run a benchmark in our own harness, the finding goes in that benchmark's documentation next to the published number rather than replacing it silently.

Latency and completion times are estimates

Sample completion times are interface estimates, not measured claims, and are marked as sample wherever they appear.

• Introducing Vision Arena

  Why build an arena?

  OnDeck AI, Justin Chan · September 10, 2026