Evaluation benchmarks for video understanding models. These benchmarks are used to evaluate the performance of video understanding models on a variety of capabilities, including long context, temporal reasoning, perception, retrieval, and more.
Perception, Spatial Reasoning, Temporal Reasoning, Action / Event Understanding, Causal Reasoning, Information Retrieval
A benchmark that scores video understanding through grouped questions, factoring in answer consistency and reasoning coherence.
Perception, Spatial Reasoning, Temporal Reasoning, Action / Event Understanding, Causal Reasoning, Information Retrieval
A benchmark tailored for comprehensive long video understanding on hour-plus videos across sports, documentaries, events, TV shows, and more.
Perception, Spatial Reasoning, Temporal Reasoning, Action / Event Understanding, Causal Reasoning, Information Retrieval
A benchmark by DeepMind for low- and mid-level visual perception, covering memory, physics, and semantics across video, audio, and text.
Perception, Spatial Reasoning, Temporal Reasoning, Action / Event Understanding, Causal Reasoning, Information Retrieval
A video question-answering benchmark for causal and temporal reasoning about object interactions in everyday activities.
Perception, Spatial Reasoning, Temporal Reasoning, Action / Event Understanding, Causal Reasoning, Information Retrieval
A benchmark specifically designed to evaluate LMMs' proficiency in discerning video quality.
Perception, Spatial Reasoning, Temporal Reasoning, Action / Event Understanding, Causal Reasoning, Information Retrieval
A benchmark for video understanding using temporal certificate sets to measure the actual reasoning length a task requires.
Action / Event Understanding
A benchmark built to evaluate MLLM "capacity for denial," a measurement of model agreeableness and the tendency to hallucinate actions occurring.
These are the public video-understanding benchmarks worth aggregating.
Public benchmarks saturate, get trained on, and are written to isolate a capability rather than to resemble anyone’s work. A high score in these benchmarks does not signal real-world capability, but rather that the model can do the thing the benchmark measures.
Closing that gap needs tasks drawn from real work, held closed, and scored against what a domain expert would have written. Drafts below; none are scored yet.
Watch ROV survey video of a subsea pipeline and fill in the inspection report: corrosion level on the anodes, freespan and vibration, any UXO, or hazards that may compromise the pipeline. Whether it warrants a scheduled repair.
Given a story brief, search across archival & the incoming feed for every shot that could relate to the segment, return results a producer can build the highlight reel from.
Watch the commercial vessel traffic (not the pleasure craft) for suspicious vessels and anomalous behaviour, such as repetitive rendezvous or behaviour consistent with illegal fishing.
Watch a night of camera feeds from around the facility, and rank the events worth a guard’s attention from the four hundred that aren’t, describing each the way an incident log would.
Review dashcam footage and flag the moments a claims adjuster would call dangerous or aggressive driving, such as tailgating, driver on their phone, or running red lights.
Detect and count every bird crossing the frame, and then identify the species across buoy cameras & turbine mounted cameras.