Skip to content
Vision Arena

All Benchmarks

LVBench

A benchmark tailored for comprehensive long video understanding on hour-plus videos across sports, documentaries, events, TV shows, and more.

Project PagePaperGitHubData Source

Background

Multimodal large language models have gotten much better at understanding short videos (<1m), but real-world applications like embodied intelligence for long-term decision-making, in-depth movie discussion, and live sports commentary all require understanding videos that span hours of footage.

Methodology

LVBench measures on six core temporal understanding capabilities that thoroughly test a model's ability to process and reason over extended temporal sequences. Videos are drawn from a range of domains and average about four times longer than those in existing benchmarks, giving a broader foundation for evaluating temporal understanding across contexts.

Dataset Structure

1549 unique questions that draw on 103 high-quality public videos, totaling 117 hours of content. Each video has an average length of 67.3 minutes, giving LVBench the longest average per-video length of any video benchmark.

Benchmark Measured Capabilities

Task Examples

  • Question

    What are the key events or actions that occur from 01:17 - 05:07?

    Question type
    Multiple choice
    Options
    • A.The photographer has recorded the magnificent marine landscape. While seated in a pneumatic boat and diving, he captured scenes of pods of sperm whales active in the area.
    • B.The photographer captured scenes of pods of orca active in the marine area from aboard a cruise ship.
    • C.Many sperm whales are active in this sea region, and the photographer, seated in a helicopter, pursues them to film and document their behavior.
    • D.The documentary captures the terrain and weather conditions of the filming location, where humans are fishing in the marine area. Numerous orca are active in this sea region, and the photographer, seated in a pneumatic boat, dives to pursue and film their behavior.
    Correct answer
    D. The documentary captures the terrain and weather conditions of the filming location, where humans are fishing in the marine area. Numerous orca are active in this sea region, and the photographer, seated in a pneumatic boat, dives to pursue and film their behavior.
    Capability
    Information Retrieval
  • Question

    Why did player number 4 in white push down player number 17 in purple during the match?

    Question type
    Multiple choice
    Options
    • A.Player number 4 in white was attempting to maneuver past player number 17 in purple during an offensive play.
    • B.Player number 4 in white was retaliating for an earlier foul by player number 17 in purple.
    • C.Player number 4 in white was making a defensive play to prevent player number 17 in purple from successfully dribbling past him.
    • D.It was an accidental collision as both players were competing for the ball.
    Correct answer
    C. Player number 4 in white was making a defensive play to prevent player number 17 in purple from successfully dribbling past him.
    Capability
    Causal Reasoning

@article{wang2024lvbench,
title   = {LVBench: An Extreme Long Video Understanding Benchmark},
author  = {Weihan Wang and Zehai He and Wenyi Hong and Yean Cheng and Xiaohan Zhang and Ji Qi and Xiaotao Gu and Shiyu Huang and Bin Xu and Yuxiao Dong and Ming Ding and Jie Tang},
journal = {arXiv preprint arXiv:2406.08035},
year    = {2024},
url     = {https://arxiv.org/abs/2406.08035},
}