Skip to content
Vision Arena

All Benchmarks

Video-MME v2

A benchmark that scores video understanding through grouped questions, factoring in answer consistency and reasoning coherence.

Project PagePaperGitHubData Source

Background

Despite advancements in model capability and saturation at the top of many benchmarks, practical user experience with these models has not matched leaderboard performance. This gap suggests existing evaluation methods are not fully capturing real video understanding ability and are instead overfitting to these benchmarks. The first Video-MME was one of the benchmarks affected by this, motivating the need for a more comprehensive vision benchmark.

Methodology

Video-MME v2 is built around two guiding questions: what video understanding should evaluate, and how to evaluate it sufficiently and reliably. Questions are grouped in sets of 4 per video to assess capability consistency, reasoning coherence, and temporal understanding. As a result, scores reflect performance across multiple questions per video, providing an evaluation closer to real-world video understanding.

Dataset Structure

800 videos with 4 questions per video and 8 answer options per question.

Benchmark Measured Capabilities

Task Examples

  • Question

    How many arrows did each of the two participants shoot in the video?

    Question type
    Multiple choice
    Options
    • A.5.
    • B.7.
    • C.6.
    • D.8.
    • E.4.
    • F.9.
    • G.10.
    • H.12.
    Correct answer
    C. 6.
    Capability
    Perception
  • Question

    What impact did missing that technical free throw have on the tactics of both sides for the final possession?

    Question type
    Multiple choice
    Options
    • A.The offensive team needs to complete the attack quickly to fight for an extra offensive possession; the defensive team's strategy remains unchanged.
    • B.The offensive team needs to actively complete the last attack; the defensive team's strategy remains unchanged.
    • C.The offensive team needs to make a three-pointer to overcome the deficit; the defensive team's strategy remains unchanged.
    • D.Neither the offensive nor defensive strategies are affected, as the game has entered garbage time.
    • E.The offensive team's strategy remains unchanged, needing only to run out the clock to win; the defensive team needs to foul quickly to gain an extra offensive possession.
    • F.The offensive team can choose to play conservatively to force overtime, or take some risk to attack actively; the defensive team's strategy remains unchanged.
    • G.The offensive team can only choose to shoot a three-pointer on the final possession; the defensive team only needs to focus on defending beyond the three-point line.
    • H.The offensive team's strategy remains unchanged; the defensive team only needs to focus on defending the paint.
    Correct answer
    B. The offensive team needs to actively complete the last attack; the defensive team's strategy remains unchanged.
    Capability
    Causal Reasoning

@article{fu2026videommev2,
title   = {Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding},
author  = {Chaoyou Fu and Haozhi Yuan and Yuhao Dong and Yi-Fan Zhang and Yunhang Shen and Xiaoxing Hu and Xueying Li and Jinsen Su and Chengwu Long and Xiaoyao Xie and Yongkang Xie and Xiawu Zheng and Xue Yang and Haoyu Cao and Yunsheng Wu and Ziwei Liu and Xing Sun and Caifeng Shan and Ran He},
journal = {arXiv preprint arXiv:2604.05015},
year    = {2026},
url     = {https://arxiv.org/abs/2604.05015},
}