Skip to content
Vision Arena

All Benchmarks

NExT-QA

A video question-answering benchmark for causal and temporal reasoning about object interactions in everyday activities.

Project PagePaperGitHub

Background

Events in video are often sequential and dependent on a preceding event. Modern neural networks can already recognize objects and describe individual actions in a video, but reasoning about how those actions relate causally and temporally, and answering natural language questions like "Why did the car swerve?", remains a major challenge.

Methodology

NExT-QA defines three question categories: causal questions that ask why something happened or how an effect came about, temporal questions that ask about actions before, during, or after a reference point, and descriptive questions covering scene elements like location, objects, and counting. The benchmark sets up two task formats at different difficulty levels: Multi-choice QA gives each question five candidate answers, whereas Open-ended QA is evaluated using a similarity score that credits semantically close answers rather than requiring exact matches. Scores for open-ended evaluation (including for the extended open-ended set) are coming soon.

Dataset Structure

5,440 videos averaging 44 seconds in length, split into training (3,870 videos), validation (570 videos), and testing (1,000 videos). 99,736 questions are split into multi-choice (47,692 QA) and open-ended (52,044 QA), and also categorized as causal (48%), temporal (29%), and descriptive (23%).

Benchmark Measured Capabilities

Task Examples

  • Question

    How did the boy in red try to move his body when he is lying flat on the ground?

    Question type
    Open answer
    Correct answer
    He tries to push himself forward
    Capability
    Action Understanding
  • Question

    How is the fish moving its body?

    Question type
    Open answer
    Correct answer
    The fish moves from left to right
    Capability
    Action Understanding

@inproceedings{xiao2021nextqa,
title     = {NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions},
author    = {Junbin Xiao and Xindi Shang and Angela Yao and Tat-Seng Chua},
booktitle = {CVPR 2021},
year      = {2021},
url       = {https://arxiv.org/abs/2105.08276},
}