Skip to content
Vision Arena

All Benchmarks

EgoSchema

A benchmark for video understanding using temporal certificate sets to measure the actual reasoning length a task requires.

Project PagePaper

Background

Understanding long natural videos requires a range of interconnected abilities, including action and scene understanding, tracking object states, long-term visual memory, abstract reasoning, and more. EgoSchema is a diagnostic benchmark for assessing long-form understanding of video in modern multimodal systems. EgoSchema's videos are set at 3 minutes each which is short compared to other long-form benchmarks, but EgoSchema claims to measure long-form understanding by evaluating true temporal difficulty rather than raw video length.

Methodology

For each question, EgoSchema requires selecting the correct answer from five options based on a three-minute video clip. To capture how temporally difficult a video task actually is, EgoSchema introduces temporal certificate sets, defined as the minimum span of video a human verifier needs to watch to confirm a marked answer is correct (~100s).

Dataset Structure

EgoSchema consists of over 5,000 human-curated multiple-choice question-answer pairs, spanning more than 250 hours of real video footage covering a broad range of natural human activity and behavior.

Benchmark Measured Capabilities

Task Examples

  • Question

    In the video, identify the key tools and machines that C interacts with and how they contribute to the overall process.

    Question type
    Open answer
    Correct answer
    Dough mixer for combining ingredients, dough sheeter to flatten dough, and adjustable rolling pin for precise shaping.
    Capability
    Information Retrieval
  • Question

    Based on the video, identify the three most crucial moments in the construction process and explain their significance in achieving the desired outcome.

    Question type
    Open answer
    Correct answer
    Measuring lumber, drilling, and straightening deck rail cable.
    Capability
    Information Retrieval

@article{mangalam2023egoschema,
title   = {EgoSchema: A Diagnostic Benchmark for Video Language Understanding},
author  = {Karttikeya Mangalam and Raiymbek Akshulakov and Jitendra Malik},
journal = {arXiv preprint arXiv:2308.09126},
year    = {2023},
url     = {https://arxiv.org/abs/2308.09126},
}