Skip to content
Vision Arena

All Benchmarks

Q-Bench Video

A benchmark specifically designed to evaluate LMMs' proficiency in discerning video quality.

GitHubPaperData Source

Background

Many studies into LMMs for video understanding have emphasized general video comprehension capabilities. However, few benchmarks have effectively targeted understanding of video quality, which is key for optimizing compression, improving viewer experience, and establishing standards for high-quality video generation.

Methodology

To ensure the diversity of video sources, Q-Bench Video encompasses videos from natural scenes, computer graphics (CG), and AI-generated content (AIGC). Open-ended questions are included alongside more structured formats to better evaluate complex scenarios.

Dataset Structure

1800 videos from 7 source datasets - 1000 natural videos, 600 AI generated content (AIGC) videos, and 200 computer graphics (CG) videos. 2378 question-answer pairs were annotated, including yes-no questions, what-how questions, and open-ended.

Benchmark Measured Capabilities

Task Examples

  • Question

    What are the most severe artifacts in this video?

    Question type
    Multiple choice
    Options
    • A.Overexposure
    • B.Low brightness
    • C.Severe tearing and blurring
    • D.Insufficient color saturation
    Correct answer
    C. Severe tearing and blurring
    Capability
    Perception
  • Question

    How is the clarity of this video compared to the second video?

    Question type
    Multiple choice
    Options
    • A.A little lower
    • B.Much lower
    • C.Much higher
    • D.A little higher
    Correct answer
    C. Much higher
    Capability
    Perception
  • Question

    How does the clarity of this video change over time?

    Question type
    Multiple choice
    Options
    • A.Always relatively poor
    • B.From poor to good
    • C.From good to poor
    • D.Always relatively good
    Correct answer
    A. Always relatively poor
    Capability
    Perception

@article{zhang2024qbenchvideo,
title   = {Q-Bench-Video: Benchmarking the Video Quality Understanding of LMMs},
author  = {Zicheng Zhang and Ziheng Jia and Haoning Wu and Chunyi Li and Zijian Chen and Yingjie Zhou and Wei Sun and Xiaohong Liu and Xiongkuo Min and Weisi Lin and Guangtao Zhai},
journal = {arXiv preprint arXiv:2409.20063},
year    = {2024},
url     = {https://arxiv.org/abs/2409.20063},
}