Introducing Vision Arena
Why build an arena?
OnDeck AI, Justin Chan · September 10, 2026
Pretext
Vision Arena is created by OnDeck AI, a visual intelligence research company. We are both an applied AI lab that serves a proprietary vision model and a deployment company working with customers to solve real-world tasks.
This means we inherently need to evaluate vision models and systems a lot. We see a growing need for robust evaluation, but few readily available solutions.
OnDeck's primary goal is to build recursive self-improving mechanisms for vision. That requires a hill to climb: measurable objectives we can repeatedly improve aginst. This hill exists for text and image-based tasks and continues to mature with arenas (Arena, Artificial Analysis, Design Arena) and independent benchmark companies (Andon Labs, Vals, Datacurve). The comparable hill for video understanding tasks does not.
Vision Arena is a first step towards solving this problem. We're aiming to share some of the learnings, practices, and conviction we've gathered from working on hard frontier vision problems with the community.
We hope it will be useful for others in choosing and deploying vision models for real world tasks.
Video is hard
How do we evaluate a model's visual intelligence? Video understanding is an active research area and existing evaluations range from action recognition and temporal grounding to long-video reasoning and question answering, often under very different settings.
The harder question is how well any of this represents visual intelligence in the real world. Humans rarely watch video just to answer a question about it. We watch with a goal, use outside context, decide where to focus, connect evidence across time, and ultimately use what we understand to do something economically valuable.
Evaluating this is difficult, which explains why current evaluation providers largely don't address them. Video is orders of magnitude larger than text and harder to validate. Hallucinations are difficult to detect, while public benchmarks are prone to contamination and language priors.
Reducing visual intelligence to video question answering (often multiple choice) is not representative of real tasks. Real tasks are less constrained: systems need to decide what matters, find the relevant evidence, bring in context, and determine what to do with it efficiently.
What we're building
While public benchmarks won't get us all the way there, they provide useful signal on the core capabilities behind visual intelligence. We aggregate them into capability indexes and leaderboards, informed by what we've learned evaluating models internally. This is where Vision Arena starts today.
Our view is that evaluation should get much closer to the real tasks these systems are used for. That means private benchmarks beyond question answering, from closed real-world tasks to full harnesses that simulate production systems around a vision model. It also means an open arena, where crowdsourced voting can provide an independent signal.
Evaluating the model alone also won't be enough. With agentic methods like our own Perception-0, we'll want to understand the reasoning traces and trajectories of video agents, and expand to world models, VLAs, and other new approaches.