VibeVideoBench: New benchmark for multimodal agents in code-driven video generation
In brief
- VibeVideoBench tests multimodal agents on 239 code-driven video generation tasks.
- Agents must deliver an MP4 with audio within one hour, without image/video models.
- GPT-6 Astra and Claude Opus 5.5 rank first, but 23.7% of tasks fail due to rendering issues.
Video diffusion models produce impressive clips, but struggle with precise instructions like exact formulas or synchronized narration. A new approach is emerging: multimodal agents that write code to render every frame and every sound.
VibeVideoBench introduces a benchmark of 239 self-contained tasks in English and Chinese across 12 categories (data stories, classroom lessons, etc.). Each agent must deliver an MP4 file with audio within one hour, without relying on image or video generation models. Evaluation combines program checks, 13,000 task-specific rubric checks, and five quality dimensions scored by three VLM judge panels. Across 2,151 rollouts, GPT-6 Astra and Claude Opus 5.5 lead, but 23.7% of tasks fail due to missing final renders, often caused by exceeded computation time.
What it changes for you
- Media & EntertainmentContent creators could leverage these agents to automate complex video production, but render reliability remains a challenge.
- EducationEducational video tools could be generated on demand with precise instructions, but synchronization or rendering errors limit their use.
This summary gives you the essentials. The full article, its figures and quotes are on Hugging Face Daily Papers.
Read the original articleAI-generated summary of the source, checked by rules. How we write · Report an error