ChronoReplay enhances streaming spatial reasoning in VLMs
In brief
- Spatial VLMs suffer from interface lock-in, limiting cross-task generalization.
- ChronoReplay organizes data into chronological episodes and balances task families during training.
- A single model improves three benchmarks at once, including +13.41 points on OST-Bench.
Research reveals that vision-language models (VLMs) trained for spatial tasks like object localization or distance estimation develop interface lock-in: their performance concentrates on one type of response at the expense of others, reducing generalization.
ChronoReplay, a training framework, addresses this by structuring spatial supervision into chronological episodes and balancing learning across task families. A single model trained with this method improves three benchmarks (OVO-S-Bench, OST-Bench, UCS-Bench) simultaneously, including a 13.41-point gain on OST-Bench. Experiments confirm that answer serialization, family balancing, and chronological replay each contribute to these improvements.
The approach highlights the need to preserve complementary spatial information, structure learning around temporal sequences, and efficiently select visual cues during inference for streaming VLMs.
This summary gives you the essentials. The full article, its figures and quotes are on Hugging Face Daily Papers.
Read the original articleAI-generated summary of the source, checked by rules. How we write · Report an error