New method to cut compute costs for diffusion vision-language models
In brief
- LOWN cuts DVLM decoding compute by 64.7 % by processing images only when necessary.
- The method retains 99.6 % of a baseline model's performance across ten benchmarks.
- It integrates with existing techniques like token pruning and key-value caching.
Researchers introduce LOWN, an adaptive decoding method for diffusion vision-language models (DVLMs). According to the announcement, these models process their full input at each denoising step, with 89 to 97 % of tokens being visual on the studied benchmarks. LOWN decides at every step whether to process the image and how many masked tokens to unmask by comparing predictions with and without the image. The method retains 99.6 % of a baseline model's score while reducing decoding compute by 64.7 % on average, without additional training or tuned thresholds. It applies to multiple LLaDA-based DVLMs and composes with visual token pruning techniques.
LOWN adapts its decision based on varying needs across benchmarks, avoiding a fixed schedule. It also works with key-value caching for further efficiency gains. Code and a project page are available online.
This summary gives you the essentials. The full article, its figures and quotes are on Hugging Face Daily Papers.
Read the original articleAI-generated summary of the source, checked by rules. How we write · Report an error