New benchmark and model for audio-visual target speaker extraction
In brief
- RealSSA, a realistic benchmark for streaming audio-visual target speaker extraction, is released.
- Falcon, a causal method, reduces parameters by 91.9% and GPU latency by 11.1 times versus the baseline.
- Falcon improves extraction quality on four benchmarks while eliminating near-field leakage.
Researchers release RealSSA, a realistic benchmark for streaming audio-visual target speaker extraction (AVTSE). Unlike existing benchmarks, RealSSA includes simulated scenes with 1 to 5 dynamic interfering speakers in the near field and 10 to 15 in the far field, along with a real-recorded evaluation dataset. Current systems are constrained by oversimplified setups and architectures too heavy for edge devices.
To address these gaps, the team introduces Falcon, a causal AVTSE method. Its separator performs multi-scale time-frequency separation in a single encoder-decoder pass. Trained with an encoder-space speaker contrastive loss, Falcon reduces near-field leakage at no extra inference cost. On RealSSA-Sim and three other benchmarks, it outperforms existing models while cutting parameters by 91.9% and compute by 20.2 times compared to a causal baseline.
What it changes for you
- Media & EntertainmentMedia professionals can now deploy lighter and more accurate audio-visual extraction tools for real-time streaming applications.
This summary gives you the essentials. The full article, its figures and quotes are on Hugging Face Daily Papers.
Read the original articleAI-generated summary of the source, checked by rules. How we write · Report an error