Soutenance de thèse : Raphaël Bagat (Multispeech)
Raphaël Bagat, doctorant dans l’équipe Multispeech, soutiendra sa thèse le 25 novembre à 9 heures 30 en salle C005.
Sa présentation sera en anglais.
One of the important limitations in ASR adaptation is the high cost of human supervision. Transcribing speech requires domain expertise. ATC communications present acoustic and linguistic variability, including non-native accents, diverse operational roles, noise, and bandwidth-limited radio channels. Collecting sufficient labeled data to cover all these factors is impractical.
Since different non-native accents exhibit distinct characteristics, pooling them into a single adaptation may fail to capture accent-specific patterns, while maintaining a separate model for each accent does not scale. We propose Mixture of Accent-Specific LoRAs, which combines accent-specific low-rank experts within a shared model. Our results show that this specialization improves multi-accent speech recognition and remains beneficial even when the speaker’s accent is unknown at inference.
While self-supervised learning with untranscribed data is commonly used to pre-train speech encoders before supervised training, exploiting it to adapt an already supervised encoder-decoder model, such as Whisper, is less straightforward because modifying the encoder may disrupt its compatibility with the pre-trained decoder. To address this issue, we propose BEST-RQ Encoder Adaptation with Re-training and Distillation, an adaptation method combining a self-supervised objective with knowledge distillation. BEARD adapts Whisper’s encoder while preserving its compatibility with the pre-trained decoder. BEARD significantly improves supervised adaptation.
The third contribution exploits generative data augmentation. Since the generation of additional speech data is only beneficial when the synthetic data faithfully capture the characteristics of the target domain, it is essential to account for both speaker variability and acoustic conditions. We propose a generative augmentation framework combining text-to-speech, voice conversion, accent conversion and an ATC acoustic simulation. Moreover, we introduce a novel controllable L1-to-L2 accent conversion system. We show that synthetic speech provides the greatest benefits when it reproduces accent diversity and the acoustic characteristics of the target radio channel.
Linguistic knowledge can be exploited when facing acoustic ambiguity. External language models provide a way of introducing an additional linguistic prior during decoding. We show that external LLMs improve recognition on general-domain speech through reranking and shallow fusion. Then, we study their use for ATC speech recognition.
Since ATC communications provide contextual cues such as callsigns and preceding dialogue, this information can help recognize the current utterance. We leverage Whisper’s native prompting mechanism to provide domain-specific information at inference. We show that prompting can successfully exploit contextual information such as callsigns and dialogue history.
Finally, we combine two complementary adaptation strategies. BEARD exploits untranscribed acoustic information during adaptation while prompting exploits contextual information at inference. Their combination allows us to investigate whether these two sources of information are complementary. By combining self-supervised encoder adaptation with contextual prompting, we achieve the best recognition performance on ATC data reported in this thesis.

