SENSE multilingual semantic speech embeddings

This repository provides a pretrained SENSE model for extracting multilingual semantic speech embeddings with SpeechBrain.

The model was trained on 90 Common Voice languages following the SENSE framework described in:

SENSE models: an open source solution for multilingual and multimodal semantic-based tasks
https://arxiv.org/abs/2509.12093

Pipeline description

SENSE produces utterance-level semantic speech embeddings from 16 kHz audio.

Install SpeechBrain

pip install speechbrain

Compute SENSE embeddings

from speechbrain.inference.interfaces import foreign_class

sense = foreign_class(
    source="LIA-AvignonUniversity/SENSE",
    pymodule_file="custom.py",
    classname="SENSEEncoder",
)

embedding = sense.encode_file("audio.wav")

print(embedding.shape)
# torch.Size([1, 1024])

Inference on GPU

from speechbrain.inference.interfaces import foreign_class

sense = foreign_class(
    source="LIA-AvignonUniversity/SENSE",
    pymodule_file="custom.py",
    classname="SENSEEncoder",
    run_opts={"device": "cuda:0"},
)

embedding = sense.encode_file("audio.wav")

Training

The model was trained with SpeechBrain on multilingual Common Voice data.

The complete training recipe is available here:

https://github.com/speechbrain/speechbrain/tree/develop/recipes/CommonVoice/SENSE

Referencing SENSE

If you use this model, please cite:

@article{mdhaffar2025sense,
  title={SENSE models: an open source solution for multilingual and multimodal semantic-based tasks},
  author={Mdhaffar, Salima and Elleuch, Haroun and Chellaf, Chaimae and Nguyen, Ha and Est{\`e}ve, Yannick},
  journal={arXiv preprint arXiv:2509.12093},
  year={2025}
}

Citing SpeechBrain

Please cite SpeechBrain if you use it in your research:

@misc{speechbrainV1,
  title={Open-Source Conversational AI with SpeechBrain 1.0},
  author={Mirco Ravanelli and Titouan Parcollet and Adel Moumen and Sylvain de Langen and Cem Subakan and Peter Plantinga and Yingzhi Wang and Pooneh Mousavi and Luca Della Libera and Artem Ploujnikov and Francesco Paissan and Davide Borra and Salah Zaiem and Zeyu Zhao and Shucong Zhang and Georgios Karakasidis and Sung-Lin Yeh and Pierre Champion and Aku Rouhe and Rudolf Braun and Florian Mai and Juan Zuluaga-Gomez and Seyed Mahed Mousavi and Andreas Nautsch and Xuechen Liu and Sangeet Sagar and Jarod Duret and Salima Mdhaffar and Gaelle Laperriere and Mickael Rouvier and Renato De Mori and Yannick Esteve},
  year={2024},
  eprint={2407.00463},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  url={https://arxiv.org/abs/2407.00463}
}

About SpeechBrain

• Website: https://speechbrain.github.io/
• Code: https://github.com/speechbrain/speechbrain/
• Hugging Face: https://huggingface.co/speechbrain/