Listen-to-Reason: the tree and the trained heads

Checkpoints of the paper LISTEN-to-Reason: Listen with Experts, Retrieve over a Graph, Reason with LLMs. Code: https://github.com/poonehmousavi/listen-to-reason. Download with python -m l2r.checkpoints, which places these files under the code's checkpoints/ folder.

Folder	Content
tree/	tree.json (speech, music and sound attributes), sound_tree.json (the sound identity classes)
tree_raw/llm_groups.json, tree/sound/	the text LLM's answers used to build the tree
routers/	the eight region routers (small heads on frozen CLAP / MuQ-MuLan / Whisper / emotion2vec / WavLM-SV embeddings), their calibration, and the two domain heads of the paper (birds_k5 on BirdNET, marine_k5 on CLAP)
sound/, events/, music_properties/, question/, language/	the sound-identity heads, the event-timeline head, the music-property heads, the question classifier, the transcript language head
experts/gender_centroids.npz	the gender expert of the annotation

No audio and no dataset content is included; the encoders and LLMs are downloaded from their own hub repositories.

Usage

The files are read by the code in the GitHub repository; python -m l2r.checkpoints places them under its checkpoints/ folder.

git clone https://github.com/poonehmousavi/listen-to-reason.git && cd listen-to-reason
python3.10 -m venv .venv && source .venv/bin/activate
pip install torch==2.12.0 torchaudio==2.11.0 torchvision==0.27.0 --index-url https://download.pytorch.org/whl/cu130
pip install -r requirements.txt && pip install muq==0.1.0 --no-deps

python -m l2r.checkpoints                   # this repository -> checkpoints/
python -m l2r.retrieve                      # route MMAU / MMAR / SAKURA clips into the tree, write the contexts
python -m l2r.reason --reader qwen2.5-7b    # a frozen text LLM answers from the contexts
python -m l2r.retrieve.trace --benchmark mmau --clip MMAU/audio/<id>.wav    # the nodes of one clip, chunk by chunk

The PANNs, BEATs and BirdNET weights are not included; the README of the code lists where to download them and which config keys name their paths. The frozen encoders (CLAP, MuQ-MuLan, Whisper, emotion2vec, WavLM-SV) and the reader LLM are fetched from the hub by id.

One clip, from the command line or in Python: its nodes chunk by chunk, the context the reader receives and, given a question with options, the reader's answer (project page: https://poonehmousavi.github.io/listen-to-reason/).

python -m l2r.retrieve.clip --audio clip.wav
python -m l2r.retrieve.clip --audio clip.wav --question "Which genre is this music?" --options jazz metal hip-hop classical --answer

from l2r.retrieve import clip

r = clip.describe("clip.wav", question="Which genre is this music?", options=["jazz", "metal", "hip-hop", "classical"])
r["context"]      # the text the reader receives
r["trace"]        # every served node with its chunk span and probability, e.g.
                  #   music genre: jazz (whole clip, p 0.85); music instrument: saxophone (1.5-16.5 s, p 0.93); ...
clip.answer(r)    # {"answer": "a", "option": "jazz", "without_context": "b", "model": "Qwen/Qwen2.5-7B-Instruct"}

A region router (one small body on a frozen encoder's chunk embedding, one head per attribute) also loads on its own, and the tree is plain JSON:

import json
from l2r.router import model

router, ck = model.load("sound")          # heads for sound.event, sound.scene, sound.activity, ... on CLAP embeddings
print(ck["encoder"], list(ck["space"]))  # which encoder it reads, which attributes it serves
tree = json.load(open("checkpoints/tree/tree.json"))
print(tree["sound"]["event"].keys())     # classes of an attribute; each holds its leaves

A context the reader receives looks like this:

Audio analysis of the recording:
Audio graph nodes matched to this recording: cross families present: speech present, environmental sound present; speech gender: female voice; sound animal: dog (Bark).
Transcript: "[0:00.0] Would you like some candy?"