Transformers documentation

Qwen3 ASR

Transformers

You are viewing main version, which requires installation from source. If you'd like regular pip install, checkout the latest stable version (v5.22.2).

Join the Hugging Face community

and get access to the augmented documentation experience

Collaborate on models, datasets and Spaces

Faster examples with accelerated inference

Switch between documentation themes

to get started

该模型于 2026-01-29 在 HF papers 上发布，并于 2026-06-26 贡献至 Hugging Face Transformers。

Qwen3 ASR

概述

Qwen3 ASR 是阿里巴巴 Qwen 团队推出的自动语音识别模型，它将 Whisper 风格的音频编码器与 Qwen3 语言模型解码器相结合，用于语音转文本。该模型支持自动语言检测和多语言转录。

该库还包含一个强制对齐器模型。它可以用于为提供的转录文本及其音频添加时间戳。该模型使用相同的音频编码器，并附加一个分类头来预测单词的长度。此模型可与任意 ASR 模型的转录文本配合使用（参见下方使用 Parakeet CTC 的示例）。

可用的检查点：

支持以下语言：

Qwen3-ASR-1.7B 和 Qwen3-ASR-0.6B：中文 (zh)、英语 (en)、粤语 (yue)、阿拉伯语 (ar)、德语 (de)、法语 (fr)、西班牙语 (es)、葡萄牙语 (pt)、印度尼西亚语 (id)、意大利语 (it)、韩语 (ko)、俄语 (ru)、泰语 (th)、越南语 (vi)、日语 (ja)、土耳其语 (tr)、印地语 (hi)、马来语 (ms)、荷兰语 (nl)、瑞典语 (sv)、丹麦语 (da)、芬兰语 (fi)、波兰语 (pl)、捷克语 (cs)、菲律宾语 (fil)、波斯语 (fa)、希腊语 (el)、匈牙利语 (hu)、马其顿语 (mk)、罗马尼亚语 (ro)。
Qwen3-ForcedAligner-0.6B：中文（zh）、英语（en）、粤语（yue）、法语（fr）、德语（de）、意大利语（it）、日语（ja）、韩语（ko）、葡萄牙语（pt）、俄语（ru）、西班牙语（es）。

有关更多详细信息，请参阅原始仓库 QwenLM/Qwen3-ASR 和报告。

该模型由 Eric Bezzam 和 Muhammed Tariq 贡献。

用法

简单转录

最简单的音频转录方式是使用 apply_transcription_request，它会自动处理聊天模板的格式化，即它是 apply_chat_template 的便捷封装（参见下文聊天模板）。

from transformers import AutoProcessor, AutoModelForMultimodalLM

model_id = "Qwen/Qwen3-ASR-1.7B-hf"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(model_id, device_map="auto")
print(f"Model loaded on {model.device} with dtype {model.dtype}")

inputs = processor.apply_transcription_request(
    audio="https://huggingface.co/datasets/bezzam/audio_samples/resolve/main/librispeech_mr_quilter.wav",
).to(model.device, model.dtype)

output_ids = model.generate(**inputs, max_new_tokens=256)
generated_ids = output_ids[:, inputs["input_ids"].shape[1]:]

# Raw output includes language tag and <asr_text> marker
raw = processor.decode(generated_ids)[0]
print(f"Raw: {raw}")

# Parsed output: dict with "language" and "transcription"
parsed = processor.decode(generated_ids, return_format="parsed")[0]
print(f"Parsed: {parsed}")

# Extract only the transcription text
transcription = processor.decode(generated_ids, return_format="transcription_only")[0]
print(f"Transcription: {transcription}")

"""
Raw: language English<asr_text>Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.
Parsed: {'language': 'English', 'transcription': 'Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.'}
Transcription: Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.
"""

强制指定语言

您可以按照以下所示强制指定转录语言。

from transformers import AutoProcessor, AutoModelForMultimodalLM

model_id = "Qwen/Qwen3-ASR-1.7B-hf"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(model_id, device_map="auto")

# Without language hint (auto-detect)
inputs = processor.apply_transcription_request(
    audio="https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/mandarin_voxcpm_zh.wav",
).to(model.device, model.dtype)
output_ids = model.generate(**inputs, max_new_tokens=256)
generated_ids = output_ids[:, inputs["input_ids"].shape[1]:]
print(f"Auto-detect: {processor.decode(generated_ids, return_format='transcription_only')[0]}")

# With forced language
inputs = processor.apply_transcription_request(
    audio="https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/mandarin_voxcpm_zh.wav",
    language="Chinese",  # or language code "zh"
).to(model.device, model.dtype)
output_ids = model.generate(**inputs, max_new_tokens=256)
generated_ids = output_ids[:, inputs["input_ids"].shape[1]:]
print(f"Forced:      {processor.decode(generated_ids, return_format='transcription_only')[0]}")

上下文 / 热词

您可以通过 prompt 传入自由形式的上下文（例如特定领域的词汇、名称或背景信息），以引导转录结果。

from transformers import AutoProcessor, AutoModelForMultimodalLM

model_id = "Qwen/Qwen3-ASR-1.7B-hf"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(model_id, device_map="auto")

inputs = processor.apply_transcription_request(
    audio="https://huggingface.co/datasets/bezzam/audio_samples/resolve/main/librispeech_mr_quilter.wav",
    prompt="Vocabulary: Quilter, apostle, gospel.",
    language="English",
).to(model.device, model.dtype)

output_ids = model.generate(**inputs, max_new_tokens=256)
generated_ids = output_ids[:, inputs["input_ids"].shape[1]:]
print(processor.decode(generated_ids, return_format="transcription_only")[0])

批量推理

通过传入音频列表，以及（如果提供的话）语言列表和/或提示词列表，即可执行批量推理。

from transformers import AutoProcessor, AutoModelForMultimodalLM

model_id = "Qwen/Qwen3-ASR-1.7B-hf"
audio = [
    "https://huggingface.co/datasets/bezzam/audio_samples/resolve/main/librispeech_mr_quilter.wav",
    "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/mandarin_voxcpm_zh.wav",
]

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(model_id, device_map="auto")

inputs = processor.apply_transcription_request(
    audio,
    prompt=["Vocabulary: Quilter, apostle, gospel.", None],
    language=[None, "zh"],  # language codes ("zh") and full names ("Chinese") are both accepted
).to(model.device, model.dtype)

output_ids = model.generate(**inputs, max_new_tokens=256)
generated_ids = output_ids[:, inputs["input_ids"].shape[1]:]
transcriptions = processor.decode(generated_ids, return_format="transcription_only")

for i, text in enumerate(transcriptions):
    print(f"Audio {i + 1}: {text}")

聊天模板

Qwen3 ASR 也接受聊天模板输入。上述用法是 apply_transcription_request 的便捷封装，其底层调用的是 apply_chat_template。

可以通过在聊天模板中预填充助手轮次为 language <NAME><asr_text> 并传入 continue_final_message=True 来强制指定语言，apply_transcription_request 在底层正是这样实现的。请注意，如果强制指定语言，则必须为批次中的所有音频设置预填充（如下所示）。

from transformers import AutoProcessor, Qwen3ASRForConditionalGeneration

model_id = "Qwen/Qwen3-ASR-1.7B-hf"
processor = AutoProcessor.from_pretrained(model_id)
model = Qwen3ASRForConditionalGeneration.from_pretrained(model_id, device_map="auto")

chat_template = [
    [
        # Context/hotwords as system message
        {"role": "system", "content": [{"type": "text", "text": "Vocabulary: Quilter, apostle, gospel."}]},
        {
            "role": "user",
            "content": [
                {
                    "type": "audio",
                    "path": "https://huggingface.co/datasets/bezzam/audio_samples/resolve/main/librispeech_mr_quilter.wav",
                },
            ],
        },
        # empty prefill since forcing language in the other sample
        {"role": "assistant", "content": [{"type": "text", "text": ""}]},
    ],
    [
        {
            "role": "user",
            "content": [
                {
                    "type": "audio",
                    "path": "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/mandarin_voxcpm_zh.wav",
                },
            ],
        },
        {"role": "assistant", "content": [{"type": "text", "text": "language Chinese<asr_text>"}]},
    ],
]

inputs = processor.apply_chat_template(
    chat_template, tokenize=True, return_dict=True, continue_final_message=True,
).to(model.device, model.dtype)

output_ids = model.generate(**inputs, max_new_tokens=256)
generated_ids = output_ids[:, inputs["input_ids"].shape[1]:]
transcriptions = processor.decode(generated_ids, return_format="transcription_only")
for text in transcriptions:
    print(text)

训练

将目标转录文本通过 transcription 传递给 apply_transcription_request，以获取用于计算损失的 labels。

from datasets import Audio, load_dataset
from transformers import AutoProcessor, Qwen3ASRForConditionalGeneration

model_id = "Qwen/Qwen3-ASR-1.7B-hf"
NUM_SAMPLES = 2

processor = AutoProcessor.from_pretrained(model_id)
model = Qwen3ASRForConditionalGeneration.from_pretrained(model_id, device_map="auto")
model.train()

ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate))
speech_samples = [el["array"] for el in ds["audio"][:NUM_SAMPLES]]
text_samples = ds["text"][:NUM_SAMPLES]

inputs = processor.apply_transcription_request(speech_samples, language="en", transcription=text_samples)
inputs.to(model.device, dtype=model.dtype)

loss = model(**inputs).loss
print("Loss:", loss.item())
loss.backward()

强制对齐（词级时间戳）

使用 Qwen3ASRForTokenClassification 从转录文本中获取词级时间戳。首先使用 ASR 模型进行转录，然后使用强制对齐器进行对齐。

支持以下语言：中文（zh）、英语（en）、粤语（yue）、法语（fr）、德语（de）、意大利语（it）、日语（ja）、韩语（ko）、葡萄牙语（pt）、俄语（ru）、西班牙语（es）。

日语需要 nagisa 库，而韩语需要 soynlp 库：

pip install nagisa soynlp

使用 Qwen3 ASR

import torch
from transformers import AutoProcessor, AutoModelForMultimodalLM, AutoModelForTokenClassification

asr_model_id = "Qwen/Qwen3-ASR-0.6B-hf"
aligner_model_id = "Qwen/Qwen3-ForcedAligner-0.6B-hf"

asr_processor = AutoProcessor.from_pretrained(asr_model_id)
asr_model = AutoModelForMultimodalLM.from_pretrained(asr_model_id, device_map="auto")

aligner_processor = AutoProcessor.from_pretrained(aligner_model_id)
aligner_model = AutoModelForTokenClassification.from_pretrained(
    aligner_model_id, dtype=torch.bfloat16, device_map="auto"
)

audio_url = "https://huggingface.co/datasets/bezzam/audio_samples/resolve/main/librispeech_mr_quilter.wav"

# Step 1: Transcribe
inputs = asr_processor.apply_transcription_request(audio=audio_url)
inputs = inputs.to(asr_model.device, asr_model.dtype)
output_ids = asr_model.generate(**inputs, max_new_tokens=256)
generated_ids = output_ids[:, inputs["input_ids"].shape[1]:]
parsed = asr_processor.decode(generated_ids, return_format="parsed")[0]
transcript = parsed["transcription"]
language = parsed["language"] or "English"

# Step 2: Prepare alignment inputs
aligner_inputs, word_lists = aligner_processor.prepare_forced_aligner_inputs(
    audio=audio_url, transcript=transcript, language=language,
)
aligner_inputs = aligner_inputs.to(aligner_model.device, aligner_model.dtype)

# Step 3: Run forced aligner
with torch.inference_mode():
    outputs = aligner_model(**aligner_inputs)

# Step 4: Decode timestamps
timestamps = aligner_processor.decode_forced_alignment(
    logits=outputs.logits,
    input_ids=aligner_inputs["input_ids"],
    word_lists=word_lists,
    timestamp_token_id=aligner_model.config.timestamp_token_id,
)[0]

for item in timestamps:
    print(f"{item['text']:<20} {item['start_time']:>8.3f}s → {item['end_time']:>8.3f}s")

"""
Word                  Start (s)    End (s)
------------------------------------------
Mr                        0.560      0.800
Quilter                   0.800      1.280
is                        1.280      1.440
the                       1.440      1.520
apostle                   1.520      2.080
...
"""

使用另一个 ASR 模型

强制对齐器与模型无关，这意味着可以提供来自任何 ASR 系统的转录文本。下面是一个使用 NVIDIA Parakeet CTC 进行转录的批量推理示例。

import torch
from datasets import Audio, load_dataset
from transformers import AutoModelForCTC, AutoProcessor, AutoModelForTokenClassification

parakeet_processor = AutoProcessor.from_pretrained("nvidia/parakeet-ctc-1.1b")
parakeet_model = AutoModelForCTC.from_pretrained(
    "nvidia/parakeet-ctc-1.1b", dtype="auto", device_map="auto",
)

aligner_model_id = "Qwen/Qwen3-ForcedAligner-0.6B-hf"
aligner_processor = AutoProcessor.from_pretrained(aligner_model_id)
aligner_model = AutoModelForTokenClassification.from_pretrained(
    aligner_model_id, dtype=torch.bfloat16, device_map="auto",
)

ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
ds = ds.cast_column("audio", Audio(sampling_rate=parakeet_processor.feature_extractor.sampling_rate))
audio_arrays = [ds[i]["audio"]["array"] for i in range(3)]
sr = ds[0]["audio"]["sampling_rate"]

# Batch transcribe with Parakeet
inputs = parakeet_processor(audio_arrays, sampling_rate=sr, return_tensors="pt", padding=True).to(
    parakeet_model.device, dtype=parakeet_model.dtype
)
with torch.inference_mode():
    outputs = parakeet_model.generate(**inputs)
transcripts = parakeet_processor.decode(outputs)

# Batch align with Qwen3 Forced Aligner
aligner_inputs, word_lists = aligner_processor.prepare_forced_aligner_inputs(
    audio=audio_arrays, transcript=transcripts, language="English",
)
aligner_inputs = aligner_inputs.to(aligner_model.device, aligner_model.dtype)

with torch.inference_mode():
    aligner_outputs = aligner_model(**aligner_inputs)

batch_timestamps = aligner_processor.decode_forced_alignment(
    logits=aligner_outputs.logits,
    input_ids=aligner_inputs["input_ids"],
    word_lists=word_lists,
    timestamp_token_id=aligner_model.config.timestamp_token_id,
)

for i, (transcript, timestamps) in enumerate(zip(transcripts, batch_timestamps)):
    print(f"\n[Sample {i}] {transcript}")
    for item in timestamps[:5]:
        print(f"  {item['text']:<20} {item['start_time']:>8.3f}s → {item['end_time']:>8.3f}s")
    if len(timestamps) > 5:
        print(f"  ... ({len(timestamps) - 5} more words)")

Torch 编译

ASR 模型和强制对齐模型均支持 torch.compile 以实现更快的推理。强制对齐模型尤其适合编译，因为它只执行一次前向传播（无需自回归解码）。这使其成为 批量音频时间戳标注 的理想选择：使用任意 ASR 模型进行转录，然后使用编译后的强制对齐模型进行批量对齐，以获得最大吞吐量。

强制对齐器

在 A100 上，当批处理大小为 4 时，我们观察到约 1.2 倍的加速效果（脚本）。

import torch
from transformers import AutoProcessor, AutoModelForTokenClassification

model_id = "Qwen/Qwen3-ForcedAligner-0.6B-hf"
num_warmup = 5
batch_size = 4

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForTokenClassification.from_pretrained(model_id, dtype=torch.bfloat16).to("cuda")

# Prepare a batch of 4 samples
audio_url = "https://huggingface.co/datasets/bezzam/audio_samples/resolve/main/librispeech_mr_quilter.wav"
transcript = "Mr. Quilter is the apostle of the middle classes."

aligner_inputs, word_lists = processor.prepare_forced_aligner_inputs(
    audio=[audio_url] * batch_size,
    transcript=[transcript] * batch_size,
    language=["English"] * batch_size,
)
aligner_inputs = aligner_inputs.to(model.device, torch.bfloat16)

# Warm-up and apply model
model = torch.compile(model)
with torch.no_grad():
    for _ in range(num_warmup):
        _ = model(**aligner_inputs)
with torch.no_grad():
    _ = model(**aligner_inputs)

ASR 模型（生成）

对于自回归转录，在 generate 设置中提供 CompileConfig 对象时，torch.compile 会加速其中每个 token 的前向传播。

在 A100 上，当批处理大小为 4 时，我们观察到约 3.8 倍的加速效果（脚本）。

import torch
from transformers import AutoProcessor, AutoModelForMultimodalLM, CompileConfig

model_id = "Qwen/Qwen3-ASR-1.7B-hf"
num_warmup = 3
max_new_tokens = 256

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(model_id, dtype=torch.bfloat16).to("cuda").eval()

audio_url = "https://huggingface.co/datasets/bezzam/audio_samples/resolve/main/librispeech_mr_quilter.wav"
inputs = processor.apply_transcription_request(
    audio=[audio_url] * 4,  # batch of 4
).to(model.device, torch.bfloat16)

compile_config = CompileConfig()

# Warmup
with torch.inference_mode():
    for _ in range(num_warmup):
        _ = model.generate(
            **inputs, max_new_tokens=max_new_tokens, do_sample=False,
            cache_implementation="static", compile_config=compile_config,
        )
torch.cuda.synchronize()

# Apply model
with torch.inference_mode():
    output_ids = model.generate(
        **inputs, max_new_tokens=max_new_tokens, do_sample=False,
        cache_implementation="static", compile_config=compile_config,
    )
generated_ids = output_ids[:, inputs["input_ids"].shape[1] :]
text_compiled = processor.decode(generated_ids, return_format="transcription_only")[0]
print(f"Output:  {text_compiled}")

Pipeline 用法

from transformers import pipeline

model_id = "Qwen/Qwen3-ASR-1.7B-hf"
pipe = pipeline("any-to-any", model=model_id, device_map="auto")

chat_template = [
    {
        "role": "user",
        "content": [
            {
                "type": "audio",
                "path": "https://huggingface.co/datasets/bezzam/audio_samples/resolve/main/librispeech_mr_quilter.wav",
            },
        ],
    }
]
outputs = pipe(text=chat_template, return_full_text=False)
raw_text = outputs[0]["generated_text"]
print(f"Raw: {raw_text}")

# Use processor helper to extract transcription
transcription = pipe.processor.extract_transcription(raw_text)
print(f"Transcription: {transcription}")

Transformers

Qwen3 ASR

概述

用法

简单转录

强制指定语言

上下文 / 热词

批量推理

聊天模板

训练

强制对齐（词级时间戳）

使用 Qwen3 ASR

使用另一个 ASR 模型

Torch 编译

强制对齐器

ASR 模型（生成）

Pipeline 用法

Qwen3ASRConfig

类 transformers.Qwen3ASRConfig

Qwen3ASREncoderConfig

类 transformers.Qwen3ASREncoderConfig

Qwen3ASRFeatureExtractor

类 transformers.Qwen3ASRFeatureExtractor

__call__

Qwen3ASRProcessor

class transformers.Qwen3ASRProcessor

__call__

apply_transcription_request

prepare_forced_aligner_inputs

decode_forced_alignment

decode

Qwen3ASREncoder

class transformers.Qwen3ASREncoder

forward

Qwen3ASRModel

class transformers.Qwen3ASRModel

forward

get_audio_features

get_placeholder_mask

Qwen3ASRForConditionalGeneration

类 transformers.Qwen3ASRForConditionalGeneration

forward

get_audio_features

Qwen3ASRForTokenClassification

class transformers.Qwen3ASRForTokenClassification

forward

call

call