Qwen3.8-27B-Quark-AWQ-MXFP4

Qwen/Qwen3.8-27B quantized to MXFP4 weights + MXFP4 activations with AMD Quark, using the AWQ (Activation-aware Weight Quantization) algorithm.

• Weights: MXFP4 (OCP microscaling FP4, group size 32, E8M0 shared scale)
• Activations: MXFP4, dynamic (per-block, computed at runtime)
• Algorithm: AWQ
• Calibration: 128 samples, seq_len 512, pileval dataset (Quark default)
• Base model: Qwen/Qwen3.8-27B (bf16)
• Vision tower: left unquantized (bf16) — only the language-model decoder (model.language_model.layers) is quantized; model.visual.* is excluded

Benchmark results

Benchmark	Setting	This model (AWQ)	BF16 base	Recovery %
GSM8K, 5-shot (flexible-extract / strict-match)	Thinking: temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0, max_gen_toks=3072	94.996% / 95.30%	93.33% / 93.33%	101.8%
GSM8K, 5-shot (flexible-extract / strict-match)	Non-thinking: temperature=0.7, top_p=0.80, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0, max_gen_toks=1024	89.92% / 89.76%	90.67% / 89.76%	99.2%

Recovery % is computed on the flexible-extract metric (this model / bf16 base).

Eval commands

Thinking mode:

lm-eval run --model local-completions \
  --model_args model=awq,base_url=http://localhost:8002/v1/completions,tokenizer=amd/Qwen3.8-27B-Quark-AWQ-MXFP4,num_concurrent=16,max_retries=5,timeout=900,tokenized_requests=False,max_length=16384 \
  --gen_kwargs max_gen_toks=3072 temperature=1.0 top_p=0.95 top_k=20 min_p=0.0 presence_penalty=0.0 repetition_penalty=1.0 do_sample=True \
  --tasks gsm8k \
  --output_path ./results/awq_gsm8k \
  --log_samples

Non-thinking mode: the raw-completions endpoint used above doesn't support passing enable_thinking=False through a chat template, so non-thinking mode is approximated with a custom task that pre-closes an empty <think></think> block in the prompt:

gsm8k_nothink.yaml:

tag:
  - math_word_problems
task: gsm8k_nothink
dataset_path: openai/gsm8k
dataset_name: main
output_type: generate_until
training_split: train
fewshot_split: train
test_split: test
doc_to_text: "Question: {{question}}\nAnswer:<think>\n\n</think>\n\n"
doc_to_target: "{{answer}}"
metric_list:
  - metric: exact_match
    aggregation: mean
    higher_is_better: true
    ignore_case: true
    ignore_punctuation: false
    regexes_to_ignore:
      - ","
      - "\\$"
      - "(?s).*#### "
      - "\\.$"
generation_kwargs:
  until:
    - "Question:"
    - "</s>"
    - "<|im_end|>"
  do_sample: false
  temperature: 0.0
repeats: 1
num_fewshot: 5
filter_list:
  - name: "strict-match"
    filter:
      - function: "regex"
        regex_pattern: "#### (\\-?[0-9\\.\\,]+)"
      - function: "take_first"
  - name: "flexible-extract"
    filter:
      - function: "regex"
        group_select: -1
        regex_pattern: "(-?[$0-9.,]{2,})|(-?[0-9]+)"
      - function: "take_first"
metadata:
  version: 3.0

lm-eval run --model local-completions \
  --model_args model=awq,base_url=http://localhost:8002/v1/completions,tokenizer=amd/Qwen3.8-27B-Quark-AWQ-MXFP4,num_concurrent=16,max_retries=5,timeout=900,tokenized_requests=False,max_length=16384 \
  --include_path ./custom_tasks \
  --gen_kwargs max_gen_toks=1024 temperature=0.7 top_p=0.80 top_k=20 min_p=0.0 presence_penalty=1.5 repetition_penalty=1.0 do_sample=True \
  --tasks gsm8k_nothink \
  --output_path ./results/awq_gsm8k_nothink \
  --log_samples

Quantization command

HIP_VISIBLE_DEVICES=<gpu> python3 quantize_quark.py \
  --model_dir Qwen/Qwen3.8-27B \
  --output_dir ./Qwen3.8-27B-w_mxfp4_a_mxfp4-awq \
  --quant_scheme mxfp4 \
  --num_calib_data 128 \
  --seq_len 512 \
  --quant_algo awq \
  --model_export hf_format \
  --data_type auto \
  --device cuda \
  --trust_remote_code

Serving

vllm serve amd/Qwen3.8-27B-Quark-AWQ-MXFP4 \
  --served-model-name awq \
  --trust-remote-code \
  --tensor-parallel-size 1 \
  --max-model-len 16384

License

Modifications Copyright(c) 2026 Advanced Micro Devices, Inc. All rights reserved.