Instructions to use PaoAI/Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use PaoAI/Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf PaoAI/Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED-GGUF
# Run inference directly in the terminal:
llama cli -hf PaoAI/Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED-GGUF

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf PaoAI/Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED-GGUF
# Run inference directly in the terminal:
llama cli -hf PaoAI/Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED-GGUF

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf PaoAI/Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED-GGUF
# Run inference directly in the terminal:
./llama-cli -hf PaoAI/Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED-GGUF

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf PaoAI/Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED-GGUF
# Run inference directly in the terminal:
./build/bin/llama-cli -hf PaoAI/Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED-GGUF

Use Docker

docker model run hf.co/PaoAI/Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED-GGUF

LM Studio
Jan

vLLM

How to use PaoAI/Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "PaoAI/Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "PaoAI/Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/PaoAI/Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED-GGUF

Ollama
How to use PaoAI/Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED-GGUF with Ollama:
```
ollama run hf.co/PaoAI/Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED-GGUF
```
Unsloth Desktop

How to use PaoAI/Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf PaoAI/Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED-GGUF

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "PaoAI/Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED-GGUF"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use PaoAI/Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED-GGUF with Docker Model Runner:
```
docker model run hf.co/PaoAI/Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED-GGUF
```

Lemonade

How to use PaoAI/Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull PaoAI/Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED-GGUF

Run and chat with the model

lemonade run user.Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED-GGUF-{{QUANT_TAG}}

List all available models

lemonade list

Hermes Agent

How to use PaoAI/Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf PaoAI/Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED-GGUF

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default PaoAI/Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED-GGUF

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use PaoAI/Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf PaoAI/Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED-GGUF

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "PaoAI/Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED-GGUF" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED-GGUF

The balanced Qwen3.8-27B quant for AMD Strix Halo (Ryzen AI Max+ 395 / gfx1151). Measured, not marketed: one 15 GB file that serves 128K of context on the iGPU — full attention, no offload flags, speculative decoding built in.

Brand: PaoAI · community quant, not affiliated with Alibaba or the Qwen team.

Updated 2026-09-20 — re-swept on the latest engine, weights unchanged. Every depth point re-measured on the latest version (paoai-strix-engine strix-main @ 4905ff417): deeper than ever — the full 256K window served and integrity-checked for the first time (261,519 tokens) — with decode faster at every depth from 32K up. The previous version's numbers stay on this card labeled as such. See Update log.

What is this? (30 seconds)

Qwen3.8-27B is a hybrid: 48 cheap linear-attention layers and only 16 full-attention layers that actually re-read your context. We compressed it organ by organ — precision where it matters (attention K/V, the MTP draft head, every norm), 4-bit FP4 where it doesn't. One 15 GB file, no offload flags, speculative decoding built in, intact from 8K to 128K of filled context.

The recipe (what's inside)

Model part	What it does	Format
Full attention Q/K/V/O (16 layers)	reads your prompt, tracks context	Q4_0_ROCMFP4_STRIX (~4.5 bpw)
⚡ Linear attention (48 gated-delta layers)	the fast path — constant-time state	ROCmFP4 (qkv, type 100)
FFN gate/up + down (dense)	the knowledge (biggest block: 9.2 GB)	ROCmFP4 (~4.5 bpw, imatrix)
✍️ Draft head (blk.64)	MTP speculative guessing	Q8_0 + FP32 (kept sharp)
📖 Token embeddings	vocab lookup	Q6_K (~6.6 bpw)
Norms, gates, glue	keep signals correct	FP32 (never touched)

Measured tensor sizes: FFN gate/up 6.16 GB · FFN down 3.08 GB · linear-attn qkv 1.42 GB · GDN state ops 0.81 GB · token embeddings 1.04 GB · output 0.68 GB · full-attn 0.62 GB · draft head 0.10 GB · norms <0.01 GB → 15.01 GB total.

Chain-test results (N=3, graded by execution)

We run a fixed coding exam where tasks are graded by execution, not opinion. Three identical runs per engine; we report medians.

Task (what it measures)	previous version (published engine)	previous version (Vulkan fork)
Hunt — read lying code: trace it + find the planted bug	100 · 100 · 100	100 · 100 · 100
Automaton — implement novel rules + exact state trace	100 · 100 · 100	100 · 100 · 100
Ring buffer — multi-threaded code under stress	100 · 100 · 100	100 · 100 · 100
Task median	100	100

A same-session reference run of the published engine also scored 100/100/100 · task 100. Zero protocol events on either engine — quality is a straight tie.

Speed (decode, tokens/second, per phase)

Phase	previous version (published)	previous version (fork, same session)	latest version (same session)	latest vs fork
Spec (planning)	19.2	20.4	19.3	+6 %
Hunt (analysis)	21.6	26.3	24.7	+6 %
Automaton (code writing)	23.2	29.8	26.1	+14 %
Ring buffer (code writing)	20.7	22.9	19.8	+16 %
Publish (long output)	23.2	27.5	25.2	+9 %

*The published column is from September. The defensible comparison is the last column — same session, same flags, engine the only variable: +6 % to +16 %.*

Does quantization hurt? (perplexity + KL test — measured 2026-09-17, published 2026-09-18)

Plain words: the chain test above can only tell you if the model still passes. This test asks a different question: did compressing the weights change what the model actually predicts? We ran the industry-standard check — llama.cpp's own llama-perplexity tool — comparing this 15 GB FP4 file against a full-precision F16 copy of the same model (built by us from Qwen's official BF16 release, sha-verified shards), reading the same ~295,000 tokens of standard Wikipedia text through both at 8K windows.

Metric	What it means	Result
Perplexity delta	how much worse the quant predicts text vs its own F16 parent (lower = better)	−1.0 % (6.587 vs 6.654 — statistically identical)
Mean KL divergence	average "extra surprise" per token vs the parent — 0 = identical	0.086 (median 0.029)
Same top token	how often both models pick the exact same next word	88.8 % of positions

Plain read: on standard text, the FP4 cut is statistically indistinguishable from the full-precision model — no measurable quality loss. The small difference that exists is concentrated in a tail of rare positions (~1 % of tokens shift noticeably), and the chain test above shows those shifts didn't cost task success. Not lossless in the mathematical sense — lossy in a way we cannot measure on standard text.

Reproducible: llama.cpp llama-perplexity, engine commit 11bfe8a, wikitext-2-raw, -c 8192, 36 chunks (~295K tokens). Reference F16 converted by us from Qwen's official BF16 release (sha-verified shards). Full data in our results file.

Context decay — what happens when the window fills (8K → 256K)

Depth	Verified fill (latest)	prev pp	latest pp	Δpp	prev tg	latest tg	latest acc	391
8K	8,247	306.0	294.6	−4%	31.1	26.4	0.60	✓
16K	16,439	281.4	274.8	−2%	28.5	27.3	0.66	✓
32K	32,784	243.7	246.9	+1%	28.6	32.8	0.98	✓
64K	65,596	173.1	186.4	+8%	21.6	21.3	0.71	✓
96K	98,364	138.5	146.4	+6%	14.0	17.9	0.75	✓
128K	131,128	113.8	122.3	+7%	14.0	17.1	0.87	✓
256K	261,519	—	79.3	new	—	11.1	0.84	✓

Latest version (2026-09-20, 4905ff417): fresh prompt and fresh server per depth, default batching. Previous-version columns (2026-09-17, fork 6d1d80e3f) from the same protocol — comparable.

Deep range note (new, measured): past ~200K the default batching can lose the Vulkan device (vk::DeviceLostError, reproduced at ~213K). On the latest version the fix is the same small-batch rule the Flash-Next card uses: serve the deep range with -ub 256 — verified clean to 261,519 tokens (integrity ✓, acceptance 0.84). The auto-switch in the engine handles this automatically when serving with wide batches; at default batching, add -ub 256 for 200K+ work.

Plain read: decode is faster on the latest version at every depth from 32K up (+15–27% at 96K–128K), prefill catches up and wins from 64K (+6–8%), and the window now honestly reaches 256K with integrity — previously only reached 192K, and never on the card.

Facts

Metric	Result
File size	15.01 GB (13.98 GiB), single file
sha256	`3727594804afef8140165e36c0919c70a98b43751a3153d43dd10385c1f100fd`
Load	full GPU (GTT ≈ 51.6 GB at 64K ctx), no special flags needed
Context	standard serve `-c 65536`; full 262,144-token window measured and integrity-checked (deepest: 261,519 tokens)
Served with	MTP speculative decoding (n-max 4), flash attention, KV q8_0
State of the art here	100/100/100/100 chain median (3 runs, both engines) · 7/7 depth points integrity-pass (8K→256K, latest version) · decode up to 32.8 t/s and prefill 295 t/s at 8K
Weights	unchanged by this update — same sha256 as the original publish

Requirements — which llama.cpp build

This model uses two things stock llama.cpp doesn't have: the qwen3_5 architecture (hybrid linear + full attention + MTP) and the Q4_0_ROCMFP4_STRIX quant type (type 105, gfx1151-native).

Recommended (latest version): guevae2/paoai-strix-engine, branch strix-main @ 4905ff417 — every number labeled "latest" on this card
Previous version: kingjones30/ROCmFPX, branch z13-main, commit dfeacaf2f — the build the original card measured.

Build	Error you'll see
stock llama.cpp	`unknown model architecture: 'qwen3_5'`
llama.cpp without the ROCmFP4 types	`invalid ggml type 105. should be in [0, 43)`

How to run (measured, locked)

llama-server -m Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED.gguf \
  --host 0.0.0.0 --port 8091 -ngl 99 -c 65536 \
  --spec-type draft-mtp --spec-draft-n-max 4 \
  -fa on --cache-type-k q8_0 --cache-type-v q8_0 --reasoning-budget 1500

Same flags for both engines. Raise -c for long context (up to 262144). Do not add -b/-ub 2048, and do not enable adaptive drafting — both were measured and rejected (see Known limits).

Thinking budget — --reasoning-budget caps how long the model thinks before answering:

Value	Behaviour
`-1` (or omit)	unrestricted thinking, best quality, slowest
`1500`	what every number on this card was measured with
`4000`	noticeably deeper reasoning on hard tasks
`12000`	deep work — long waits before the answer starts

Known limits (honest)

Depth costs speed: prefill 306 → 114 t/s and decode 31 → 14 t/s from 8K → 128K of filled context. The 16 full-attention layers re-read history for every token; the 48 linear layers stay cheap. Linear, no cliff — but plan for it at 100K+.
Decode is not a clean win on the new engine: better at 8K–64K, worse at 96K (14.0 vs 16.5), level at 128K.
Caveat on the comparison: the "prev" numbers were measured under an unrecorded power state; the fork's are at the AC performance profile — so the deltas blend engine + profile. The clean single-variable figure is +18–27 % prefill (4-arm test, identical fills, same-session reference).
Not adopted, and why: adaptive drafting (1 formatting failure in 3 chain runs) · wide batch -b 2048 -ub 2048 (no gain, and it lost the GPU device at depth).
Vulkan only: every number here is from the gfx1151 Vulkan backend on a Strix Halo iGPU. CPU builds and other GPUs are not comparable.
Long thinking: a bigger --reasoning-budget means noticeably longer waits before the answer starts.
Don't crank context blindly: a single 15 GB file, but GTT grows with context (≈51.6 GB at 64K). Raise -c in steps on a 128 GB box; don't merge shards.

Serving features explained (plain words)

Flag	What it does	Cost
`--spec-type draft-mtp --spec-draft-n-max 4`	the model's own draft head guesses 4 tokens ahead, the model verifies them in one pass — roughly 2× decode	needs the Q8_0 draft head we kept
`-fa on`	flash attention — faster prefill and decode	none
`--cache-type-k/v q8_0`	half the context memory	negligible
`--reasoning-budget 1500`	caps thinking so answers always arrive	complex tasks may need more
`prompt caching` (default)	follow-up turns prefill in milliseconds	none

Draft acceptance on the depth-sweep probes measured 0.58–0.83 on the fork — lower than the 0.86–0.90 on the original card. The sweep prompts are repetitive filler, which a draft head predicts poorly, so acceptance there is not comparable across tests. On the chain (real work) the fork decoded faster than the previous engine.

Update log

Date	What changed
2026-09-20	Re-swept 8K→256K on the latest version (paoai-strix-engine `4905ff417`): first full-window 256K integrity pass (261,519 tokens), decode faster at every depth from 32K up, chart regenerated. Weights unchanged — same sha256.
2026-09-18	Added quant-loss measurement (perplexity + KL vs own F16 parent): no measurable loss. Card text only — weights untouched.
2026-09-18	Changed the recommended engine to our fork paoai-strix-engine (`6d1d80e3f`), a fork of `11bfe8a` with a few of our patches. Re-ran the battery on it: same speed, same quality. Weights untouched.
2026-09-17	Numbers re-measured on the STRIX Vulkan fork (`11bfe8a`). Prefill +11–33 % at matching fills; quality unchanged (3 runs 100/100/100); 128K holds with integrity; 192K bonus row. Weights untouched — same sha256, same 15.01 GB.
2026-09-16	Model published: 15.01 GB single file, MTP draft head, sweep measured on the ROCmFPX engine.

Earlier engine numbers are not repeated here — they were measured on a different build; the tables above are the current measurement. If you reproduce a number, quote the engine commit with it.

Fleet recipes

Serving configs and our benchmark battery: paoai-strix-recipes (GitHub).

Credits & license

Qwen team (Alibaba) — Qwen3.8-27B base model (Apache-2.0)
ggml-org / llama.cpp contributors — the engine, the quant formats, the Vulkan backend
LaurentZuijdwijk — the Strix-Halo-tuned Vulkan fork measured on this card
nathanw1014 — ROCmFP4 types, qwen3.5/MTP work, Strix Halo gfx1151 performance
kingjones30 / charlie12345 — ROCmFPX format & recipes
vcruz305 — converter branch work (qwen3_5 support path)
unsloth — imatrix methodology reference
deepu105 — same-box Strix Halo proof and serving-config intel
antirez — GGUF layout reference
PaoAI — the STRIX recipe (per-tensor cut/keep table), imatrix-guided builds, sweep benchmarking, chain test

Apache-2.0, same as the base model. Not affiliated with Alibaba or the Qwen team.

Downloads last month: 974

GGUF

Model size

27B params

Architecture

qwen35

Hardware compatibility

We're not able to determine the quantization variants.

View all variants

Model tree for PaoAI/Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED-GGUF

Base model

Qwen/Qwen3.8-27B

Quantized

(1174)

this model