meshapplied/Qwen3.8-Flash-Next-Ternary-Latest

Qwen3.8-Flash-Next ternary experts

Ternary routed-expert overlay for Qwen3.8-Flash-Next. Hadamard-rotated ternary packs (ternary-g128-hadamard-gptq, 48 layers). Attention, router, shared experts, and MTP stay in the base checkpoint (not re-uploaded here).

Drafter

Path	Role
mtp.* in the base checkpoint	NEXTN / MTP drafter (ternary pack skips MTP layers)
hot-vocab map (optional)	Draft lm_head token map (--speculative-token-map)

Default: NEXTN with base MTP (--speculative-num-steps 3, --speculative-num-draft-tokens 4).

Serve: single GPU, concurrency 1

git clone https://github.com/meshapplied/goethe
cd goethe
pip install -e ./ternary-kernels

import ternary_runtime
m = ternary_runtime.fetch("meshapplied/Qwen3.8-Flash-Next-Ternary-Latest")
ternary_runtime.bootstrap()

CUDA_VISIBLE_DEVICES=0 python -m sglang.launch_server \
  --model-path "$TERNARY_MODEL_DIR" \
  --tp-size 1 \
  --dtype bfloat16 \
  --trust-remote-code \
  --kv-cache-dtype fp8_e4m3 \
  --speculative-algorithm NEXTN \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --speculative-token-map /path/to/hot_self_32k.pt \
  --mem-fraction-static 0.90 \
  --context-length 16384 \
  --max-running-requests 1 \
  --chunked-prefill-size 4096 \
  --page-size 64 \
  --cuda-graph-backend-decode full \
  --cuda-graph-backend-prefill disabled \
  --cuda-graph-max-bs-decode 1 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen \
  --served-model-name qwen38-flash-next-ternary \
  --host 127.0.0.1 --port 31003

TERNARY_PACK_DIR / TERNARY_PROFILE=flash_next must be set (via fetch / bootstrap). Example TOML: Goethe examples/ternary-flash-next.toml.

Serve: single GPU, concurrency 4

Same command with:

  --max-running-requests 4 \
  --cuda-graph-max-bs-decode 4

Performance

Hardware unless noted: single RTX PRO 6000 96 GB, TP1, Goethe ternary_runtime + SGLang, FP8 KV, NEXTN 3/4 + hot vocab, T=0. Expert overlay: meshapplied/Qwen3.8-Flash-Next-Ternary-Latest on base Qwen/Qwen3.8-Flash-Next.

VRAM

Serve knobs for this snapshot: mem 0.90, chunk 4096, context 65536, max_running 1, --disable-radix-cache.

GPU used	85.2 GB (87 266 MiB)
GPU total	95.6 GB (97 887 MiB)
Scheduler process	86.4 GB (88 482 MiB, warm max_running=4)
max_total_num_tokens	~1.7M–2.5M (depends on radix / max_running)

Agentic coding throughput

Replay of 10 SWE-bench Pro / DeepSWE mini-swe-agent conversations × 6 turns each. Full chat history + tool observations fed back each turn; 256 output tokens/turn; prefix cache on; hot32; NEXTN 3/4. Headline concurrency-1 decode rate: 168 tok/s.

Concurrency	out tok/s	accept length	TTFT ms	ITL ms
1	168	2.99	338	5.2
4	265	3.03	451	13.3

Random prompts

sglang.benchmark.serving backend sglang-oai-chat, 32 prompts, in=512, out=256, radix cache on, hot32.

Concurrency	out tok/s	accept length	TTFT ms	ITL ms
1	155	2.73	183	5.2
4	326	2.74	165	11.0

Cold prefill and long-context decode

Same GPU/stack with --disable-radix-cache and --flush-cache between lengths (no hot map on this cold boot). sglang.benchmark.serving random, 8 prompts, out=256, concurrency 1. Prefill tok/s ≈ prompt_len / mean_TTFT.

prompt	decode tok/s	prefill tok/s	TTFT mean / p50 / p90 / p99 (ms)	ITL mean / p50 / p90 / p99 (ms)	accept
1024	135	~4.2k	247 / 179 / 448 / 482	5.7 / 4.1 / 8.3 / 16.6	2.88
4096	114	~9.2k	444 / 402 / 701 / 820	5.7 / 4.2 / 8.4 / 16.8	2.98
16384	61	~11k	1490 / 1524 / 2322 / 2521	5.9 / 4.2 / 8.4 / 16.9	2.82
32768	43	~13k	2517 / 2267 / 4240 / 4861	5.7 / 4.2 / 8.4 / 17.0	3.00

Decode slows as context grows; ITL stays flat; TTFT scales with prompt length.

Speculative decoding

knob	value
algorithm	NEXTN (MTP in base checkpoint)
steps	3
draft_tokens	4
eagle_topk	1
accept thresholds	1.0 / 1.0 (lossless vs target)

Format

• ternary_config.json: profile flash_next, gate_up_down layout, 48 layers
• layer-00.safetensors … layer-47.safetensors
• manifest-layer-*.json, eval.json
• Hadamard-rotated ternary GPTQ, group 128. Fused gate_up_proj / down_proj.

License

• This pack / card: Apache-2.0
• Base model: Qwen license / community terms on the base repo