RedHatAI

/

GLM-5.2-speculator.dspark

like 9

Follow

Red Hat AI 3.05k

@ehartford Interesting, this seems a little odd. Do you mind sharing a little more about your use case? Would really appreciate it. Prompt length, category, etc? Thanks!

I'm also having the same issue. We get a mean acceptance length of 2-3 tokens, sometimes 1.8. It never reaches 4. With native vLLM mtp it was consistently in the 3-4 range.

• Mostly python code
• Claude Code as harness
• Using nvidia/glm-5.2-nvfp4 as model with 512K context.

I'm also having the same issue. We get a mean acceptance length of 2-3 tokens, sometimes 1.8. It never reaches 4. With native vLLM mtp it was consistently in the 3-4 range.

• Mostly python code
• Claude Code as harness
• Using nvidia/glm-5.2-nvfp4 as model with 512K context.

what is your serving setup? vllm/sglang version, hardware etc.

I'm also having the same issue. We get a mean acceptance length of 2-3 tokens, sometimes 1.8. It never reaches 4. With native vLLM mtp it was consistently in the 3-4 range.

• Mostly python code
• Claude Code as harness
• Using nvidia/glm-5.2-nvfp4 as model with 512K context.

what is your serving setup? vllm/sglang version, hardware etc.

I'm using the recipe from vLLM from here https://recipes.vllm.ai/zai-org/GLM-5.2 but with the NVFP4 + this Spec model.

vllm serve nvidia/GLM-5.2-NVFP4 \
  --kv-cache-dtype fp8_e4m3 \
  --enable-expert-parallel \
  --tensor-parallel-size 8 \
  --max-num-batched-tokens 8192 \
  --max-num-seqs 32 \
  --gpu-memory-utilization 0.93 \
  --tool-call-parser glm47 \
  --enable-auto-tool-choice \
  --reasoning-parser glm45 \
  --max-model-len 512k \
  --speculative-config '{"model": "RedHatAI/GLM-5.2-speculator.dspark","num_speculative_tokens": 7,"method": "dspark"}' \
  --kv-transfer-config '{"kv_connector":"SimpleCPUOffloadConnector","kv_role":"kv_both","kv_connector_extra_config":{"cpu_bytes_to_use_per_rank":136223201280,"lazy_offload":false}}'

Software: vLLM v0.27.1 (Docker)
Drivers: 595 - CUDA 13.x
Hardware: 8xB200 with NVLink

@g-a-b-y Is there a specific dataset we're targeting for benchmarking? If not, could you briefly describe what kind of tasks you're using this model for? Would be super helpful! Performance of these drafters varies a lot for different tasks. Perhaps we could try finetuning for your specific usecase.

@shanjiaz We are serving the model for a big group of users. Tasking includes:

• python code development.
• golang code development.
• research questions.
• bug finding/planning.
• managing docker, vms, ansible and systemd services via claude.

After a day running our mean acceptance length now averages 3.5-4.0, so maybe we just need to lower the num_speculative_tokens to like 5 ?

Recently zai-org and nvidia released new datasets for terminal bench/coding, would be nice for those to be included. At the same time GLM-5.3 is coming out today so...

Datasets:

• https://huggingface.co/datasets/zai-org/terminal-bench-2-verified
• https://huggingface.co/datasets/nvidia/Open-SWE-Traces

GLM-5.3 countdown: https://huggingface.co/zai-org/GLM-5.3

One explanation is this model has been updated once, so you might be using a cached version? Thanks for the dataset recommendation! Agentic workflow is definitely our focus. 🫡 Thanks again for trying our model! Reach out to us on vLLM slack if you have more questions/suggestions. My name is Helen Zhao

@shanjiaz I redownloaded the model this week. I'm gonna start using GLM-5.3-Flash now, so it won't matter in the new few days.

Any plans to release a DSpark for GLM-5.3 and GLM-5.3-Flash ?

@g-a-b-y Stay tuned