MTP drafter for Gemma 4 31B Scotoma-2
A multi-token-prediction (MTP) drafter for
ReadyArt/gemma-4-31B-it-scotoma-2,
fine-tuned from Google's official gemma-4-31B-it-assistant drafter on Scotoma-2's own outputs.
It is a drop-in replacement for Google's drafter: same architecture, same size, same files. Speculative decoding is lossless, so the text Scotoma-2 writes does not change. Only the speed does.
Results
llama.cpp llama-server on an NVIDIA GB10 (DGX Spark class), target Scotoma-2 Q4_K_M,
4 draft tokens, 104 prompts (30 per category, 14 for tools), up to 1024 tokens per answer.
Tokens per second, draft acceptance in parentheses. No drafter: 11.0 tok/s.
Measured with the GGUF builds of each drafter (this drafter's are in NeuR0mancR/Scotoma-2-31B-MTP-GGUF).
| Drafter | Writing | RP | Tech | Tools | All |
|---|---|---|---|---|---|
| Temperature 1 | |||||
| Google, Q8_0 | 22.4 (36%) | 22.6 (37%) | 33.0 (66%) | 34.6 (73%) | 26.3 (47%) |
| This drafter, Q8_0 | 23.1 (37%) | 23.1 (39%) | 33.8 (67%) | 35.3 (75%) | 26.9 (48%) |
| Google, Q4_0 | 22.4 (35%) | 22.7 (36%) | 33.8 (67%) | 35.2 (73%) | 26.4 (46%) |
| This drafter, Q4_0 | 23.0 (36%) | 23.1 (38%) | 33.9 (66%) | 35.8 (75%) | 26.9 (47%) |
| Greedy | |||||
| Google, Q8_0 | 23.0 (38%) | 23.9 (41%) | 35.0 (72%) | 35.8 (77%) | 27.3 (50%) |
| This drafter, Q8_0 | 23.6 (39%) | 24.8 (44%) | 35.5 (72%) | 37.5 (80%) | 28.1 (52%) |
| Google, Q4_0 | 23.9 (37%) | 24.6 (41%) | 36.1 (71%) | 37.3 (77%) | 28.3 (50%) |
| This drafter, Q4_0 | 24.2 (39%) | 24.8 (43%) | 35.6 (72%) | 37.8 (80%) | 28.4 (52%) |
In short:
- At temperature 1 (typical RP and writing use): about +2% over Google's drafter, in Q8_0 and Q4_0, with no category slower.
- Greedy: the gain is marginal. Google's drafter quantized to Q4_0 is as fast as this one in Q8_0.
- Q4_0 costs about one point of acceptance but runs as fast as Q8_0 (the drafter is lighter to read).
- RP prompts in the benchmark use characters never seen in training.
Usage
This repository holds the bf16 weights. GGUF builds for llama.cpp (Q8_0, Q4_0) are in NeuR0mancR/Scotoma-2-31B-MTP-GGUF.
llama-server -m gemma-4-31B-scotoma-2-Q4_K_M.gguf \
--spec-type draft-mtp --model-draft Scotoma-2-31B-MTP-Q8_0.gguf \
--spec-draft-n-max 4 --spec-draft-p-min 0.0
How it was trained
- Data: about 8.6M learned tokens of Scotoma-2 responses (writing 33%, tech 30%, RP 29%, tools 7%; 42% reasoning), generated by Scotoma-2 itself at temperature 1. Prompts from WritingPrompts, Gryphe's RP set, PIPPA, Tulu, Magicoder, Hermes function calling, and custom multi-turn tool scenarios (including game-master tools for RP).
- Target: Scotoma-2 bf16 probabilities. Loss: total variation distance, over the 4 draft rounds, following llama.cpp's convention (the drafter sees the target's state before the token).
- 2 epochs, learning rate 2e-5 with cosine decay; vocabulary table frozen.
- Validation and benchmark are split by character card, so no RP character appears on both sides.
Limitations
- Each benchmark configuration was run once; differences of about 1% are close to run-to-run noise.
- Not measured with DRY or XTC samplers. XTC in particular should reduce gains for any drafter.
- Trained and evaluated on sequences up to about 6k tokens; long contexts are untested.
- Mostly English data. Tested only with llama.cpp.
- The RP and writing benchmark prompts are synthetic (written by an LLM); responses come from Scotoma-2.
Credits
Scotoma-2 by AesSedai / ReadyArt. Base drafter by Google.
Special thanks to AesSedai and the BeaverAI community for their support.
- Downloads last month
- 202