Qwen3.5-0.8B: Astra Weight Edit

Qwen3.5-0.8B with 2 tensors edited directly by an LLM (GPT-6 Astra), with no training. This fixes the model's thinking-mode repeat loops.

All credit goes to Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab (MIT CSAIL). I only applied their released diff and checked that it matches their evaluated checkpoint hash for hash. Not affiliated with or endorsed by the authors, OpenAI or Qwen.

๐Ÿšง Independent benchmarking in progress: MATH-500 base vs. edited, thinking on and off. Results will be added here.

What changed

The loop direction was subtracted from 64 output columns of mlp.down_proj in layers 15 and 19 (about 130k weights, 0.016% of the model). Everything else is byte-identical to the base model.

Reported results (from the authors, not yet reproduced)

Thinking on, temperature 0.6, top_p 0.95, top_k 20, presence_penalty 1.5, 32k tokens:

MATH-500 GSM8K IFEval MMLU-Pro
Base 7.9 13.8 21.0 40.3
Edited 48.1 67.1 62.3 45.5

โš ๏ธ This removes a failure mode; it doesn't add capability. Most of the gain comes from answers that no longer hit the token limit (93% โ†’ 33% cut off). With thinking off, the base model doesn't loop much, so compare against that too.

TODO

  • Independent MATH-500 benchmark (in progress)
  • GSM8K / IFEval reproduction
  • ONNX / Transformers.js build for in-browser demo

Notes

  • The authors' grader (mathqa.py) imports an internal module (mcqa) that isn't in their release. It's only used for loop diagnostics, so my benchmark uses a one-function stand-in and leaves their file untouched.
  • evaluate.py also needs their internal package, so I wrote a small MATH-500 script that copies its prompt, sampling and vLLM settings.
  • Rebuild it yourself with exp8_math/code/apply_edit.py (base revision 2fc06364). The script checks every tensor's hash.

License

The base model is Apache 2.0 ยฉ Qwen (LICENSE). This is a modified version: 2 tensors changed. The edit is CC BY 4.0, by the authors above.

@misc{takehi2026weightspace,
  title  = {Sparks of Weight-Space Programming: LLMs can natively read \& write neural network weights},
  author = {Takehi, Rikiya and Teoh, Jayden and Zhang, Alex and Khattab, Omar},
  year   = {2026},
  url    = {https://rikiyat.github.io/blog/weight-space-programming/}
}
Downloads last month
113
Safetensors
Model size
0.9B params
Tensor type
BF16
ยท
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for OxBee/Qwen3.5-0.8B-Astra-WeightEdit

Finetuned
(979)
this model