BitMTP-Bonsai-4B-D8 / README.md
VCLab-PolyU's picture
Initial release
2a55a30 verified
|
Raw History Blame Contribute Delete
1.41 kB
metadata
license: apache-2.0
base_model: prism-ml/Ternary-Bonsai-4B-unpacked
base_model_relation: finetune
tags:
  - bitmtp
  - bonsai
  - ternary
  - multi-token-prediction
inference: false

BitMTP-Bonsai-4B-D8

Bonsai-4B checkpoint trained with the joint NTP + MTP objective at block size D=8.

Code and installation

Format and use

This repository contains a latent QAT checkpoint, configuration, and matching tokenizer. Convert it with BitMTP to group-128 ternary runtime weights before inference.

After installing BitMTP, run:

python -m bonsai_sft.download_model \
  --model bonsai_4b_d8 --output weights/BitMTP-Bonsai-4B-D8

python -m bonsai_sft.gpu.convert_bonsai_mtp_to_gpu \
  --src weights/BitMTP-Bonsai-4B-D8 --out weights/BitMTP-Bonsai-4B-D8-packed --quant ternary

python -m bonsai_sft.gpu.run_generation \
  --ckpt weights/BitMTP-Bonsai-4B-D8-packed --hf-ckpt weights/BitMTP-Bonsai-4B-D8 \
  --block-size 8 --mode hybrid --threshold 0.9 --min-chain 6 --draft-precision fp8 --verify-precision native \
  --chat --prompt 'Write a short greeting.' --max-new-tokens 128

The example targets H20.

Attribution

Created using Bonsai by Prism ML, based on Qwen3. BitMTP modifies the model through supervised fine-tuning and joint NTP + MTP training. The upstream Apache-2.0 license and notices are included in LICENSE and NOTICE.txt.