leapeto/mindcube-latent-ckpts

MindCube latent-reasoning checkpoints

LoRA adapters for Qwen/Qwen2.5-VL / Qwen2.5-VL-3B-Instruct that reason over MindCube spatial questions in a compressed latent vocabulary — the model emits <latent_start> <lat_…> <latent_end> <answer>X</answer> — instead of a natural-language chain of thought. Each <lat_i> is one BPE-merged unit standing for several base tokens, so the reasoning decodes in far fewer steps at no accuracy cost.

Training traces (text): leapeto/mindcube-latent-data. Images: Inevitablevalor/MindCube data.zip.

Checkpoints

Frozen-3B reference ≈ 37.8%; uncompressed full-CoT SFT baseline (plain_3b_ep5) = 46.6%. Accuracy is flat at ~49–51% across the whole 1.5×–4.05× compression sweep — compression is free down to the merge-table ceiling. Tinybench = MindCube's 1,050-question eval split, greedy two-stage decode.

adapter	base	target compression	M (<lat_i>)	median decode tok	tinybench acc
plain_3b_ep5	3B	1.0× (uncompressed full-CoT)	—	321	46.6%
c1.5mn_3b_ep5	3B	1.5×	1,961	207	49.7%
c2.0mn_3b_ep5	3B	2.0×	1,983	145	51.1%
c2.5mn_3b_ep5	3B	2.5×	2,082	124	51.0%
c3.0mn_3b_ep5	3B	3.0×	2,294	102	47.6%
c3.5mn_3b_ep5	3B	3.5×	3,011	88	50.1%
c2.0mn_7b_ep4	7B	2.0×	1,983	144	49.6%
ffrsn_3b_ep5	3B	2.0× (free-form reasoning target, no map)	—	33	51.9%
c4.0mn2_3b_ep5	3B	4.0×	9,081	76	49.3%
c4.5mn2_3b_ep5	3B	4.05× (merge ceiling)	11,895	73	49.1%
c2.0mn2_3b_ep5	3B	2.0× (regen-corpus anchor)	1,993	150	49.1%

The mn2 checkpoints are the high-compression extension: same recipe, trained on a regenerated (self-distilled) copy of the trace corpus. c2.0mn2 reproduces the original c2.0mn within noise (49.1% vs 51.1%) and exists only to confirm the corpora are comparable. Targets 4.5× and 5.0× both collapse to the same 4.05× vocabulary — the merge table runs out of recurring structure — so c4.5mn2 is the ceiling.

Usage

Each adapter dir bundles its tokenizer (defines the <lat_i> tokens) and — for the mn2 checkpoints — a latent_vocab.json sidecar mapping every <lat_i> back to its base-token span (needed only to expand the latent output into readable text or to retrain; plain inference does not need it). Decode in two greedy stages with a logit mask:

1. Latent — from the prompt + <latent_start>, restrict next-token logits to the <lat_i> units ∪ <latent_end>; stop on <latent_end> (or a cap ~400).
2. Answer — block the <lat_i> units and <latent_end>; decode <answer>X</answer>.

from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration
from peft import PeftModel
import torch

name = "c4.0mn2_3b_ep5"
proc = AutoProcessor.from_pretrained(f"leapeto/mindcube-latent-ckpts/{name}")
base = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    "Qwen/Qwen2.5-VL-3B-Instruct", dtype=torch.bfloat16, device_map="cuda")
base.resize_token_embeddings(len(proc.tokenizer))     # absorb the added <lat_i> units
model = PeftModel.from_pretrained(base, f"leapeto/mindcube-latent-ckpts/{name}").eval()
# then the two-stage masked greedy decode above.

M-RoPE gotcha: in a manual KV-cache decode loop, pass cache_position explicitly every step. Qwen2.5-VL derives decode positions from cache_position + rope_deltas; omitting it feeds every generated token at position 0 and produces repetition/garbage (transformers.generate / vLLM handle this automatically).

Prompt format: MindCube aug_cgmap_ffr_out. Benchmark: MindCube (arXiv:2506.21458).

---

Full-parameter checkpoints — the ground-truth-corpus line (docs 45–54)

Everything below is full fine-tuning (ViT included) of Qwen/Qwen2.5-VL-3B-Instruct on MindCube's ground-truth map+reasoning corpus (not LoRA, not self-distilled). Each directory is a complete HF model (4 fp32 safetensors shards, ~16 GB, + processor/tokenizer): load it directly with Qwen2_5_VLForConditionalGeneration.from_pretrained(dir) — no PEFT, no resize_token_embeddings. Eval protocol for every number: greedy, N=1,050 tinybench, best of ep4–6.

dir	what	tinybench acc
plain_s42_ep3/	stage-1 base: 3 epochs plain SFT on the GT corpus, seed 42. Warm-start base of every arm below and of the latent arms of docs 46–53.	65.9 (doc 45)
text_baselines_s777/answer_only_ep{4,5,6}/	+ 6 epochs, target = <answer> only (no reasoning). Rerun pass; first pass scored 70.19.	69.62 (ep6)
text_baselines_s777/concise_ep{4,5,6}/	+ 6 epochs, compact map + boilerplate-free sentences (1.91× fewer decode steps)	74.38 (ep5)
text_baselines_s777/cod_ep{4,5,6}/	+ 6 epochs, Chain-of-Draft targets (2.29×)	74.10 (ep5)
text_baselines_s777/tokenskip2.4_ep{4,5,6}/	+ 6 epochs, LLMLingua-2-pruned targets (2.32×)	66.76 (ep5)

Reference points at the same recipe and seed (checkpoints pruned, decodes in the data repo): plain +6 epochs = 70.95; latent BPE vocab cb 1.8× / 2.0× / 2.4× = 67.81 / 64.48 / 67.33. Report: docs/54_TEXT_COMPRESSION_BASELINES.md in the code repo (branch mindcube); training data under gt_corpus/text_baselines/ in leapeto/mindcube-latent-data. Recipe: warm-start from plain_s42_ep3, lr 1e-5 cosine, effective batch 512, bf16 FSDP, seed 777; on 32 GB cards use micro-batch 1 × grad-accum 128.

from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration
from huggingface_hub import snapshot_download
import torch
d = snapshot_download("leapeto/mindcube-latent-ckpts", allow_patterns=["text_baselines_s777/cod_ep5/*"]) + "/text_baselines_s777/cod_ep5"
proc = AutoProcessor.from_pretrained(d)
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(d, dtype=torch.bfloat16, device_map="cuda").eval()
# plain greedy generate on the MindCube `plain_cgmap_ffr_out` prompt; parse the letter from <answer>…</answer>