MiniCPM5-2B โ€” GSQ-RCO GGUF

GSQ-RCO quantized GGUF build of the stock openbmb/MiniCPM5-2B weights, made for llama.cpp.

What this model is

openbmb/MiniCPM5-2B (developer: OpenBMB) is a dense 2B-class causal language model built on the standard LlamaForCausalLM architecture: 42 layers, GQA (16 query heads / 2 KV heads), 2048 hidden, 6144 intermediate, 130,560-token vocab, 131,072-token context. Total 2,516,756,480 parameters.

License: Apache-2.0 (inherited from the base model).

Files

File Size BPW Ladder (42-layer, mixed IQ)
quant/MiniCPM5-2B-GSQ-RCO-IQ3_S.gguf 1.14 GiB 3.88 whole-file (3.48 selected MLP) all attn_* @ IQ4_NL; per-tensor MLP mix of IQ1_Mโ€“IQ4_NL; token_embd/output @ IQ4_NL; norms fp32

Only the final quantized build is hosted here. The source checkpoint is linked below instead of being mirrored in this repository.

Source and provenance

openbmb BF16 master
   โ†’ BF16 GGUF conversion (no intermediate quantization)
   โ†’ this 3.5-tier build
  • Source: openbmb/MiniCPM5-2B @ revision f97400052a43d642bbc6e9975e2397e3ae6a6b52 โ€” vendor BF16 safetensors, converted directly to BF16 GGUF for this build.

  • Importance signal: instead of an imatrix, the per-tensor bitwidth assignment was optimized by the GSQ-RCO pipeline, calibrated against the stock weights on the HuggingFaceFW/fineweb-edu calibration set (32 sequences ร— 256 tokens). Objective: cross-entropy โ€” measured NLL 4.888 / KL 2.109 against the BF16 reference at the assigned bitwidths.

Quantization

The bitwidth assignment was produced by the actual GSQ/RCO 3-bit recipe, run on the BF16 checkpoint, then emitted as a per-tensor mixed-precision GGUF using llama.cpp's IQ quant formats. This build targets the 3.5 tier.

The allocation spans every 2-D matrix: dense MLPs (ffn_gate, ffn_up, ffn_down), and attention (attn_q, attn_k, attn_v, attn_output). Fixed policies:

  • All attention weight matrices: IQ4_NL (42/42).

  • MLP matrices: per-tensor mixed ladder of IQ1_M, IQ2_XXS, IQ2_S, IQ2_XS, IQ3_XXS, IQ3_S, IQ4_XS, IQ4_NL โ€” selected by GSQ/RCO, averaging 3.48 bpw across MLP weights.

  • token_embd / output: IQ4_NL (tied-vocab-size embeddings, 130,560 ร— 2048).

  • 1-D norms/biases: fp32.

Whole-file size is 1.14 GiB (3.88 bits per source weight element, 2.52B params).

The build was verified as an exact tensor-payload match to the optimized assignment (381 tensors), and smoke-tested for load and coherent output with pinned llama.cpp CPU inference (short chat prompts).

Usage

llama-cli -m quant/MiniCPM5-2B-GSQ-RCO-IQ3_S.gguf \
  -p "Explain mixed-precision quantization." -ngl 99

Recommended sampling from the upstream card: temperature=1.0, top_p=0.95, min_p=0.0 (llama.cpp's default min_p=0.05 can cause repetition loops with this model).

Honest limits

  • This is a single 3.5-tier IQ3_S build; no 2-bit or 3.0-tier variants are hosted here.

  • These builds come directly from the vendor BF16 master; no intermediate quantization step is involved.

  • The smoke tests used short completion/chat prompts only; this is not a benchmark accuracy result.

  • The bitwidth ladder was optimized against the BF16 reference on a small web-text calibration sample; no published benchmark was re-run at this precision.

Downloads last month
201
GGUF
Model size
3B params
Architecture
llama
Hardware compatibility
Log In to add your hardware

3-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for lavendercomputing/MiniCPM5-2B-GSQ-RCO-GGUF

Quantized
(144)
this model

Dataset used to train lavendercomputing/MiniCPM5-2B-GSQ-RCO-GGUF