Instructions to use lavendercomputing/MiniCPM5-2B-GSQ-RCO-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use lavendercomputing/MiniCPM5-2B-GSQ-RCO-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf lavendercomputing/MiniCPM5-2B-GSQ-RCO-GGUF:IQ3_S # Run inference directly in the terminal: llama cli -hf lavendercomputing/MiniCPM5-2B-GSQ-RCO-GGUF:IQ3_S
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf lavendercomputing/MiniCPM5-2B-GSQ-RCO-GGUF:IQ3_S # Run inference directly in the terminal: llama cli -hf lavendercomputing/MiniCPM5-2B-GSQ-RCO-GGUF:IQ3_S
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf lavendercomputing/MiniCPM5-2B-GSQ-RCO-GGUF:IQ3_S # Run inference directly in the terminal: ./build/bin/llama cli -hf lavendercomputing/MiniCPM5-2B-GSQ-RCO-GGUF:IQ3_S
Use Docker
docker model run hf.co/lavendercomputing/MiniCPM5-2B-GSQ-RCO-GGUF:IQ3_S
- Llama new
- LM Studio
- Jan
- vLLM
How to use lavendercomputing/MiniCPM5-2B-GSQ-RCO-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "lavendercomputing/MiniCPM5-2B-GSQ-RCO-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "lavendercomputing/MiniCPM5-2B-GSQ-RCO-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/lavendercomputing/MiniCPM5-2B-GSQ-RCO-GGUF:IQ3_S
- Ollama
How to use lavendercomputing/MiniCPM5-2B-GSQ-RCO-GGUF with Ollama:
ollama run hf.co/lavendercomputing/MiniCPM5-2B-GSQ-RCO-GGUF:IQ3_S
- Unsloth Desktop
- Pi
How to use lavendercomputing/MiniCPM5-2B-GSQ-RCO-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf lavendercomputing/MiniCPM5-2B-GSQ-RCO-GGUF:IQ3_S
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "lavendercomputing/MiniCPM5-2B-GSQ-RCO-GGUF:IQ3_S" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use lavendercomputing/MiniCPM5-2B-GSQ-RCO-GGUF with Docker Model Runner:
docker model run hf.co/lavendercomputing/MiniCPM5-2B-GSQ-RCO-GGUF:IQ3_S
- Lemonade
How to use lavendercomputing/MiniCPM5-2B-GSQ-RCO-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull lavendercomputing/MiniCPM5-2B-GSQ-RCO-GGUF:IQ3_S
Run and chat with the model
lemonade run user.MiniCPM5-2B-GSQ-RCO-GGUF-IQ3_S
List all available models
lemonade list
- Hermes Agent
How to use lavendercomputing/MiniCPM5-2B-GSQ-RCO-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf lavendercomputing/MiniCPM5-2B-GSQ-RCO-GGUF:IQ3_S
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default lavendercomputing/MiniCPM5-2B-GSQ-RCO-GGUF:IQ3_S
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use lavendercomputing/MiniCPM5-2B-GSQ-RCO-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf lavendercomputing/MiniCPM5-2B-GSQ-RCO-GGUF:IQ3_S
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "lavendercomputing/MiniCPM5-2B-GSQ-RCO-GGUF:IQ3_S" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
MiniCPM5-2B โ GSQ-RCO GGUF
GSQ-RCO quantized GGUF build of the stock openbmb/MiniCPM5-2B weights, made for llama.cpp.
What this model is
openbmb/MiniCPM5-2B (developer: OpenBMB) is a dense 2B-class causal language model built on the standard LlamaForCausalLM architecture: 42 layers, GQA (16 query heads / 2 KV heads), 2048 hidden, 6144 intermediate, 130,560-token vocab, 131,072-token context. Total 2,516,756,480 parameters.
License: Apache-2.0 (inherited from the base model).
Files
| File | Size | BPW | Ladder (42-layer, mixed IQ) |
|---|---|---|---|
quant/MiniCPM5-2B-GSQ-RCO-IQ3_S.gguf |
1.14 GiB | 3.88 whole-file (3.48 selected MLP) | all attn_* @ IQ4_NL; per-tensor MLP mix of IQ1_MโIQ4_NL; token_embd/output @ IQ4_NL; norms fp32 |
Only the final quantized build is hosted here. The source checkpoint is linked below instead of being mirrored in this repository.
Source and provenance
openbmb BF16 master
โ BF16 GGUF conversion (no intermediate quantization)
โ this 3.5-tier build
Source:
openbmb/MiniCPM5-2B@ revisionf97400052a43d642bbc6e9975e2397e3ae6a6b52โ vendor BF16 safetensors, converted directly to BF16 GGUF for this build.Importance signal: instead of an imatrix, the per-tensor bitwidth assignment was optimized by the GSQ-RCO pipeline, calibrated against the stock weights on the
HuggingFaceFW/fineweb-educalibration set (32 sequences ร 256 tokens). Objective: cross-entropy โ measured NLL 4.888 / KL 2.109 against the BF16 reference at the assigned bitwidths.
Quantization
The bitwidth assignment was produced by the actual GSQ/RCO 3-bit recipe, run on the BF16 checkpoint, then emitted as a per-tensor mixed-precision GGUF using llama.cpp's IQ quant formats. This build targets the 3.5 tier.
The allocation spans every 2-D matrix: dense MLPs (ffn_gate, ffn_up, ffn_down), and attention (attn_q, attn_k, attn_v, attn_output). Fixed policies:
All attention weight matrices: IQ4_NL (42/42).
MLP matrices: per-tensor mixed ladder of IQ1_M, IQ2_XXS, IQ2_S, IQ2_XS, IQ3_XXS, IQ3_S, IQ4_XS, IQ4_NL โ selected by GSQ/RCO, averaging 3.48 bpw across MLP weights.
token_embd/output: IQ4_NL (tied-vocab-size embeddings, 130,560 ร 2048).1-D norms/biases: fp32.
Whole-file size is 1.14 GiB (3.88 bits per source weight element, 2.52B params).
The build was verified as an exact tensor-payload match to the optimized assignment (381 tensors), and smoke-tested for load and coherent output with pinned llama.cpp CPU inference (short chat prompts).
Usage
llama-cli -m quant/MiniCPM5-2B-GSQ-RCO-IQ3_S.gguf \
-p "Explain mixed-precision quantization." -ngl 99
Recommended sampling from the upstream card: temperature=1.0, top_p=0.95, min_p=0.0 (llama.cpp's default min_p=0.05 can cause repetition loops with this model).
Honest limits
This is a single 3.5-tier IQ3_S build; no 2-bit or 3.0-tier variants are hosted here.
These builds come directly from the vendor BF16 master; no intermediate quantization step is involved.
The smoke tests used short completion/chat prompts only; this is not a benchmark accuracy result.
The bitwidth ladder was optimized against the BF16 reference on a small web-text calibration sample; no published benchmark was re-run at this precision.
- Downloads last month
- 201
3-bit
Model tree for lavendercomputing/MiniCPM5-2B-GSQ-RCO-GGUF
Base model
openbmb/MiniCPM5-2B