Instructions to use nassimjp/MiniCPM5-2B-SFT-Pashto-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use nassimjp/MiniCPM5-2B-SFT-Pashto-Instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="nassimjp/MiniCPM5-2B-SFT-Pashto-Instruct")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("nassimjp/MiniCPM5-2B-SFT-Pashto-Instruct")
model = AutoModelForCausalLM.from_pretrained("nassimjp/MiniCPM5-2B-SFT-Pashto-Instruct", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use nassimjp/MiniCPM5-2B-SFT-Pashto-Instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "nassimjp/MiniCPM5-2B-SFT-Pashto-Instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nassimjp/MiniCPM5-2B-SFT-Pashto-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/nassimjp/MiniCPM5-2B-SFT-Pashto-Instruct

SGLang

How to use nassimjp/MiniCPM5-2B-SFT-Pashto-Instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "nassimjp/MiniCPM5-2B-SFT-Pashto-Instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nassimjp/MiniCPM5-2B-SFT-Pashto-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "nassimjp/MiniCPM5-2B-SFT-Pashto-Instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nassimjp/MiniCPM5-2B-SFT-Pashto-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use nassimjp/MiniCPM5-2B-SFT-Pashto-Instruct with Docker Model Runner:
```
docker model run hf.co/nassimjp/MiniCPM5-2B-SFT-Pashto-Instruct
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

🧬 MiniCPM5-2B-SFT-Pashto-Instruct

Pashto Instruction-Tuned Model (Tokenizer Surgery + SFT) 🚀🧬🇦🇫

MiniCPM5-2B-SFT-Pashto-Instruct is a specialized Pashto instruction-tuned model developed as part of the iPashto.ai initiative.

Two-Stage Development Pipeline:

Stage 1: Tokenizer Surgery 🔧
- Applied tokenizer surgery to openbmb/MiniCPM5-2B
- Added 46 missing Pashto/Urdu characters as single tokens
- Result: nassimjp/MiniCPM5-2B-Pashto
Stage 2: Supervised Fine-Tuning 🎯
- Fine-tuned the surgically-enhanced model on Pashto instruction data
- Used LoRA/QLoRA adapter techniques with very low learning rate (5e-6)
- Fully merged into 16-bit (bfloat16) safetensors format

📋 Model Overview

Property	Value
Developed by	Nassimjp / iPashto.ai
Base Model (Pre-Surgery)	`openbmb/MiniCPM5-2B`
Tokenizer Surgery Model	`nassimjp/MiniCPM5-2B-Pashto`
Final Model	`nassimjp/MiniCPM5-2B-SFT-Pashto-Instruct`
Original Vocab Size	130,560
Vocab Size After Surgery	130,606
Atoms Added	46
Language	Pashto (`ps`), English (`en`)
License	Apache-2.0
Architecture	Causal LM (MiniCPM Architecture)
Precision	16-bit (`bfloat16` Merged)
Training Method	Supervised Fine-Tuning (SFT) via LoRA/QLoRA
Training Samples	< 10,000 Pashto instruction pairs
Sequence Length	Up to 40K tokens per sample
Learning Rate	5e-6 (very low, conservative fine-tuning)
Hardware	NVIDIA RTX 4070 Ti SUPER (16GB VRAM)

🎯 Why This Model Exists

The Problem:

❌ Pashto is a low-resource language with limited AI support
❌ Original MiniCPM5-2B tokenizer splits many Pashto characters into multiple tokens
❌ Inefficient tokenization → poor instruction following
❌ No open-weight Pashto instruction-tuned models available

The Solution (Two-Stage Approach):

✅ Stage 1: Tokenizer surgery — added 46 missing Pashto/Urdu atoms as single tokens
✅ Stage 2: Fine-tuned on < 10K high-quality Pashto instruction pairs with conservative LR (5e-6)
✅ Supports long-form generation (up to 40K tokens per sample)
✅ Preserves Pashto script integrity
✅ Openly available for the community

📊 Stage 1: Tokenizer Surgery Report

✅ Existing Single-Token Atoms (20)

Atom	ID	Atom	ID
`ا`	20541	`د`	57692
`ب`	79922	`ر`	37101
`پ`	120234	`ز`	124105
`ت`	55761	`س`	75848
`ح`	118797	`ش`	107172
`ع`	84439	`ک`	77297
`ف`	79312	`ل`	29673
`ق`	74411	`م`	36367
`ن`	35964	`ه`	64732
`و`	41636	`ی`	46703

⚠️ Split Atoms — FIXED via Surgery (46)

#	Atom	Old IDs	New ID
1	`ښ`	[172, 270]	130560
2	`څ`	[172, 249]	130561
3	`ځ`	[172, 245]	130562
4	`ڼ`	[172, 142]	130563
5	`ږ`	[172, 266]	130564
6	`ډ`	[172, 253]	130565
7	`ټ`	[171, 142]	130566
8	`ړ`	[172, 263]	130567
9	`ې`	[173, 260]	130568
10	`ۍ`	[173, 257]	130569
11	`ګ`	[172, 126]	130570
12	`ث`	[170, 126]	130571
13	`ج`	[170, 127]	130572
14	`چ`	[172, 250]	130573
15	`خ`	[170, 128]	130574
16	`ذ`	[170, 130]	130575
17	`ص`	[170, 135]	130576
18	`ض`	[170, 136]	130577
19	`ط`	[170, 137]	130578
20	`ظ`	[170, 138]	130579
21	`غ`	[170, 140]	130580
22	`ئ`	[170, 121]	130581
23	`ے`	[173, 262]	130582
24	`ۀ`	[173, 244]	130583
25	`ٹ`	[171, 139]	130584
26	`ڈ`	[172, 252]	130585
27	`ڑ`	[172, 261]	130586
28	`ں`	[172, 140]	130587
29	`ھ`	[172, 144]	130588
30	`گ`	[172, 129]	130589
31	`أ`	[170, 118]	130590
32	`إ`	[170, 120]	130591
33	`آ`	[170, 117]	130592
34	`ؤ`	[170, 119]	130593
35	`ء`	[170, 116]	130594
36	`ٱ`	[171, 131]	130595
37	`۰`	[173, 130]	130596
38	`۱`	[173, 131]	130597
39	`۲`	[173, 132]	130598
40	`۳`	[173, 133]	130599
41	`۴`	[173, 134]	130600
42	`۵`	[173, 135]	130601
43	`۶`	[173, 136]	130602
44	`۷`	[173, 137]	130603
45	`۸`	[173, 138]	130604
46	`۹`	[173, 139]	130605

🔬 Embedding Forensics (After Surgery)

Metric	Value
Mean embedding norm	0.882395
Max cosine similarity	0.27078322
Most Similar Pair	`ب` ↔ `م` (cosine = 0.27078322)

✅ No exact duplicate embeddings detected

📊 Stage 2: Training Highlights & Loss Metrics

The fine-tuning run was executed locally using an NVIDIA RTX 4070 Ti SUPER (16GB VRAM) across 8,952 steps.

Key Training Parameters

Parameter	Value
Learning Rate	5e-6 (very low, conservative)
Training Steps	8,952
Training Samples	< 10,000 Pashto instruction pairs
Max Sequence Length	40K tokens
Tokenizer	Surgically enhanced (46 new Pashto tokens)

Loss Metrics

Metric	Value
Initial Training Loss	4.61
Final Training Loss	~0.54
Evaluation Loss	~0.91 (no overfitting)
Gradient Norm	1.4 - 2.2 (stable updates)

Why 5e-6 Learning Rate?

Using a very low learning rate (5e-6) was a deliberate choice:

✅ Preserves surgically-added token embeddings — prevents catastrophic forgetting of the 46 new tokens
✅ Conservative adaptation — the base model already has strong language capabilities; we only need to teach it to follow instructions
✅ Prevents overfitting — with < 10K samples, aggressive LR would cause overfitting
✅ Stable gradient norms — maintained 1.4-2.2 throughout training
✅ Excellent convergence — loss dropped from 4.61 to 0.54 without instability

🚀 Quickstart & Usage

1. Using Transformers (Python)

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "nassimjp/MiniCPM5-2B-SFT-Pashto-Instruct"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)

# Example 1: Pashto Question
prompt = "د پښتو ژبې او مصنوعي ځیرکتیا پرمختګ څه اهمیت لري؟"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

outputs = model.generate(
    **inputs,
    max_new_tokens=256,
    temperature=0.7,
    do_sample=True,
    pad_token_id=tokenizer.eos_token_id
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

# Example 2: Check that Pashto atoms are single tokens
test_chars = "ښڅځڼږډټړېۍګ"
for char in test_chars:
    ids = tokenizer.encode(char, add_special_tokens=False)
    print(f"'{char}' → {ids}")  # All should be single tokens!

2. Long-Form Generation (up to 40K tokens)

long_prompt = "د افغانستان تاریخ..."  # Can be up to 40K tokens
inputs = tokenizer(long_prompt, return_tensors="pt", truncation=True, max_length=40000)

3. Streaming Generation

from transformers import TextStreamer

streamer = TextStreamer(tokenizer, skip_prompt=True)
outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    temperature=0.7,
    streamer=streamer
)

4. Batch Inference

prompts = [
    "پښتو ژبه څنګه زده کړم؟",
    "د افغانستان پلازمینه څه ده؟",
    "پښتو ادب په اړه وږغاړئ"
]

inputs = tokenizer(prompts, padding=True, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=128, temperature=0.7)

5. Running via llama.cpp / GGUF

./llama-cli -m ./MiniCPM5-2B-SFT-Pashto-Instruct.F16.gguf \
  -p "د پښتو ژبې او مصنوعي ځیرکتیا پرمختګ څه اهمیت لري؟" \
  -n 256 \
  -t 8

⚙️ Training Details

Dataset

Size: < 10,000 Pashto instruction-response pairs
Format: Instruction + Response pairs
Language: Pashto (ps) with some English (en) examples
Max Length: Up to 40K tokens per sample

Training Configuration

Parameter	Value
Method	LoRA/QLoRA
Precision	bfloat16
Optimizer	AdamW
Learning Rate	5e-6 (very low, conservative)
LR Scheduler	Cosine decay
Warmup Steps	100-200
Batch Size	Tuned for 16GB VRAM
LoRA Rank	16-32
LoRA Alpha	32-64
Dropout	0.05

Hardware

GPU: NVIDIA RTX 4070 Ti SUPER
VRAM: 16GB
Training Duration: Several days (8,952 steps)

📂 Model Files

MiniCPM5-2B-SFT-Pashto-Instruct/
├── config.json                      ~1 KB
├── generation_config.json           ~1 KB
├── model.safetensors                ~4-5 GB
├── tokenizer.json                   ~9 MB (surgically enhanced)
├── tokenizer_config.json            ~1 KB
├── special_tokens_map.json          ~1 KB
└── training_logs.json               Training metrics

⚠️ Important Notes

Note	Description
Tokenizer	Surgically enhanced — all 66 Pashto/Urdu atoms are single tokens
Base Model	Built on `nassimjp/MiniCPM5-2B-Pashto` (surgically modified)
Language	Primarily Pashto (`ps`), with some English instruction examples
Size	~2B parameters, ~4-5 GB in bfloat16
Long Context	Supports up to 40K tokens per sample
Learning Rate	5e-6 — very conservative to preserve token embeddings
Fine-tuning	LoRA/QLoRA merged weights (not just adapter)

🧪 Testing Results

All 66 Pashto/Urdu atoms are now single tokens:

Category	Count	Status
Pashto-Specific	11	✅ All single tokens
Shared Arabic/Persian	30	✅ All single tokens
Positional Forms	3	✅ All single tokens
Urdu/South Asian	6	✅ All single tokens
Arabic Variants	6	✅ All single tokens
Eastern Arabic Digits	10	✅ All single tokens
TOTAL	66	✅ 100% Coverage

🔗 Related Resources

Base Model: openbmb/MiniCPM5-2B
Tokenizer Surgery Model: nassimjp/MiniCPM5-2B-Pashto
Original Paper: MiniCPM
GitHub: OpenBMB/MiniCPM

📝 Citation

If you use this model, please cite:

@misc{minicpm5-pashto-instruct,
  author = {Nassimjp},
  title = {MiniCPM5-2B-SFT-Pashto-Instruct: Pashto Instruction-Tuned Model with Tokenizer Surgery},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/nassimjp/MiniCPM5-2B-SFT-Pashto-Instruct}}
}

@article{minicpm2024,
  title={MiniCPM: Unveiling the Potential of Small Language Models},
  author={Hu, Shengding and Ding, Ning and others},
  journal={arXiv preprint arXiv:2404.06395},
  year={2024}
}

📄 License

This model is released under the Apache License 2.0.

🤝 Acknowledgements

OpenBMB for developing MiniCPM5-2B
Hugging Face for the transformers library and model hosting
iPashto.ai community for dataset contributions and testing

🌐 Project Context

Developed under the iPashto.ai framework dedicated to advancing Pashto Natural Language Processing (NLP), specialized datasets, and open-weight AI accessibility for low-resource languages.