Instructions to use opencerebral/littlerock-1M with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use opencerebral/littlerock-1M with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="opencerebral/littlerock-1M")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("opencerebral/littlerock-1M")
model = AutoModelForCausalLM.from_pretrained("opencerebral/littlerock-1M", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use opencerebral/littlerock-1M with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "opencerebral/littlerock-1M"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "opencerebral/littlerock-1M",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/opencerebral/littlerock-1M

SGLang

How to use opencerebral/littlerock-1M with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "opencerebral/littlerock-1M" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "opencerebral/littlerock-1M",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "opencerebral/littlerock-1M" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "opencerebral/littlerock-1M",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use opencerebral/littlerock-1M with Docker Model Runner:
```
docker model run hf.co/opencerebral/littlerock-1M
```

littlerock-1M

A 1,002,720-parameter decoder-only base language model, pretrained from scratch on 5B tokens (40% FineWeb-Edu / 60% DCLM-baseline). No finetuning.

Architecturally a stock Llama, so it loads with plain transformers — no trust_remote_code, no custom modelling file:

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("opencerebral/littlerock-1M")
model = AutoModelForCausalLM.from_pretrained("opencerebral/littlerock-1M")

Results

lm-eval 0.4.12, zero-shot, acc_norm, float32, CUDA. ArithMark-3 via AxiomicLabs' bencharithmark-3.py (float32, default acc_norm).

Benchmark	Score
HellaSwag	26.43
ARC-Easy	30.98
ARC-Challenge	21.76
PIQA	52.23
ArithMark-3	25.20
Avg	32.56
Intelligence Index	2.29

ArithMark-3 sits at the 25% chance floor, which is the honest number for a 1M-parameter base model that was never trained on arithmetic.

A finetuned variant of this model reaches 40.40 on the public ArithMark-3 set and 35.5% on AxiomicLabs' private memorization test, but drops to 21.5% on their private variety set after the problems are rephrased — so its arithmetic does not generalize across wording, and it was declined for leaderboard ranking on that basis. Those weights and the full write-up are at opencerebral/littlerock-1M-arithmax. This repository is the base model, with no arithmetic finetuning at all.

Architecture


Parameters	1,002,720
Non-embedding	609,504
Layers	6
Hidden size	96
Heads	6 (head dim 16)
FFN hidden (SwiGLU)	224
Context	512
Vocab	4,096
Position encoding	RoPE
Norm	RMSNorm
Embeddings	tied

The 4,096-token vocabulary is a deliberate constraint: at d=96 a stock 50k tokenizer's embedding table alone would be several times the entire parameter budget. The tokenizer is a byte-level BPE with byte fallback (no UNK is possible), fitted on 400k documents sampled from this model's own pretraining mixture.

Training


Tokens	5,000,000,000
Mixture	40% FineWeb-Edu, 60% DCLM-baseline
Steps	19,073
Batch	262,144 tokens/step
Optimizer	AdamW (0.9, 0.95), wd 0.1
LR	3e-3 cosine → 3e-4, 50M-token warmup
Precision	bf16 autocast, fp32 master weights
Final val loss	3.6251 (ppl 37.6)
Hardware	1x RTX 3060, 1h 49m

Data was materialised to a fixed local binary before training (no streaming), with documents EOS-separated, and the token budget enforced exactly at 5,000,000,000.

Limitations

This is a ~1M-parameter base model. It is a next-token predictor with no instruction tuning and no alignment, it has very limited world knowledge, and it will produce fluent but frequently false text. It exists to measure what a 1M-parameter budget can do.

Sample (prompt in italics): The capital of France is a national world of fashion, a country and many other nationwide nationality. In this case, the public has been banned in the early 17th century…

Copyright & License

This project and all associated files (the "Work") are licensed under the Apache License, Version 2.0 (the "License"); you may not use this project except in compliance with the License. You may obtain a copy of the License at:

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Downloads last month: 329

Safetensors

Model size

1M params

Tensor type

F32

opencerebral
/

littlerock-1M