linuxid10t/G9v3-39A5B-GGUF

G9v3-39A5B GGUF

This repository contains unofficial BF16 and Q4_K_M GGUF conversions of ai9stars/G9v3-39A5B, a preview Mixture-of-Experts language model from AI9Stars.

The conversion was created and tested with a llama.cpp implementation of the G9v3 architecture available on the feature/g9v3-support branch. Until G9v3 support is merged upstream, standard llama.cpp builds will not load this model.

Available files

File	Format	Size	SHA-256
G9v3-39A5B-BF16.gguf	BF16	72.63 GiB (77,988,959,200 bytes)	016c0c8a708fc5d4e2378df1eb9b5f089d7e91b581914e2565fb819bcebe9196
G9v3-39A5B-Q4_K_M.gguf	Q4_K_M	21.94 GiB (23,559,222,240 bytes)	0c28e98eef634ca2304a492c7b2c92d8599189ae4409177f7edf50da5b082eb0

Choose the file according to your hardware and quality requirements:

Format	Description	Recommended system RAM
BF16	Highest-fidelity GGUF. Source BF16 tensors are preserved, along with required F32 tensors.	96 GiB or more
Q4_K_M	Mixed K-quant that keeps selected tensors at higher precision; practical for local CPU inference.	32 GiB or more

The memory recommendations include headroom for the runtime, context, and operating system. Actual requirements vary with context size and llama.cpp settings.

Model details

Property	Value
Architecture	G9v3ForCausalLM, sparse MoE
Parameters	Approximately 39B total / 5B active per token
Layers	38 (1 dense layer followed by 37 MoE layers)
Hidden size	2,048
Attention	32 query heads, 2 key/value heads, head dimension 128
Experts	320 routed experts, 32 selected per token, plus 1 shared expert
Expert intermediate size	512
Vocabulary	130,560 tokens
Maximum context	131,072 tokens
Languages	English and Chinese
Features	Think/No-Think modes, tool calling, long context

The base model is a preview release. Refer to the ai9stars/G9v3-39A5B model card for the original description and usage guidance.

llama.cpp setup

Build the G9v3-enabled branch:

git clone --branch feature/g9v3-support --single-branch \
  https://github.com/linuxid10t/llama.cpp.git
cmake -S llama.cpp -B llama.cpp/build -DCMAKE_BUILD_TYPE=Release
cmake --build llama.cpp/build --target llama-cli llama-server -j

Command line

Both GGUF files contain the model's chat template. Start an interactive conversation with the Q4_K_M file using:

./llama.cpp/build/bin/llama-cli \
  -m G9v3-39A5B-Q4_K_M.gguf \
  -c 4096 -t 32 -cnv

For BF16, use the same command with the other filename:

./llama.cpp/build/bin/llama-cli \
  -m G9v3-39A5B-BF16.gguf \
  -c 4096 -t 32 -cnv

Start with a modest context such as 4,096 tokens, then increase it according to your available memory and workload.

OpenAI-compatible server

./llama.cpp/build/bin/llama-server \
  -m G9v3-39A5B-Q4_K_M.gguf \
  -c 4096 -t 32 -tb 32 \
  --host 127.0.0.1 --port 8080

Replace the model filename with G9v3-39A5B-BF16.gguf to serve the BF16 version on a machine with sufficient memory.

Example request:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "G9v3-39A5B-Q4_K_M.gguf",
    "messages": [{"role": "user", "content": "Who are you?"}],
    "temperature": 0.7,
    "top_p": 0.95,
    "max_tokens": 256
  }'

Bind the server to a non-loopback address only after configuring appropriate authentication and network controls.

Conversion provenance

The source BF16 checkpoint was converted to GGUF and then quantized with:

python convert_hf_to_gguf.py G9v3-39A5B \
  --outfile G9v3-39A5B-BF16.gguf \
  --outtype bf16

./build/bin/llama-quantize \
  G9v3-39A5B-BF16.gguf \
  G9v3-39A5B-Q4_K_M.gguf \
  Q4_K_M 32

Conversion environment:

• llama.cpp base commit: 9d57ce456 (b10436)
• G9v3 support commit: 77b13da1a
• GGUF version: V3
• Tensor count: 568 in each file
• BF16 weight size: 74,371.15 MiB, 16.01 bits per weight
• Q4_K_M weight size: 22,462.90 MiB, 4.84 bits per weight

Validation and performance

The Q4_K_M model was tested end-to-end with both llama-cli and llama-server on a CPU-only system. The performance figures below apply only to Q4_K_M.

Hardware	Context	Threads	Prompt processing	Generation
AMD Ryzen 9 9950X, 64 GiB RAM, CPU only	4,096	32	50.18–57.9 tokens/s	10.0–10.03 tokens/s

The exact rate will vary with prompt length, thread count, memory speed, build options, and operating system.

Validation performed:

• Successful load and text generation from the Q4_K_M file
• Successful /v1/chat/completions request with separate reasoning content
• BF16 GGUF metadata and all 568 tensor shapes loaded successfully
• BF16 file successfully used as the source for Q4_K_M quantization
• Converter dry run passed against all 15 source checkpoint shards after rebasing onto llama.cpp b10436
• Synthetic G9v3 CPU architecture test: zero numerical error
• GGUF serialization round-trip test: passed

The BF16 file was not generation-benchmarked on the 64 GiB validation machine; it requires a system with more memory for representative testing.

Sampling

The base model authors recommend the following settings:

Mode	Temperature	Top-p
Think	1.0	0.95
No Think	0.7	0.95

Limitations

• Q4_K_M is a lossy quantization and may perform differently from the BF16 file and original checkpoint.
• The BF16 GGUF preserves source precision but has much higher memory and storage requirements.
• G9v3 is a preview model and may produce inaccurate, biased, or unsafe output.
• Outputs should be independently reviewed before use in medical, legal, financial, safety-critical, or other high-stakes settings.
• This GGUF currently depends on the linked G9v3-enabled llama.cpp branch.
• Long-context capability was not benchmarked as part of this conversion test.

License and attribution

The original model is released by AI9Stars under the Apache License 2.0. This conversion retains that license. See the base model repository for its complete license terms and attribution information.

This conversion is independently produced and is not an official AI9Stars release.