Asteromorph/AsteroRank-Lepton-GGUF

AsteroRank-Lepton — GGUF

GGUF quantizations of Asteromorph/AsteroRank-Lepton, a 4B SOTA open-weights reranker for STEM and general search.

The AsteroRank blog post shows more information about AsteroRank model family which includes Lepton, Drachma which is better, and Talent which outperforms every reranker known to date. The blog post presents full evaluation of 3 models, the idea of context reranking, and how to prompt search agents to write good context.

Available quantizations

	BF16	Q8_0	Q6_K	Q5_K_M	Q5_K_S	Q4_K_M	Q4_K_S	IQ4_NL	IQ4_XS	Q3_K_L	Q3_K_M	Q3_K_S	Q2_K	IQ2_XS	IQ2_XXS
Size	9.7 GB	5.2 GB	4.0 GB	3.5 GB	3.4 GB	3.1 GB	2.9 GB	3.0 GB	2.9 GB	2.7 GB	2.5 GB	2.3 GB	2.1 GB	1.8 GB	1.7 GB

Files

• AsteroRank-Lepton-*.gguf — Model weights with the rerank chat template embedded (see table above for sizes)
• rerank.py — Reranker client for llama-server
• AsteroRank-Lepton.imatrix — Importance matrix (GGUF format) used to quantize the variants below BF16

Requirements

• A recent llama.cpp build with Qwen3.5 support (llama-server); no fork or patch is needed
• Python 3.8+ with jinja2

Usage

Download one quantization and the client:

hf download Asteromorph/AsteroRank-Lepton-GGUF AsteroRank-Lepton-Q4_K_M.gguf rerank.py --local-dir .

Start llama-server with one of the files (-ngl 99 offloads all layers to the GPU; -np sets the number of parallel requests):

llama-server -m AsteroRank-Lepton-Q4_K_M.gguf -ngl 99 -np 8 -c 20480 --port 8080

Then rerank:

from rerank import GGUFReranker

reranker = GGUFReranker("http://127.0.0.1:8080")

results = reranker.rerank(
    query="para-selective borylation aryl sulfonate ion pair",
    documents=[
        "A bulky bipyridine ligand on iridium steers C–H borylation of monosubstituted arenes to the para position through steric repulsion alone.",
        "Ion pairing between aryl sulfonate anions and tetrabutylammonium cations blocks the ortho and meta sites, so iridium-catalyzed C–H borylation proceeds with high para selectivity.",
        "Suzuki–Miyaura coupling of aryl boronic esters with aryl halides under palladium catalysis.",
    ],
    context=(
        "The article discusses para-selective iridium-catalyzed C–H borylation of aryl sulfonate salts "
        "using ion-pairing with bulky ammonium cations to shield ortho and meta positions, enabling high "
        "para selectivity."
    ),
)

for r in results:
    print(f"[{r['index']}] {r['relevance_score']:.4f}  {r['document'][:60]}")

Returns: List of dicts sorted by relevance descending — document, relevance_score in (0, 1), and index. Leave context out to rank on the query alone; top_n limits how many results come back.

The score is sigmoid(logit('yes') - logit('no')) at the end of the prompt in the embedded chat template, the same prompt and score that SGLang's /v1/rerank and vLLM's /rerank produce for AsteroRank-Lepton.

llama-server's built-in /v1/rerank takes only a query and documents, with no context field. rerank.py scores through /completion instead: it renders the embedded rerank template with your context and reads the yes/no log-probabilities of the next token.

License

CC BY-NC 4.0, like AsteroRank-Lepton, which is fine-tuned from Qwen3.5-4B (Apache 2.0); see NOTICE.

Contact

For any inquiry about AsteroRank-Lepton contact us via comm@asteromorph.com.