aufklarer/GLiNER2.5-Decide-340M-MLX-8bit

GLiNER2.5-Decide 340M MLX INT8 (mixed)

MLX conversion of fastino/GLiNER2.5-Decide for native inference on Apple Silicon with speech-swift. Given text and a label schema, it returns a probability for every label (single-label classification) or spans of the original text (entity extraction). It does not generate text.

Converted from the upstream checkpoint at revision 7ee5da4c2415e32259bcdc0b1a7367c32ce8d6f6.

Model

Architecture	DeBERTa-v3-large encoder, GLiNER2 span and classification heads
Parameters	340M (upstream figure)
Precision	INT8 (mixed): 8-bit affine weights (group 64) for encoder matrices and token embeddings; float16 heads, norms, relative embeddings and activations
Weights	567 MB, SHA-256 61e91848b390fbf23dcbe76419d835644f76afefe554de3e6d1789d5e1e4f282
Context	512 encoded tokens (schema and text together)
Language	English

Files

File	Size	Contents
LICENSE	11.4 kB	Apache-2.0 (model weights)
LICENSE-gliner2-mlx	1.1 kB	MIT (reference conversion code)
LICENSE-mlx-reference	1.1 kB	MIT (MLX affine quantization equations used by the INT8 converter)
UPSTREAM_README.md	16.9 kB	Original Fastino model card
config.json	0.6 kB	GLiNER2 head configuration
encoder_config/config.json	0.9 kB	DeBERTa-v3-large encoder configuration
export.json	10.2 kB	Source revision, format and weight hash
quantization.json	9.3 kB	Quantized tensor list, bits and group size
special_tokens_map.json	2.4 kB	Special tokens
tokenizer.json	8.3 MB	Unigram tokenizer with GLiNER schema tokens
tokenizer_config.json	3.4 kB	Tokenizer settings
weights.safetensors	567.0 MB	Encoder and span/classification head weights

Performance

Apple M5 Pro, idle machine, one process per variant, model loaded once. Median of full requests including tokenization: 16 routing cases with six labels, 8 entity cases with two labels, five timed calls each after five warmups.

Routing (median)	Extraction (median)	Peak process memory
7.6 ms	8.9 ms	0.85 GB

Fidelity to the upstream PyTorch model on 24 reference cases: identical labels, spans and offsets; largest confidence difference 0.0060 (gate 0.02).

Usage

Swift (speech-swift GLiNER module):

import GLiNER

let model = try await GLiNER.fromPretrained(variant: .int8)
let choices = try model.classify(
    "Remind me to call Dad at six PM.",
    labels: ["create_reminder", "send_message", "other"]
)
let spans = try model.extractEntities(
    "Remind me to call Dad at six PM.",
    labels: ["person", "time"]
)

CLI:

speech gliner classify "Remind me to call Dad at six PM." --variant int8 \
    --labels create_reminder,send_message,other
speech gliner extract "Remind me to call Dad at six PM." --variant int8 \
    --labels person,time --json

Probabilities are model scores, not guarantees. Entity spans are mentions: dates and times are returned as text for your code to normalize. Offsets are UTF-16.

Source and licenses

• Model: fastino/GLiNER2.5-Decide, Apache-2.0 (LICENSE). The weights are converted to MLX layout without retraining; the original card is kept as UPSTREAM_README.md.
• Conversion follows gliner2-mlx by Andrew Chen Wang, MIT (LICENSE-gliner2-mlx).
• GLiNER2 paper: arXiv:2507.18546.