Sing VL S1

Sing VL S1 is a multimodal decision model built on Qwen3.5. It answers named questions over text or JSON and up to four images. Each question is encoded as a complete prompt. In text requests with multiple questions, the engine batches those prompts by length with a budget of 65,536 padded tokens per forward pass.

  • Choice selects an option and returns a probability for each candidate.
  • Noul returns the probability that a yes/no proposition is true.
  • Score rates the input against 2โ€“10 levels ordered from low to high. It returns the probability of each level and the expected level, computed as the sum of each zero-based level index times its probability.

Public Decision Index 0.3

The 65,536-token batch engine scored 69.75 Text (140,178 requests) and 73.74949871316771 Vision (12,210 requests) on the complete public suites. The table and per-dataset figures below document the original serial submission.

Panel Sing VL S1 (original submission) TypeSafe Jev 1.14 StartLux-Decision v1.1 4B Perplexity Decider Flash v1.2 (4B)
Text 69.76 60.70 67.8 60.1
Vision 73.75 โ€” 67.90 72.5

The submitted Vision result uses the 1,638,400-pixel cap. The older 73.94 result used a larger image budget and is retained below as historical context.

Per-dataset results

Show the Vision and Text breakdown

Historical Public Vision (11 datasets; larger image budget)

The breakdown below belongs to the historical 73.94 run. The submitted 1.6 MP-capped run scores 73.7495; its per-benchmark values are in the pinned vision-scores.json.

Dataset Non-Skill (Accuracy) Skill
CV-Bench 87.34 79.12
BLINK 72.11 56.83
RealWorldQA 82.17 70.87
CharXiv 86.65 82.21
InfographicVQA 96.31 95.08
Mind2Web 87.67 83.56
Winoground 93.12 86.25
KIE (CORD+FUNSD) 95.82 93.12
Moderation (Hateful Memes) 71.64 43.28
R-Bench-M 49.77 39.72
MMMU-Pro vision 50.17 42.85
Public Vision Index โ€” 73.94

Public Text (43 datasets)

Dataset Original metric Non-Skill Skill
BFCL Case exact accuracy 96.81 95.70
ToolRet nDCG@10 67.96 63.00
API-Bank Accuracy 84.45 84.15
BANKING77 Macro-F1 94.00 93.92
CLINC150+OOS Macro-F1 97.03 97.01
RouterBench โ€  Selected quality 79.83 52.71
Home appliance simulator Case exact accuracy 100.00 100.00
SGD/SGD-X โ€  Macro-F1 8.33 0.00
ContractNLI Macro-F1 84.05 76.94
ANLI Macro-F1 78.29 67.48
BPoMP * Accuracy 95.18 90.70
Humicroedit Accuracy 67.16 34.32
POP909-CL * Accuracy 80.60 79.74
cfcolor * Accuracy 70.14 44.04
MMLU โ€  Accuracy 82.93 77.24
GPQA Diamond Accuracy 53.06 37.41
ARC-Easy โ€  Accuracy 99.03 98.71
ARC-Challenge โ€  Accuracy 96.25 95.00
WinoGrande Accuracy 89.82 79.64
HellaSwag Accuracy 96.48 95.31
GSM8K Accuracy 73.84 68.68
ChessBench Accuracy 26.88 20.36
MuSR Accuracy 65.82 45.66
SATA-Bench Case exact accuracy 53.88 53.27
SimpleBench โ€  Accuracy 50.00 40.00
BRIGHT nDCG@10 47.51 40.62
Amazon ESCI Macro-F1 60.74 50.75
ACOS Per-review F1 64.56 63.43
FinEntity Macro-F1 95.19 92.93
iSarcasmEval Sarcasm F1 (A ยท English) 64.98 54.95
VAST Macro-F1 79.69 69.53
NLI4CT Macro-F1 86.41 73.56
CRUXEval Accuracy 78.07 65.21
CLadder Accuracy 94.72 89.44
HLE Accuracy 12.77 0.00
Habermas Machine Accuracy 63.60 47.18
PhishNChips Accuracy 85.55 71.10
MMLU-Pro Accuracy 68.40 64.46
BBH Accuracy 88.61 83.49
RAGTruth Hallucinated-class F1 83.49 65.77
HoVer Accuracy 89.25 78.50
When2Call MCQ Accuracy 90.61 87.48
New Yorker caption matching Accuracy 80.49 75.61
Public Text Index Weighted aggregate 77.26 69.76

โ€  Decision Index reports these datasets separately and excludes them from the Text Index.

Usage

Use Python 3.12 and hardware-compatible PyTorch and torchvision builds. From the model repository root, enter inference/ before running the examples. If you have a standalone copy of the inference code, run the examples from that directory and replace .. with the checkpoint path.

cd inference
python -m pip install -r requirements.txt

Python

from PIL import Image

from model import SingVLS1

model = SingVLS1("..", device="cuda:0")

# Text
text, _ = model.predict(
    state="The color is red. ",
    questions={
        "color": {
            "type": "choice",
            "instructions": "Which color is named?",
            "criteria": {"red": "Red", "blue": "Blue"},
        },
        "yesno": {"type": "noul", "instructions": "Is the named color red?"},
        "redness": {
            "type": "score",
            "instructions": "How clearly does the text identify the color as red?",
            "criteria": ["Not red", "Possibly red", "Clearly red"],
        },
    },
)
print(text["answers"])

# Images in blue-then-red order. Both state values work.
questions = {
    "which": {
        "type": "choice",
        "instructions": "Which image is red?",
        "criteria": {"first": "The first image", "second": "The second image"},
    }
}
with Image.open("examples/blue.png") as blue, Image.open("examples/red.png") as red:
    for state in ({}, "<image>\n<image>\n"):
        vision, _ = model.predict(state=state, questions=questions, images=[blue, red])
        print(vision["answers"])

With no <image> markers, the model inserts them before the state text. With explicit markers, use one per image. The markers match the images in list order.

Use the same score question format with images. The answer includes probabilities and a legend, both keyed by the zero-based level number as a string. For example, probabilities of 0.1, 0.3, and 0.6 on three levels give 0 ร— 0.1 + 1 ร— 0.3 + 2 ร— 0.6 = 1.5.

HTTP API

Start the FastAPI server. Send requests to POST http://127.0.0.1:8173/v1/systemone from another terminal on the same machine. Interactive API docs are at http://127.0.0.1:8173/docs.

python serve.py --model .. --port 8173 --image-root ./examples

Send a text request with an inline JSON body.

curl -fsS http://127.0.0.1:8173/v1/systemone \
  -H 'Content-Type: application/json' -d '{
  "model": "sing-vl-s1",
  "state": "The color is red. ",
  "questions": {
    "color": {
      "type": "choice",
      "instructions": "Which color is named?",
      "criteria": {"red": "Red", "blue": "Blue"}
    },
    "yesno": {"type": "noul", "instructions": "Is the named color red?"},
    "redness": {
      "type": "score",
      "instructions": "How clearly does the text identify the color as red?",
      "criteria": ["Not red", "Possibly red", "Clearly red"]
    }
  }
}'

For a Vision request, add an ordered images array. Its entries can be Base64 data URLs (data_url), raw Base64 strings (base64), or server-side file paths (path). Paths are relative to --image-root or absolute paths inside that directory. With the server command above, send the example filenames directly.

curl -fsS http://127.0.0.1:8173/v1/systemone \
  -H 'Content-Type: application/json' -d '{
  "model": "sing-vl-s1",
  "state": {},
  "images": ["blue.png", "red.png"],
  "questions": {
    "which": {
      "type": "choice",
      "instructions": "Which image is red?",
      "criteria": {"first": "The first image", "second": "The second image"}
    }
  }
}'

For the same request with explicit image markers, use "state": "<image>\n<image>\n".

The response puts results under answers using the supplied question IDs. The text example selects red and returns answers.redness.score on a 0โ€“2 scale. The Vision example selects second.

Decision Index compatibility and submission

Both the Public and private suites use inference.engine:D3Engine with the same model weights and the same default request-level batching. Download the model and DI adapter together.

python -m pip install "huggingface_hub>=0.34,<2"
hf download inclusionAI/SingVL_S1_4B --revision dd9fcbe2412068def01666b1d1a443e412f11214 --local-dir ./singvl-s1-4b
python -m pip install -r ./singvl-s1-4b/inference/requirements-di.txt

# With already-built and verified Public 0.3 suites:
python ./singvl-s1-4b/inference/run_decision_index.py \
  --panel text --suite-dir ./suite-0.3 --out ./runs/singvl-65k-text
python ./singvl-s1-4b/inference/run_decision_index.py \
  --panel vision --suite-dir ./work/vision --out ./runs/singvl-65k-vision

Use fresh output directories for the batch engine so that prior serial predictions are not resumed.

The launcher resolves its own code and checkpoint paths, so it can be called from any directory. The model argument is a downloaded local checkpoint path. The adapter applies the submitted prefix-bidirectional attention and decision readout.

See the complete reproduction guide for suite preparation, direct DI commands, HTTP serving, capacity limits and exact revisions. The original public results and code snapshot are pinned to 90ec7e159da6fa21d762f1f75876ea79049d0f4a. For the batch release, see the Public predictions and scores and inference/release.json for hashes and provenance.

The original serial submission scored Text 69.76 and Vision 73.7495. Both releases use a 1,638,400-pixel image cap. Private Text/Vision evaluation and official single-RTX-PRO-6000 latency qualification remain maintainer-run.

Downloads last month
1,588
Safetensors
Model size
5B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for inclusionAI/SingVL_S1_4B

Quantizations
2 models

Collection including inclusionAI/SingVL_S1_4B