Visual Jev blue and teal Victory V logo

visual-jev-8b

Accurate and Efficient Decisions from Shared Visual Context
Guanxu Yu · Yuhang Yao

🌐 Website  ·  📄 Paper  ·  💻 Code & quickstart  ·  🤗 visual-jev-4b

One image. Independent questions. Shared compute.

An answer-supervised LoRA adapter for Qwen/Qwen3-VL-8B-Instruct. Encode an image and shared context once, run independent question branches together, and return probabilities over the choices in each request. Scores come from the backbone’s existing LM head; no extra decision head is included.

The 8B counterpart uses the same data, objective, LoRA settings, and step budget as visual-jev-4b, with a larger backbone.

Measured decision quality

89.3% macro accuracy · 0.896 AUROC · 0.285 calibrated NLL

Visual Jev 8B compared with TypeSafe Jev and OpenAI Decisions on the same held-out questions

Four benchmarks, equal-weight macro averages, November 2026. Visual Jev uses task SFT; API training data are unknown. TypeSafe Jev leads on ECE and TextVQA. NLL follows task-local temperature scaling. Results and protocol ↗

Full comparison and evaluation protocol

Test images only (70% of each benchmark's images, split by a fixed hash of the image id); the same questions and option order for every system; each system gets its own temperature fitted on the same calibration images. TallyQA is the full test build with GQA-training duplicates removed. API probabilities are returned to two decimals and floored at 0.005; a failed request counts as a uniform guess. NLL and ECE after temperature scaling; AUROC of the top probability for correct vs. wrong answers. Macro is the equal-weight mean of the four benchmarks.

System GQA SNLI-VE TextVQA TallyQA Macro NLL (TS) ECE (TS) AUROC
Visual Jev 8B 0.919 0.818 0.979 0.855 0.893 0.285 0.021 0.896
Visual Jev 4B 0.916 0.808 0.975 0.841 0.885 0.309 0.025 0.890
TypeSafe Jev 0.885 0.713 0.982 0.851 0.858 0.361 0.013 0.870
OpenAI Decisions 0.834 0.732 0.955 0.818 0.835 0.440 0.020 0.833

We sent the same test questions, with the options in the same order, to two hosted APIs that also return one probability per option: OpenAI's GPT-6 Luna Decisions (openai/gpt-6-luna-decisions-20261006, through OpenRouter) and TypeSafe Jev (jev-1.14.0, whose image input TypeSafe marks as not yet production quality), queried in November 2026. Every system gets its own temperature, fitted on the same calibration images.

Read it with three caveats:

  • The adapters were trained on GQA and SNLI-VE training data; API training data are unknown. Most of the adapters' lead comes from those two tasks. On the two task families no post-training saw (†), the systems are close, and TypeSafe Jev has the best TextVQA accuracy.
  • Both APIs return probabilities rounded to two decimals; we floor them at 0.005 before taking logs. A request that failed (1.9% of questions for TypeSafe Jev, which rejected some image sizes; 0.3% for GPT-6 Luna, all refusals) counts as a uniform guess rather than being dropped.
  • TypeSafe Jev is not deterministic: sending the identical request again changed its top answer for 0.8% of GQA, 2.7% of SNLI-VE and 10.3% of short-TallyQA questions. GPT-6 Luna returned identical probabilities on every repeat.

Source: checked-in API comparison data.

Try the released model

CUDA or Apple Silicon MPS · Inference only · No training dataset required

git clone https://github.com/guanxuyu-sv/Visual-Jev.git
cd Visual-Jev
python3 -m venv .venv && source .venv/bin/activate
pip install torch torchvision
pip install -r code/requirements.txt
python code/examples/quickstart.py \
    --model Qwen/Qwen3-VL-8B-Instruct \
    --adapter guanxuyu/visual-jev-8b-answer-sft \
    --image assets/figures/demo_bottle.jpg \
    --request-file code/examples/bottle_questions.json --device auto

The runner downloads the backbone and this adapter on first use. --device auto selects CUDA first, then Apple MPS. For CUDA, install a compatible PyTorch build before the remaining dependencies; on Apple Silicon, use the regular macOS build shown above.

The request supplies named questions, shared state, and 2–16 candidate choices per question. Evaluation covers at most eight choices. The runner caches the shared input and returns a choice and probabilities for each question.

Explore the project’s interactive 4B reference examples ↗

Calibrating and acting on probabilities

Macro over the same four benchmarks:

Original 8B This adapter
NLL, raw 0.634 0.298
NLL, after temperature scaling 0.376 0.285
ECE (15 bins), after temperature scaling 0.022 0.021
AUROC, correct vs. wrong by top probability 0.851 0.896

Post-training mainly improves ranking: confident answers are more often right (AUROC, which does not depend on accuracy). It does not make the raw probabilities calibrated; the fitted temperatures are all above 1, i.e. the model is overconfident. A single temperature per task fixes that. temperatures.json lists the temperature fitted for each seed and benchmark on a separate calibration split; apply it as softmax(logits / T) over the supplied options. For a new task, fit your own temperature on a few hundred labelled questions from that task: in our measurements on the 4B adapter, temperatures transferred from other tasks calibrated clearly worse.

Two further cautions before acting on the probabilities with a threshold:

  • Option order matters. On the 4B adapter, listing the same options in a different order changed some thresholded decisions, at a rate well above that of changing the inference backend; we did not repeat that study at 8B. Fix the option order you deploy with, and audit it.
  • Probabilities are relative to the options you supply; the model has no built-in way to say that the right answer is missing. Add an option such as incorrect_question or not enough evidence if you need one.

Research details

Benchmark breakdown · full TallyQA test build

Test images only (70% of each benchmark's images, split by a fixed hash of the image id); mean ± standard deviation over three seeds. † marks a task family no post-training saw.

Benchmark Original 8B This adapter 4B adapter n test
GQA-Choice 0.885 0.919 ± 0.001 0.916 5,447
SNLI-VE 0.695 0.818 ± 0.004 0.808 1,368
TextVQA-Choice † 0.979 0.979 ± 0.003 0.975 1,742
TallyQA-Choice † 0.853 0.855 ± 0.010 0.841 3,681
Macro 0.853 0.893 ± 0.003 0.885

The gain sits on the two task families represented in training; on the two held-out families the adapter matches the untrained backbone. The TallyQA column uses the full TallyQA test build (natural count distribution, with the images that are copies of GQA training images removed), distinct from the short count-balanced TallyQA subset in the original ablation evaluation. Compare this table with the full-build API comparison on the 4B card, not its original ablation table.

Load the adapter and reproduce the readout
import torch
from transformers import AutoModelForImageTextToText, AutoProcessor
from peft import PeftModel

BASE = "Qwen/Qwen3-VL-8B-Instruct"
ADAPTER = "guanxuyu/visual-jev-8b-answer-sft"   # seed 0 lives at the root

model = AutoModelForImageTextToText.from_pretrained(BASE, dtype=torch.bfloat16,
                                                    device_map="cuda")
model = PeftModel.from_pretrained(model, ADAPTER).eval()
processor = AutoProcessor.from_pretrained(BASE)

# the other two seeds:
#   PeftModel.from_pretrained(model, ADAPTER, subfolder="seed1")

The scores are candidate-token logits read from the backbone's own LM head at one fixed readout position, not free-form generation. Prompt layout, readout position and candidate tokenisation determine the result; use the prompt construction in the code repository to reproduce the numbers above.

Training configuration and data
Base Qwen/Qwen3-VL-8B-Instruct, vision tower frozen (exclude_modules: ".*visual.*")
Method LoRA on the language tower, r=16, α=32, dropout 0.05
Targets q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Trainable 43,646,976 parameters
Objective full-vocabulary next-token cross-entropy at the readout position
Steps 3,000, batch size 8, grad checkpointing, lr 1e-4, 100 warmup, bf16
Images native resolution (see note)
Data question records derived from GQA and SNLI-VE training splits; GQA Choice items presented with K in {2, 4} options
Seeds 0 (root), 1 (seed1/), 2 (seed2/)

Note on image resolution. The training configuration sets max_pixels = 200,704 (196 visual tokens), but the Transformers image processor used ignores that attribute, so training and every evaluation above ran at the processor's default, which keeps native resolution (a 500×500 GQA image becomes 256 visual tokens). The same correction applies to the 4B adapter.

No new human annotation was collected. The training records are derived from publicly released benchmarks; no images are redistributed here, and the adapter is a set of low-rank parameter deltas carrying no image data. SNLI-VE is built on Flickr30k images, which are licensed for non-commercial research; using the backbone requires accepting its own licence.

Limitations

  • Helps only on task families seen in training (see the results table).
  • Calibration needs labelled data from the target task; thresholds fitted on it are empirical, not guarantees.
  • Decisions depend on the option order.
  • Single-machine study on one consumer GPU; English only; at most 8 options evaluated.

Citation

@misc{yu2026visualjev,
  title        = {Visual Jev: Accurate and Efficient Decisions from Shared Visual Context},
  author       = {Guanxu Yu and Yuhang Yao},
  year         = {2026},
  eprint       = {2609.25845},
  archivePrefix = {arXiv},
  primaryClass = {cs.CV},
  url          = {https://arxiv.org/abs/2609.25845}
}

License and resources

This adapter is released under Apache-2.0. The backbone and source datasets retain their respective terms.

Reproduction guide · Experimental outputs · Brand asset sources

Downloads last month
44
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for guanxuyu/visual-jev-8b-answer-sft

Adapter
(260)
this model

Paper for guanxuyu/visual-jev-8b-answer-sft