Delveta-BGE-M3-v1

Delveta-specific fine-tuned BGE-M3 dense retriever. A dense-only fine-tune of BAAI/bge-m3 for capability / semantic retrieval inside the Delveta retrieval pipeline.

A. Overview

This model is the final inference artifact of the Delveta BGE-M3 fine-tuning run. It is a dense-only retriever built on the BGE-M3 dense encoder (XLM-RoBERTa-large, 1024-dim).

• Dense-only: only the dense embedding is produced and trained.
• No classification head. The model does not classify a query into a capability.
• There is no "Task A / Task B" split. There is a single objective, not two losses.

Model variants

This repository provides three parallel variants of the same dense-only model. Each is a self-contained subfolder with its own weights, config, tokenizer, and README. There is no model copy at the repository root.

Variant	Path	Format	File size	SHA256
FP32 (master / reference)	FP32/	safetensors, float32	2.27 GB	2e7ef22274798217832200d8666d94f6ba0bdf18107a6dc321ad5ccfa5c2aac6
FP16 (derived)	FP16/	safetensors, float16	1.14 GB	a57cab2edb389ef43467103e46cd0825cf1244fbb236ff442c5134b2eaa08db7
INT8 (derived, CPU)	INT8/	ONNX, dynamic int8	568.51 MB	0e0ada7fec9367bca6d816b86953e4f9fe8531ebd4e8251f6de0721c0d2b1f0d

All three share the same embedding semantics: normalize(last_hidden_state[:, 0]) (CLS pooling + L2 normalization). FP16 and INT8 are derived from FP32/, whose weights are the trained checkpoint-2027 and are byte-identical to it. The FP32 master is the reference; do not expect it duplicated at the repo root.

Load a variant via subfolder (FP32 / FP16):

from transformers import AutoTokenizer, AutoModel
tok   = AutoTokenizer.from_pretrained("eric-ml-nlp/Delveta-BGE-M3-v1", subfolder="FP32")  # or "FP16"
model = AutoModel.from_pretrained("eric-ml-nlp/Delveta-BGE-M3-v1", subfolder="FP32")      # or "FP16"

Load the INT8 variant (ONNX Runtime, CPU):

import onnxruntime as ort
sess = ort.InferenceSession("<path>/INT8/model.onnx", providers=["CPUExecutionProvider"])

Conversion method, tooling versions, file sizes, SHA256, runtime requirements, and measured CPU figures are recorded in each variant's own README.

Embedding-level consistency (12 frozen test queries): FP16 cosine vs FP32 is ~1.0 (min 0.999999); INT8 cosine vs FP32 is min 0.988 / mean 0.991. These are embedding-consistency checks, not retrieval-effectiveness results.

Source code

This model is the artifact of the Delveta BGE-M3 fine-tuning track. For the fine-tuning scripts and run.sh entry point, see the BGE-M3 fine-tuning scripts. The per-component source links below are pinned to revision ac873de for reproducibility.

Component	Source
One-command entry (run.sh)	fine_tuning/bge_m3_finetune/
Training entry	train_bge_m3_dual_target.py
Capability-unique sampler	capability_unique_sampler.py
Trainer mixin	capability_unique_trainer.py
Row-alignment check	dual_target_alignment.py
Sampler tests	test_capability_unique_sampler.py
Training-entry tests	test_dual_target_training_entry.py
Experiment record (training + variants + evaluation)	docs/experiments/bge-m3-finetuning.md

The reproducible training procedure, the model-variant generation/verification, and the four measurement families (training metrics, embedding numerical consistency, retrieval effectiveness, runtime performance) are all documented in BGE-M3 Fine-Tuning.

B. Base model

• Base model: BAAI/bge-m3
• Base revision: 5617a9f61b028005a4858fdac845db406aefb181

C. Training objective

A single unified contrastive objective:

query -> positive target text -> negative target texts

• Input schema: query / positive / negative.
• A positive target is either a capability's canonical description or a standard/similar corpus query — both are the same contrastive task for the embedder.
• Default in-batch negatives are enabled.
• Capability-unique batching: at most one training row per capability per batch.
• gold_capability_id is used only for batch sampling (capability-unique batching); it never enters the loss.
• task is metadata only: it is not a classification label, does not enter the loss, and does not select any training branch.

This is a contrastive embedding model, not a classifier.

D. Training configuration

Setting	Value
Epochs	1
Optimizer steps	2027
per-device batch size	8
train_group_size	8
query_max_len	64
passage_max_len	64
bf16	True
Dense-only	True
unified_finetuning	False
In-batch negatives	enabled
Framework	FlagEmbedding 1.4.2
Transformers	5.17.0
Torch	2.14.0 (CUDA build, 2.14.0+cu130)

E. Training data

Training Dataset: Delveta-BGE-M3-v1-Data

• Dataset: BGE-M3 Dual-Target Dataset V1 (frozen)
• Training rows: 16216
• Test rows: 14400
• Capabilities: 18
• Objective: a single unified contrastive objective (query -> positive target text -> negative target texts); a positive target is either a capability description or a standard/similar corpus query.
• Provenance: two query families — capability descriptions and standard/similar corpus queries — combined into one contrastive objective.
• The training JSONL is not redistributed in this model repo. It is archived in the frozen dataset repo linked above.

F. Provenance

Field	Value
Git commit	f7b1f21
Dataset SHA256	163b181f4980e0fc4e0d47e9140e554f147a38a160e7a3a310e511e750fe99b8
Manifest SHA256	57a0fb79d7c0238c73ab704b67928c876aaaf6f434a0975b1ae9ff095cc3a33d
Dataset freeze commit	24ab4c7
Dataset freeze tag	bge-m3-dual-target-v1
Base revision	5617a9f61b028005a4858fdac845db406aefb181
Hardware	NVIDIA L4 24GB

G. Training Results

Final training loss: 0.8835 (epoch-average training loss).

Objective and Loss

Training uses the dense contrastive learning objective from FlagEmbedding 1.4.2 with unified_finetuning=False. This training run uses the dense objective only: sparse, ColBERT, and ensemble objectives are not trained, there is no separate Task A / Task B loss, and there is no classification objective.

For a batch of B queries with G = train_group_size passages per query group, the dense contrastive loss is computed from the query-passage scores:

$$s_{ij}=\frac{q_i^{\mathrm{\top }}{p}_j}{\tau }s\_\{ij\}\; =\; \backslash frac\{q\_i^\backslash top\; p\_j\}\{\backslash tau\}$$

$$\mathcal{L}=\frac{1}{B}\sum _{i=1}^B\mathrm{CE}(s_i,y_i)\backslash mathcal\{L\}\; =\; \backslash frac\{1\}\{B\}\; \backslash sum\_\{i=1\}^\{B\}\; \backslash operatorname\{CE\}(s\_i,\; y\_i)$$

where q_i is the query embedding, p_j is a passage embedding, s_ij is the corresponding query-passage logit, tau is the temperature configured for this training run (0.02), and y_i identifies the positive passage for query i.

For this training run, query and passage embeddings use CLS pooling with normalize_embeddings=True. Each query is associated with one positive passage and explicit negative passages, and passages from other queries in the batch are used as in-batch negatives.

Selected Logged Training Loss

Optimizer step	Logged training loss
500	1.1208
1000	0.8270
1500	0.8004
2000	0.7955

The final epoch-average training loss was 0.8835. The values above are logged training-loss values rather than single-step instantaneous losses.

Training loss decreased during training, but this alone does not establish retrieval improvement.

Post-finetuning retrieval evaluation: NOT YET COMPLETED.

No claim is made that retrieval Recall, MRR, or accuracy improved, and no claim of state-of-the-art performance or production superiority is made.

H. Evaluation status

Methodology and per-family status: BGE-M3 Fine-Tuning.

Post-finetuning retrieval evaluation: NOT YET COMPLETED.

No Recall@1/@3/@5/@10, MRR, or capability-recall numbers are reported, because no formal post-finetuning retrieval benchmark has been run.

I. Intended use

• Delveta capability retrieval
• Semantic retrieval
• Embedding generation for the Delveta retrieval pipeline

J. Limitations

• Optimized for a Delveta capability-oriented corpus; it is not a drop-in general-purpose replacement for the original BAAI/bge-m3.
• No formal post-finetuning retrieval benchmark has been completed yet.
• Users should evaluate it on their own retrieval corpus before relying on it.

K. License

The base model license is MIT (per the BAAI/bge-m3 model card). This model card's license field reflects that base-model license. The Delveta project license is defined separately by the Delveta project and is not restated here; consult the Delveta repository for the project's own terms. No legal terms have been invented or reworded beyond what the base model card states.

L. Citation

The base model and architecture are described in the BGE-M3 paper and the FlagEmbedding project:

• Paper: BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation — https://arxiv.org/abs/2402.03216
• Code: https://github.com/FlagOpen/FlagEmbedding

@misc{bge-m3,
  title={BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation},
  author={Chen, Jianlv and Xiao, Shitao and Zhang, Peitian and Luo, Kun and Lian, Defu and Liu, Zheng},
  year={2024},
  eprint={2402.03216},
  archivePrefix={arXiv},
  primaryClass={cs.CL}
}