README.md

---

language:
- en
license: apache-2.0
library_name: transformers
tags:
- text-classification
- cross-encoder
- information-retrieval
- job-skill-matching
- esco
- talentclef
- reranking
- bert
base_model: cross-encoder/ms-marco-MiniLM-L-12-v2
pipeline_tag: text-classification
model-index:
- name: skillscout-reranker
  results:
  - task:
      type: information-retrieval
      name: Information Retrieval (re-ranking)
    dataset:
      name: TalentCLEF 2026 Task B Validation
      type: talentclef-2026-taskb-validation
    metrics:
    - type: ndcg_at_10_graded
      value: 0.6896
      name: nDCG@10 Graded (pipeline, server)
    - type: ndcg_at_10_binary
      value: 0.7330
      name: nDCG@10 Binary (pipeline, server)
---

# SkillScout Reranker - Job-Skill Cross-Encoder

**SkillScout Reranker** is a cross-encoder that re-ranks candidate skills for a given job title,
predicting **graded relevance** (0=irrelevant, 1=contextual, 2=core).

This is **Stage 2** of the TalentGuide two-stage job-skill matching pipeline, trained for
[TalentCLEF 2026 Task B](https://talentclef.github.io/).

> **Best pipeline result (TalentCLEF 2026 validation set, server-side):**
> nDCG@10 graded = **0.6896** | nDCG@10 binary = **0.7330**
> SkillScout Large (bi-encoder) + SkillScout Reranker at blend alpha=0.7.

---

## Model Summary

| Property | Value |
|---|---|
| Base model | [cross-encoder/ms-marco-MiniLM-L-12-v2](https://huggingface.co/cross-encoder/ms-marco-MiniLM-L-12-v2) |
| Architecture | BERT (MiniLM-L12) + 3-class classification head |
| Hidden size | 384 | Max seq length | 128 tokens |
| Output classes | 0 = non-relevant, 1 = contextual, 2 = core |
| Training triples | ~130k (job_title, skill, label) |

| Hard negatives | 5 per job, mined from bi-encoder top-K |

| Epochs | 3 | Batch size | 32 |

| Hardware | NVIDIA RTX 3070 8GB, fp16 AMP |



---



## Usage



### Installation



```bash

pip install transformers torch

```



### Score a single (job, skill) pair



```python

from transformers import AutoTokenizer, AutoModelForSequenceClassification

import torch



tokenizer = AutoTokenizer.from_pretrained("talentguide/skillscout-reranker")
model     = AutoModelForSequenceClassification.from_pretrained("talentguide/skillscout-reranker")

model.eval()



job   = "Data Scientist"

skill = "data science"



enc = tokenizer(job, skill, return_tensors="pt", truncation=True, max_length=128)

with torch.no_grad():
    logits = model(**enc).logits          # shape [1, 3]

probs = logits.softmax(-1)[0].tolist()  # [P(irrelevant), P(contextual), P(core)]

relevance = logits.argmax(-1).item()     # 0, 1, or 2


print(f"Relevance class: {relevance}  (0=none, 1=contextual, 2=core)")
print(f"Probs: none={probs[0]:.3f}  contextual={probs[1]:.3f}  core={probs[2]:.3f}")
# Relevance class: 2  (0=none, 1=contextual, 2=core)
# Probs: none=0.031  contextual=0.142  core=0.827
```



### Re-rank a candidate list



```python

# candidates: list of skill texts from bi-encoder (e.g. top-200)

pairs = [(job, skill) for skill in candidates]

encs  = tokenizer(pairs, return_tensors="pt", truncation=True,

                  padding=True, max_length=128)



with torch.no_grad():

    logits = model(**encs).logits   # [N, 3]



# Use class-2 logit (core probability) as ranking score

scores = logits[:, 2].tolist()

ranked = sorted(zip(candidates, scores), key=lambda x: -x[1])



for rank, (skill, score) in enumerate(ranked[:10], 1):

    print(f"{rank:3d}. [{score:.3f}]  {skill}")

```

### Blend with bi-encoder (recommended, alpha=0.7)

```python

# bi_scores: cosine scores from SkillScout Large (normalised to [0,1])

# ce_scores:  class-2 logit from this model (normalised to [0,1])

alpha = 0.7

final_score = alpha * bi_score + (1 - alpha) * ce_score

```

---

## Two-Stage Pipeline Integration

```

Job title

   |

   v

[SkillScout Large]          <- talentguide/skillscout-large

   |  top-200 candidates via FAISS ANN

   v

[SkillScout Reranker]       <- this model

   |  3-class graded scoring (core=2, contextual=1, irrelevant=0)

   v

Final ranked list

```

---

## Training Details

### Data

| | Count |
|---|---|
| Positive triples (essential, label=2) | ~57,500 |
| Positive triples (optional, label=1) | ~28,600 |
| Hard negatives (label=0, from bi-encoder top-K) | ~15,200 |
| Random negatives (label=0) | ~30,000 |
| Total training triples | ~130,000 |
| Validation queries | 304 | Validation corpus | 9,052 skills |

**Hard negatives** are mined by running the fine-tuned bi-encoder (SkillScout Large) over all
training jobs, collecting the top-K retrieved skills that are NOT in the positive set.
This teaches the cross-encoder to distinguish near-miss retrievals from true positives.

### Hyperparameters

```

Base model     : cross-encoder/ms-marco-MiniLM-L-12-v2

Task           : 3-class sequence classification (BERT + linear head)

Loss           : CrossEntropyLoss

Batch size     : 32

Epochs         : 3

Learning rate  : 2e-5, linear warmup 10%

Optimizer      : AdamW

Precision      : fp16 AMP

Max seq len    : 128 tokens

Input format   : [CLS] job_title [SEP] skill_name [SEP]

```

### Pipeline Results (graded relevance, full 9052-skill ranking)

| Run | nDCG@10 graded | nDCG@10 binary | MAP |
|---|---|---|---|
| Bi-encoder only (SkillScout Large) | 0.3621 | 0.4830 | 0.4545 |
| + CE bad negatives (v1) | 0.3226 | 0.4025 | 0.4195 |
| + CE fixed negatives (v2) | 0.3315 | 0.4075 | 0.4228 |
| + CE blend alpha=0.7 (local, top-100) | 0.3816 | 0.4973 | 0.4632 |
| **+ CE blend alpha=0.7 (server, full ranking)** | **0.6896** | **0.7330** | 0.2481 |

*Local metrics use top-100 retrieval cutoff; server metrics use full 9,052-skill ranking.*

---

## Limitations

- **Must be paired with a retriever** - evaluates pairs, not full corpus ranking. Use with SkillScout Large for efficient retrieval.
- **English only** - trained on ESCO EN labels.
- **ESCO-domain optimised** - transfer to other taxonomies may require fine-tuning.
- **Speed** - re-ranks top-200 candidates (~1-2s per query on GPU). Not suitable for full-corpus scoring at inference time.

---

## Citation

```bibtex

@misc{talentguide-skillscout-reranker-2026,

  title  = {SkillScout Reranker: Graded Job-Skill Cross-Encoder for TalentCLEF 2026},

  author = {TalentGuide},

  year   = {2026},

  url    = {https://huggingface.co/talentguide/skillscout-reranker}

}



@misc{talentclef2026taskb,

  title  = {TalentCLEF 2026 Task B: Job-Skill Matching},

  author = {TalentCLEF Organizers},

  year   = {2026},

  url    = {https://talentclef.github.io/}

}

```

---

## Framework Versions

- Python 3.12.10 | Transformers 5.5.0 | PyTorch 2.11.0+cu128