cafebert-VNTC-10

This model is uitnlp/CafeBERT fine-tuned for 10-way Vietnamese news topic classification on VNTC-10.

Evaluation protocol

• Dataset size: 84,132 full news articles.
• Fixed grouped-stratified split: 67,305 train / 8,413 development / 8,414 test.
• Labels: chinh_tri_xa_hoi, doi_song, khoa_hoc, kinh_doanh, phap_luat, suc_khoe, the_gioi, the_thao, van_hoa, vi_tinh.
• Fine-tuning seeds: [42, 22, 202].
• Training: 3 epochs, AdamW, learning rate 2e-05, weight decay 0.01, warmup ratio 0.1.
• Maximum sequence length: 256; dynamic padding is used.
• Input mode: raw Vietnamese.
• Each article is right-truncated. A 1,024-whitespace-unit prefix guard removes only the far tail that cannot enter a 256-subword encoder window.
• No class weighting, resampling, data augmentation, source metadata or test-time model selection is used.
• Checkpoints are selected by development Macro-F1. The published representative checkpoint is seed 22, selected only by development Macro-F1.

Results

Metrics are mean ± sample standard deviation over seeds [42, 22, 202].

Metric	Mean ± std
Test Macro-F1	0.9240 ± 0.0012
Test accuracy	0.9312 ± 0.0009
Test macro precision	0.9255 ± 0.0010
Test macro recall	0.9232 ± 0.0013
Development Macro-F1	0.9219 ± 0.0009

Per-seed results

seed	dev_macro_f1	test_macro_f1	test_accuracy
22.000000	0.922968	0.924415	0.931543
42.000000	0.921266	0.922695	0.930116
202.000000	0.921557	0.925022	0.931899

Label mapping

{
  "0": "chinh_tri_xa_hoi",
  "1": "doi_song",
  "2": "khoa_hoc",
  "3": "kinh_doanh",
  "4": "phap_luat",
  "5": "suc_khoe",
  "6": "the_gioi",
  "7": "the_thao",
  "8": "van_hoa",
  "9": "vi_tinh"
}

Usage

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

model_id = "BaoNhan/cafebert-VNTC-10"
tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=False)
model = AutoModelForSequenceClassification.from_pretrained(model_id)

text = "Đội tuyển Việt Nam giành chiến thắng trong trận đấu bóng đá quốc tế."
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=256)
with torch.no_grad():
    probabilities = model(**inputs).logits.softmax(dim=-1)[0]
predicted_id = int(probabilities.argmax())
print(model.config.id2label[predicted_id], probabilities.tolist())

Files

• aggregate_metrics.json: aggregate metrics and protocol manifest.
• artifacts/per_seed_results.csv: one row per fine-tuning seed.
• artifacts/seed_*_confusion_matrix.csv: confusion matrix for each seed.
• artifacts/seed_*_classification_report.json: per-class metrics.
• artifacts/seed_*_test_predictions.csv: IDs, labels and probabilities; raw articles are excluded.

Limitations

VNTC is an early news corpus whose topics and lexical distribution may not reflect current Vietnamese media. Right truncation may omit decisive information near the end of long articles. Topic labels may also correlate with publisher-specific editorial style. This model is intended for research and should not be treated as a general-purpose factuality or content-quality system.

Dataset citation

@inproceedings{vu2007comparative,
  title={A Comparative Study on Vietnamese Text Classification Methods},
  author={Vu, Hoang Cong Duy and Dien, Dinh and Nguyen, Le Nguyen and Ngo, Quoc Hung},
  booktitle={2007 IEEE International Conference on Research, Innovation and Vision for the Future},
  pages={267--273},
  year={2007},
  organization={IEEE}
}