cringepnh/koelectra-korean-sentiment

KoELECTRA Korean Movie Review Sentiment

Binary negative/positive classification for Korean movie reviews. The model is fine-tuned from the raw, task-agnostic monologg/koelectra-base-v3-discriminator, not from a checkpoint already fine-tuned on sentiment.

Source code, full reproduction pipeline, and the compute-vs-accuracy benchmark this project is actually built around: GitHub. A Gradio demo lives in that repo's space/ folder and loads this model straight from the Hub, but there is no hosted demo — Hugging Face requires a PRO subscription to run Gradio Spaces, so the demo is local-only.

Is this model worth using over a linear baseline?

This model beats a character TF-IDF + logistic regression baseline by 3.72 points of accuracy (95% CI [3.44, 4.03], McNemar p < 0.001 — the gap is real). But it costs 1,129x the parameters, 106x the disk size, and 25–230x the inference latency depending on hardware. See the full benchmark before assuming the transformer is the right choice for a CPU-bound or latency-sensitive deployment.

Results

All rows use the same cleaned 48,416-review NSMC test set, with 95% bootstrap confidence intervals (1,000 resamples). Test was not used for checkpoint selection.

System	Accuracy	Precision	Recall	F1
Majority class	49.68% [49.22, 50.15]	0.00%	0.00%	0.00%
Character TF-IDF + logistic regression	86.75% [86.45, 87.04]	87.19%	86.36%	86.77%
Raw encoder + newly initialized head	50.56%	53.39%	13.66%	21.75%
Fine-tuned raw KoELECTRA	90.47% [90.23, 90.73]	89.30%	92.11%	90.68%

Per-class: negative recall 88.82%, positive recall 92.11% — the model is somewhat more likely to miss a negative review than a positive one. Confusion matrix and full per-class table are in evaluation_results.json.

When the raw checkpoint is loaded for sequence classification, every classifier.* weight is reported missing and newly initialized. No sentiment head is inherited — the untrained-head row above is the empirical proof.

Not directly comparable to the published NSMC benchmark (~90.6%): this pipeline removes 694 exact train/test text overlaps and conflicting-label rows before evaluating, which the original benchmark does not document doing.

Training and data

• Dataset: NSMC
• Train / validation / test after cleaning: 131,422 / 14,603 / 48,416
• Conflicting-label identical texts, duplicate texts, and 694 remaining train/test overlaps are removed
• Seed: 42
• Maximum length: 128
• Learning rate: 2e-5
• Batch size: 32
• Maximum epochs: 3; best checkpoint (epoch 1) selected by validation loss

Use

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

model_id = "cringepnh/koelectra-korean-sentiment"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id).eval()
inputs = tokenizer("이 영화 정말 재미있어요!", return_tensors="pt", truncation=True)
with torch.inference_mode():
    probabilities = torch.softmax(model(**inputs).logits, dim=-1)[0]
label = int(probabilities.argmax())
print(model.config.id2label[label], float(probabilities[label]))

Limitations

• Korean-only, binary, trained on movie reviews. No neutral label.
• Results are from one deterministic split and seed; bootstrap CIs quantify sampling uncertainty on this split, not across different splits or retraining runs.
• Recall on the negative class (88.82%) trails the positive class (92.11%).
• The TF-IDF baseline is dramatically cheaper for a 3.72pp accuracy cost — see the compute-vs-accuracy comparison linked above before deploying this model where a linear model would do.