On this tutorial, we develop an end-to-end sentiment evaluation workflow utilizing the Stanford NLP IMDb Massive Film Assessment Dataset and examine classical machine studying with parameter-efficient transformer fine-tuning. We start by establishing a reproducible surroundings and auditing the dataset for sophistication ordering, review-length skew, duplicate leakage, and preprocessing artifacts earlier than coaching a robust TF-IDF and Logistic Regression baseline. We then fine-tune DistilBERT with LoRA by PEFT, consider it utilizing accuracy, macro-F1, ROC-AUC, confusion matrices, and ROC curves, and study threshold choice and chance calibration by Anticipated Calibration Error and reliability evaluation. Past headline metrics, we examine assured errors, efficiency throughout evaluation lengths, word-level occlusion saliency, and head-versus-tail truncation to grasp how the mannequin reaches its predictions and the place long-context limitations have an effect on efficiency. Lastly, we use the unlabeled IMDb cut up for confidence-based pseudo-labeling, examine the ensuing semi-supervised mannequin towards our baseline, and save the merged transformer for reusable sentiment inference.
import importlib.util, subprocess, sys, os, time, random, warnings, examine, hashlib
warnings.filterwarnings("ignore")
os.environ["TOKENIZERS_PARALLELISM"] = "false"
os.environ["WANDB_DISABLED"] = "true"
_REQUIRED = {
"transformers": "transformers",
"datasets": "datasets",
"peft": "peft",
"speed up": "speed up",
"sklearn": "scikit-learn",
}
_missing = [pkg for mod, pkg in _REQUIRED.items() if importlib.util.find_spec(mod) is None]
if _missing:
print(f"Putting in: {', '.be a part of(_missing)} ...")
subprocess.run([sys.executable, "-m", "pip", "install", "-q", *_missing], verify=True)
print("Executed. (If imports fail under, restart the runtime and re-run.)n")
import numpy as np
import pandas as pd
import torch
import matplotlib.pyplot as plt
from datasets import load_dataset
from sklearn.feature_extraction.textual content import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.metrics import (accuracy_score, f1_score, roc_auc_score,
classification_report, confusion_matrix, roc_curve)
from transformers import (AutoTokenizer, AutoModelForSequenceClassification,
TrainingArguments, Coach, DataCollatorWithPadding,
EarlyStoppingCallback, set_seed)
from peft import LoraConfig, get_peft_model, TaskType
def _disable_torchao_probe():
patched = []
attempt:
import peft.import_utils as _piu
_piu.is_torchao_available = lambda: False
patched.append("peft.import_utils")
besides Exception:
move
for _name, _mod in checklist(sys.modules.gadgets()):
if _name.startswith("peft") and hasattr(_mod, "is_torchao_available"):
_mod.is_torchao_available = lambda: False
patched.append(_name)
return patched
attempt:
import torchao as _tao
_v = getattr(_tao, "__version__", "?")
if tuple(int(x) for x in _v.cut up(".")[:2]) < (0, 16):
print(f"[compat] torchao {_v} < 0.16 -> disabling PEFT's torchao probe: "
f"{', '.be a part of(_disable_torchao_probe())}")
besides Exception:
_disable_torchao_probe()
SEED = 42
MODEL_NAME = "distilbert-base-uncased"
MAX_LEN = 256
N_TRAIN = 5000
N_EVAL = 2000
N_UNSUP = 3000
EPOCHS = 2
BATCH = 16
LR = 3e-4
FULL_RUN = False
if FULL_RUN:
N_TRAIN, N_EVAL, EPOCHS = 25000, 25000, 3
set_seed(SEED); random.seed(SEED); np.random.seed(SEED); torch.manual_seed(SEED)
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
print("=" * 79)
print(f"system={DEVICE} | torch={torch.__version__} | "
f"gpu={torch.cuda.get_device_name(0) if DEVICE=='cuda' else 'n/a'}")
print("=" * 79)
t0 = time.time()
uncooked = load_dataset("stanfordnlp/imdb")
print(uncooked, f"nloaded in {time.time()-t0:.1f}sn")
print("--- instance (truncated) ---")
print("label:", uncooked["train"][0]["label"], "|", uncooked["train"][0]["text"][:300], "...n")
first_labels = np.array(uncooked["train"]["label"][:5])
last_labels = np.array(uncooked["train"]["label"][-5:])
print(f"TRAP #1 - cut up ordering: first 5 labels {first_labels}, "
f"final 5 labels {last_labels} -> ALWAYS shuffle earlier than subsampling.")
train_full = uncooked["train"].shuffle(seed=SEED)
test_full = uncooked["test"].shuffle(seed=SEED)
train_ds = train_full.choose(vary(min(N_TRAIN, len(train_full))))
eval_ds = test_full.choose(vary(min(N_EVAL, len(test_full))))
print(f" after shuffle+subsample: prepare stability = "
f"{np.bincount(train_ds['label'])}, eval stability = {np.bincount(eval_ds['label'])}")
lens = np.array([len(t.split()) for t in train_full["text"]])
q = np.percentile(lens, [50, 75, 90, 95, 99])
print(f"nTRAP #2 - size (phrases): median={q[0]:.0f} p75={q[1]:.0f} p90={q[2]:.0f} "
f"p95={q[3]:.0f} p99={q[4]:.0f} max={lens.max()}")
print(f" ~{(lens > MAX_LEN*0.75).imply()*100:.1f}% of opinions exceed MAX_LEN={MAX_LEN} "
f"tokens (tough words->tokens issue 1.3). Part 9 measures what that prices.")
h_tr = {hashlib.md5(t.encode()).hexdigest() for t in uncooked["train"]["text"]}
h_te = {hashlib.md5(t.encode()).hexdigest() for t in uncooked["test"]["text"]}
print(f"nTRAP #3 - leakage: {len(h_tr & h_te)} actual duplicate opinions throughout "
f"prepare/check; {len(uncooked['train'])-len(h_tr)} dupes inside prepare itself.")
def clear(t):
return t.substitute("<br />", " ").substitute("<br/>", " ").strip()
plt.determine(figsize=(11, 3.2))
plt.subplot(1, 2, 1)
plt.hist(np.clip(lens, 0, 1000), bins=60)
plt.axvline(MAX_LEN, ls="--", colour="okay", label=f"MAX_LEN={MAX_LEN}")
plt.title("Assessment size (phrases, clipped at 1000)"); plt.legend()
plt.subplot(1, 2, 2)
plt.bar(["neg", "pos"], np.bincount(uncooked["train"]["label"]))
plt.title("Practice class stability (completely balanced)")
plt.tight_layout(); plt.present()
We configure the Colab surroundings, set up the required libraries, apply the PEFT–torchao compatibility repair, and set deterministic seeds for reproducible experiments. We load the Stanford IMDb dataset, shuffle and subsample the prepare and check splits, and examine class stability, review-length distributions, duplicate leakage, and HTML artifacts. We additionally visualize evaluation lengths and label frequencies so we perceive the dataset construction earlier than constructing any fashions.
print("n" + "=" * 79 + "n3. TF-IDF BASELINEn" + "=" * 79)
Xtr = [clean(t) for t in train_ds["text"]]; ytr = np.array(train_ds["label"])
Xte = [clean(t) for t in eval_ds["text"]]; yte = np.array(eval_ds["label"])
t0 = time.time()
tfidf_clf = make_pipeline(
TfidfVectorizer(ngram_range=(1, 2), min_df=2, max_features=300_000,
sublinear_tf=True, strip_accents="unicode"),
LogisticRegression(C=8.0, max_iter=2000, n_jobs=-1),
)
tfidf_clf.match(Xtr, ytr)
p_tfidf = tfidf_clf.predict_proba(Xte)[:, 1]
acc_tfidf = accuracy_score(yte, p_tfidf > 0.5)
auc_tfidf = roc_auc_score(yte, p_tfidf)
print(f"educated in {time.time()-t0:.1f}s -> acc={acc_tfidf:.4f} auc={auc_tfidf:.4f}")
vec, lr = tfidf_clf.steps[0][1], tfidf_clf.steps[1][1]
feats, coefs = np.array(vec.get_feature_names_out()), lr.coef_[0]
order = np.argsort(coefs)
print("nmost NEGATIVE n-grams:", ", ".be a part of(feats[order[:12]]))
print("most POSITIVE n-grams:", ", ".be a part of(feats[order[-12:]][::-1]))
print("n" + "=" * 79 + "n4. LoRA FINE-TUNINGn" + "=" * 79)
tok = AutoTokenizer.from_pretrained(MODEL_NAME)
def tokenize(batch):
return tok([clean(t) for t in batch["text"]], truncation=True, max_length=MAX_LEN)
tr_tok = (train_ds.map(tokenize, batched=True, remove_columns=["text"])
.rename_column("label", "labels"))
ev_tok = (eval_ds.map(tokenize, batched=True, remove_columns=["text"])
.rename_column("label", "labels"))
base = AutoModelForSequenceClassification.from_pretrained(
MODEL_NAME, num_labels=2,
id2label={0: "NEGATIVE", 1: "POSITIVE"},
label2id={"NEGATIVE": 0, "POSITIVE": 1},
)
lora_cfg = LoraConfig(
task_type=TaskType.SEQ_CLS,
r=16, lora_alpha=32, lora_dropout=0.05,
target_modules=["q_lin", "v_lin"],
modules_to_save=["pre_classifier", "classifier"],
)
attempt:
mannequin = get_peft_model(base, lora_cfg)
besides ImportError as e:
_disable_torchao_probe()
print(f"[compat] retrying after backend probe failure: {e}")
mannequin = get_peft_model(base, lora_cfg)
mannequin.print_trainable_parameters()
def compute_metrics(eval_pred):
logits, labels = eval_pred
probs = torch.softmax(torch.tensor(logits), dim=-1).numpy()[:, 1]
preds = (probs > 0.5).astype(int)
return {"accuracy": accuracy_score(labels, preds),
"f1_macro": f1_score(labels, preds, common="macro"),
"roc_auc": roc_auc_score(labels, probs)}
_ta = examine.signature(TrainingArguments.__init__).parameters
_eval_key = "eval_strategy" if "eval_strategy" in _ta else "evaluation_strategy"
ta_kwargs = dict(
output_dir="./imdb_lora", learning_rate=LR,
per_device_train_batch_size=BATCH, per_device_eval_batch_size=BATCH * 2,
num_train_epochs=EPOCHS, weight_decay=0.01, warmup_ratio=0.06,
logging_steps=50, save_strategy="epoch", save_total_limit=1,
load_best_model_at_end=True, metric_for_best_model="accuracy",
fp16=(DEVICE == "cuda"), report_to="none", seed=SEED,
)
ta_kwargs[_eval_key] = "epoch"
_tr = examine.signature(Coach.__init__).parameters
_tok_key = "processing_class" if "processing_class" in _tr else "tokenizer"
coach = Coach(
mannequin=mannequin, args=TrainingArguments(**ta_kwargs),
train_dataset=tr_tok, eval_dataset=ev_tok,
data_collator=DataCollatorWithPadding(tok),
compute_metrics=compute_metrics,
callbacks=[EarlyStoppingCallback(early_stopping_patience=2)],
**{_tok_key: tok},
)
t0 = time.time()
coach.prepare()
print(f"nfine-tuned in {(time.time()-t0)/60:.1f} min")
We prepare a robust TF-IDF and Logistic Regression baseline and examine essentially the most influential constructive and destructive n-grams to determine an interpretable reference level. We then tokenize the IMDb opinions and configure DistilBERT with LoRA adapters that replace solely a small subset of mannequin parameters whereas holding the spine largely frozen. We use the Hugging Face Coach with dynamic padding, early stopping, combined precision, and a number of analysis metrics to fine-tune the transformer effectively.
print("n" + "=" * 79 + "n5. EVALUATIONn" + "=" * 79)
pred_out = coach.predict(ev_tok)
p_lora = torch.softmax(torch.tensor(pred_out.predictions), dim=-1).numpy()[:, 1]
y_true = np.array(pred_out.label_ids)
yhat = (p_lora > 0.5).astype(int)
print(classification_report(y_true, yhat, target_names=["neg", "pos"], digits=4))
cm = confusion_matrix(y_true, yhat)
fig, ax = plt.subplots(1, 2, figsize=(11, 4))
ax[0].imshow(cm, cmap="Blues")
for i in vary(2):
for j in vary(2):
ax[0].textual content(j, i, cm[i, j], ha="middle", va="middle", fontsize=14)
ax[0].set_xticks([0, 1], ["pred neg", "pred pos"])
ax[0].set_yticks([0, 1], ["true neg", "true pos"]); ax[0].set_title("Confusion matrix")
for title, p in [("TF-IDF", p_tfidf), ("DistilBERT+LoRA", p_lora)]:
fpr, tpr, _ = roc_curve(y_true, p)
ax[1].plot(fpr, tpr, label=f"{title} (AUC={roc_auc_score(y_true, p):.4f})")
ax[1].plot([0, 1], [0, 1], "k--", lw=0.8)
ax[1].set_xlabel("FPR"); ax[1].set_ylabel("TPR"); ax[1].set_title("ROC"); ax[1].legend()
plt.tight_layout(); plt.present()
print("n" + "=" * 79 + "n6. THRESHOLD & CALIBRATIONn" + "=" * 79)
ths = np.linspace(0.05, 0.95, 91)
accs = [(y_true == (p_lora > t)).mean() for t in ths]
best_t = ths[int(np.argmax(accs))]
print(f"[email protected] = {accs[45]:.4f} | finest threshold = {best_t:.2f} -> acc = {max(accs):.4f}")
def expected_calibration_error(probs, labels, n_bins=10):
"""ECE: |confidence - accuracy| averaged over confidence bins."""
conf = np.most(probs, 1 - probs)
appropriate = (probs > 0.5).astype(int) == labels
bins = np.linspace(0, 1, n_bins + 1)
ece, xs, ys = 0.0, [], []
for lo, hello in zip(bins[:-1], bins[1:]):
m = (conf > lo) & (conf <= hello)
if m.sum() == 0:
proceed
ece += m.imply() * abs(conf[m].imply() - appropriate[m].imply())
xs.append(conf[m].imply()); ys.append(appropriate[m].imply())
return ece, np.array(xs), np.array(ys)
ece, cx, cy = expected_calibration_error(p_lora, y_true)
print(f"Anticipated Calibration Error = {ece:.4f} (0 = completely calibrated)")
plt.determine(figsize=(9, 3.2))
plt.subplot(1, 2, 1); plt.plot(ths, accs); plt.axvline(best_t, ls="--", colour="r")
plt.xlabel("threshold"); plt.ylabel("accuracy"); plt.title("Threshold sweep")
plt.subplot(1, 2, 2); plt.plot([0.5, 1], [0.5, 1], "k--", lw=0.8)
plt.plot(cx, cy, "o-"); plt.xlabel("imply confidence"); plt.ylabel("empirical accuracy")
plt.title(f"Reliability diagram (ECE={ece:.3f})")
plt.tight_layout(); plt.present()
We consider the fine-tuned DistilBERT-LoRA mannequin utilizing classification metrics, a confusion matrix, and ROC curves whereas instantly evaluating its ROC-AUC efficiency with the TF-IDF baseline. We sweep classification thresholds to find out whether or not the default chance cutoff of 0.5 offers the very best accuracy on our analysis set. We additionally calculate Anticipated Calibration Error and assemble a reliability diagram to measure how carefully the mannequin’s predicted confidence corresponds to its precise correctness.
print("n" + "=" * 79 + "n7. ERROR ANALYSISn" + "=" * 79)
err = pd.DataFrame({
"textual content": eval_ds["text"], "y": y_true, "p_pos": p_lora,
"n_words": [len(t.split()) for t in eval_ds["text"]],
})
err["pred"] = (err.p_pos > 0.5).astype(int)
err["correct"] = err.pred == err.y
err["confidence"] = np.most(err.p_pos, 1 - err.p_pos)
print("--- 3 most CONFIDENT errors (the place the mannequin is confidently improper) ---")
for _, r in err[~err.correct].nlargest(3, "confidence").iterrows():
print(f"n[true={'pos' if r.y else 'neg'} pred={'pos' if r.pred else 'neg'} "
f"conf={r.confidence:.3f} words={r.n_words}]")
print(clear(r.textual content)[:400].substitute("n", " "), "...")
err["bucket"] = pd.qcut(err.n_words, 4, labels=["short", "med", "long", "v.long"])
by_len = err.groupby("bucket", noticed=True).agg(acc=("appropriate", "imply"), n=("appropriate", "measurement"))
print("n--- accuracy by evaluation size (truncation hurts lengthy opinions) ---")
print(by_len.to_string())
print("n" + "=" * 79 + "n8. OCCLUSION SALIENCYn" + "=" * 79)
infer_model = mannequin.merge_and_unload()
infer_model.to(DEVICE).eval()
@torch.no_grad()
def predict_proba(texts, bs=64):
out = []
for i in vary(0, len(texts), bs):
enc = tok([clean(t) for t in texts[i:i + bs]], truncation=True,
max_length=MAX_LEN, padding=True, return_tensors="pt").to(DEVICE)
out.append(torch.softmax(infer_model(**enc).logits, dim=-1)[:, 1].cpu().numpy())
return np.concatenate(out)
def occlusion(textual content, max_words=60):
phrases = clear(textual content).cut up()[:max_words]
base = predict_proba([" ".join(words)])[0]
variants = [" ".join(words[:i] + phrases[i + 1:]) for i in vary(len(phrases))]
dropped = predict_proba(variants)
return phrases, base - dropped, base
pattern = err[err.correct].nlargest(1, "confidence").iloc[0]
phrases, contrib, base_p = occlusion(pattern.textual content)
print(f"P(constructive) for the total excerpt = {base_p:.3f} "
f"(true label = {'pos' if pattern.y else 'neg'})n")
high = np.argsort(np.abs(contrib))[-15:]
plt.determine(figsize=(7, 5))
plt.barh(vary(len(high)), contrib[top],
colour=["tab:green" if contrib[i] > 0 else "tab:pink" for i in high])
plt.yticks(vary(len(high)), [words[i] for i in high])
plt.xlabel("Δ P(constructive) when the phrase is eliminated")
plt.title("Occlusion saliency — inexperienced pushes POSITIVE, pink pushes NEGATIVE")
plt.tight_layout(); plt.present()
print("n" + "=" * 79 + "n9. HEAD vs TAIL TRUNCATIONn" + "=" * 79)
probe = err.nlargest(600, "n_words")
W = 180
head_txt = [" ".join(clean(t).split()[:W]) for t in probe.textual content]
tail_txt = [" ".join(clean(t).split()[-W:]) for t in probe.textual content]
yp = probe.y.values
acc_head = ((predict_proba(head_txt) > 0.5).astype(int) == yp).imply()
acc_tail = ((predict_proba(tail_txt) > 0.5).astype(int) == yp).imply()
print(f"on the {len(probe)} longest opinions, utilizing solely {W} phrases:")
print(f" first {W} phrases -> acc {acc_head:.4f}")
print(f" final {W} phrases -> acc {acc_tail:.4f}")
print(" Sensible takeaway: if the tail wins, feed head+tail to the mannequin or "
"increase MAX_LEN, moderately than blindly truncating from the left.")
We study the mannequin’s most assured incorrect predictions and group opinions by size to establish truncation-related failure patterns and tough examples. We merge the LoRA adapters into the underlying mannequin and apply leave-one-word-out occlusion to estimate which phrases push particular person predictions towards constructive or destructive sentiment. We then examine predictions primarily based on the start and ending parts of lengthy opinions to find out the place the strongest sentiment data resides.
print("n" + "=" * 79 + "n10. PSEUDO-LABELLINGn" + "=" * 79)
unsup = uncooked["unsupervised"].shuffle(seed=SEED).choose(vary(N_UNSUP))
p_uns = predict_proba(unsup["text"])
preserve = (p_uns > 0.95) | (p_uns < 0.05)
pl_texts = [clean(t) for t, k in zip(unsup["text"], preserve) if okay]
pl_labels = (p_uns[keep] > 0.5).astype(int)
print(f"stored {preserve.sum()}/{N_UNSUP} pseudo-labels at conf>0.95 "
f"(stability: {np.bincount(pl_labels)})")
aug = make_pipeline(
TfidfVectorizer(ngram_range=(1, 2), min_df=2, max_features=300_000,
sublinear_tf=True, strip_accents="unicode"),
LogisticRegression(C=8.0, max_iter=2000, n_jobs=-1),
).match(Xtr + pl_texts, np.concatenate([ytr, pl_labels]))
acc_aug = accuracy_score(yte, aug.predict(Xte))
print(f"TF-IDF baseline : {acc_tfidf:.4f}")
print(f"TF-IDF + pseudo-labels: {acc_aug:.4f} (Δ {acc_aug-acc_tfidf:+.4f})")
print("Caveat: positive aspects are bounded by the instructor. Self-training additionally amplifies "
"the instructor's biases — at all times validate on clear, held-out information.")
print("n" + "=" * 79 + "n11. SAVE & INFERn" + "=" * 79)
SAVE_DIR = "./imdb-distilbert-lora-merged"
infer_model.save_pretrained(SAVE_DIR); tok.save_pretrained(SAVE_DIR)
print(f"saved merged mannequin to {SAVE_DIR}/ (load with "
f"AutoModelForSequenceClassification.from_pretrained('{SAVE_DIR}'))")
demos = [
"A masterclass in tension. The final act left the whole theatre silent.",
"Two hours I will never get back. Wooden acting, incoherent plot.",
"It's not the disaster the trailer promised, but it never really lands either.",
]
for d, p in zip(demos, predict_proba(demos)):
print(f" P(pos)={p:.3f} -> {'POSITIVE' if p > 0.5 else 'NEGATIVE'} | {d}")
print("n" + "=" * 79)
print(f"SUMMARY (n_train={N_TRAIN}, n_eval={N_EVAL}, max_len={MAX_LEN})")
print("=" * 79)
print(pd.DataFrame([
{"model": "TF-IDF + LogReg", "accuracy": acc_tfidf, "roc_auc": auc_tfidf},
{"model": "TF-IDF + pseudo-labels", "accuracy": acc_aug, "roc_auc": float("nan")},
{"model": "DistilBERT + LoRA", "accuracy": accuracy_score(y_true, yhat),
"roc_auc": roc_auc_score(y_true, p_lora)},
]).to_string(index=False))
print("""
NEXT EXPERIMENTS
- Set FULL_RUN = True for the true 25k/25k benchmark (~40 min on a T4).
- Swap MODEL_NAME to 'roberta-base' (target_modules=['query','value']) or
'answerdotai/ModernBERT-base' for an 8k context window — no truncation.
- Head+tail truncation: first 128 + final 128 tokens, motivated by part 9.
- Ablate LoRA rank r in {4, 8, 16, 64} and plot accuracy vs trainable params.
- Change the pseudo-label instructor with an ensemble and iterate self-training.
- Push to the Hub: huggingface_hub.login() then infer_model.push_to_hub(...).
""")
We use the fine-tuned transformer to generate high-confidence pseudo-labels for examples from IMDb’s unlabeled cut up and add these examples to the TF-IDF coaching corpus. We examine the augmented classifier towards the unique baseline to measure whether or not semi-supervised self-training improves predictive accuracy. Lastly, we save the merged DistilBERT mannequin and tokenizer, run sentiment inference on customized opinions, and summarize the efficiency of all fashions developed all through the tutorial.
In conclusion, we developed a rigorous sentiment classification pipeline that goes effectively past merely fine-tuning a transformer and reporting accuracy. We established a aggressive TF-IDF baseline, prepare DistilBERT effectively with LoRA, and consider each predictive high quality and chance reliability whereas figuring out how evaluation size, truncation, and extremely assured errors affect real-world efficiency. We additionally interpreted particular person predictions by occlusion-based saliency, examined whether or not sentiment data is concentrated close to the start or finish of lengthy opinions, and prolonged supervised studying with high-confidence pseudo-labels from the unlabeled dataset.
Take a look at the FULL CODES right here. Additionally, be at liberty to comply with us on Twitter and don’t neglect to affix our 150k+ML SubReddit and Subscribe to our E-newsletter. Wait! are you on telegram? now you possibly can be a part of us on telegram as effectively.
Must companion with us for selling your GitHub Repo OR Hugging Face Web page OR Product Launch OR Webinar and many others.? Join with us
Sana Hassan, a consulting intern at Marktechpost and dual-degree scholar at IIT Madras, is captivated with making use of know-how and AI to deal with real-world challenges. With a eager curiosity in fixing sensible issues, he brings a contemporary perspective to the intersection of AI and real-life options.
