Lesson 3 of 3
Evaluation metrics, baselines and class imbalance
Count classification errors, compare a simple baseline and choose a threshold for a stated goal.
Go to the exercise ↓A baseline gives a score a useful comparison
An evaluation metric turns predictions and known targets into a number. Its meaning depends on the population and on the mistakes it counts. Compare a model with a simple baseline on the same held-out examples. For regression, a constant prediction such as the training-target mean is one baseline. For classification, predicting the training set’s most frequent category is another.
Our synthetic evaluation batch has 20 messages: 4 spam and 16 legitimate. A baseline that predicts legitimate for every message gets 16 correct, or 80% accuracy. It catches zero spam. A model can match that accuracy while making a very different mix of errors, so accuracy alone cannot describe its usefulness for spam detection.
Choose the baseline rule from training data and freeze it before evaluating. Do not use evaluation labels to choose its constant or most frequent category. In this lab the always-legitimate rule is fixed in advance; the 20 labels only measure its results. This is a small arithmetic exercise, not a measured production spam filter.
Precision and recall use different denominators
Treat spam as the positive category. A true positive is spam correctly flagged. A false positive is a legitimate message incorrectly flagged. A false negative is missed spam, and a true negative is legitimate mail correctly left alone. These four counts form a confusion matrix.
Precision = true positives / (true positives + false positives). It answers how much flagged mail is actually spam. Recall = true positives / (true positives + false negatives). It answers how much of the actual spam was caught. Accuracy = (true positives + true negatives) / all messages. The denominators explain why the metrics can move differently.
If a model flags no messages, precision has a zero denominator and is undefined. The lab displays an em dash for that value. Recall is still 0% here because the dataset has four actual positives and none were caught. In a batch without any positives, recall would also be undefined; a larger or differently sampled evaluation would be needed to measure it.
The decision threshold changes which errors occur
A scoring model can assign a value to each message, and a threshold converts that score into a category. In this lab a score at or above the threshold is flagged as spam. Lowering the threshold flags more messages. On these fixed examples, true positives and false positives cannot decrease as you lower it.
The exercise requires catching at least three of the four spam messages while flagging at most two legitimate messages. That goal defines which thresholds are acceptable for this batch. It does not claim those costs suit another application. A medical screening tool, an abuse queue and an email filter can require different error tradeoffs.
The displayed scores are fixed synthetic values, not outputs from a trained model or calibrated probabilities. Use validation data to select a threshold, then freeze it for a separate test evaluation. This small lab reuses one batch so you can inspect the arithmetic; it supplies no final generalization estimate. After deployment, changing class proportions or input patterns can change the measured metrics.
Choose a threshold and compare the baseline
Loading your exercise…