Lesson 2 of 3
Training, validation, test sets and data leakage
Separate fitting from model selection and check whether related examples cross the split.
Go to the exercise ↓Each dataset answers a different evaluation question
Training data supplies the examples used to fit model parameters. Validation data measures candidate behavior while you choose features, model settings or a decision threshold. Test data is reserved for the final evaluation after those choices. This separation estimates how well the chosen procedure handles examples it has not used for development.
Suppose two model settings score 82% and 85% on validation. You can select the second setting and then evaluate it once on the reserved test set. If you inspect test errors, revise the model and measure that same test set again, its results have influenced development. The revised procedure needs a new independent final evaluation.
A split percentage alone does not make an evaluation reliable. The held-out population must match the question: future deliveries need evaluation on later deliveries; predictions for new customers need customers absent from training. Small test sets also produce uncertain estimates. Report their size and investigate error cases before generalizing a score.
Related records can expose an answer across a split
Data leakage occurs when model development uses information unavailable for the intended prediction. One source is a feature collected after the target outcome. Another is overlap between training and evaluation examples, such as copied text or several messages from the same conversation. A model can recognize a conversation-specific phrase and score well without handling a new conversation.
In this lab, each of three conversations contributes two synthetic messages. The target is a conversation-level support category, and the intended use is categorizing messages from unseen conversations. A random row split puts messages from the same conversation in training and evaluation. Keeping each conversation in one partition removes this particular overlap.
Group separation does not prove that the remaining data is clean. Different conversation IDs can still contain copied templates or duplicated source documents. Inspect source lineage and duplicates as well as IDs. When the intended task is future prediction, use chronological boundaries and check that features were available at each prediction time.
Fit preprocessing using the training partition
Preprocessing can learn values too. A scaler estimates means and spreads; an imputer may estimate a replacement for a missing value. Compute those estimates from training data, then apply the same fitted transformation to validation, test and live inputs. Computing them across the full dataset lets held-out observations affect the procedure being evaluated.
Split the source records before fitting transformations, selecting features or fitting the model. For cross-validation, refit the whole preprocessing-and-model procedure within each training fold. A pipeline helps keep these operations together, but its boundaries still need the correct group or time split.
The exercise audits partition membership. It does not train a classifier, inspect text for near-duplicates or estimate production accuracy. Its result establishes whether these six records satisfy the stated group separation and nonempty-partition requirements.
Repair a split with conversation overlap
Loading your exercise…