Title: Watch: Honest model evaluation
Course: Advanced Data Science: A Graduate Course > Section: Module 5: Rigorous model evaluation

Watch

▶ Recorded lecture (~25 min) + slides ↓:
- Fitting versus generalizing: why training error is an optimistic, biased estimate of future performance, and what the gap between training and test error tells you.
- Why we hold data back, and the ladder from a single train/test split to k-fold cross-validation to nested cross-validation when you also tune hyperparameters.
- How leakage sneaks in: fitting a scaler, imputer, feature selector, or target encoder on the full dataset before splitting; temporal leakage; and duplicate or grouped records straddling the train/test boundary.
- Honest metrics under class imbalance: why 97 percent accuracy can be worse than useless when the positive class is 3 percent, and what the confusion matrix, precision, recall, and F-scores reveal instead.
- Thresholds and the precision/recall trade-off: how a classifier outputs a score, how the decision threshold turns scores into labels, and how ROC and PR curves summarize every threshold at once.
- Uncertainty on a metric: why a point estimate is a sample from a distribution, and how bootstrap resampling and cross-fold variation give you a confidence interval you can defend.

“Advanced Data Science by Michelle Blomberg is licensed under CC BY-NC-SA 4.0.”