Title: Overview
Course: Advanced Data Science: A Graduate Course > Section: Module 5: Rigorous model evaluation

Module 5: Rigorous model evaluation

Overview

A model that fits the data you already have is worthless; a model that generalizes to data you have not yet seen is the entire point. This module teaches you to evaluate predictive power honestly. You will hold data back through resampling so that your estimate of performance is not contaminated by the data the model trained on, you will choose metrics that match the target type and the real-world cost of each kind of error rather than defaulting to accuracy, and you will locate any model on the bias-variance trade-off so you understand whether it is under-fitting or over-fitting. Crucially, you will stop reporting single numbers: every metric you produce will carry an interval that quantifies how much it could have moved by luck of the split. This module serves course outcome O4, evaluating and communicating model performance with defensible evidence, and it is the analytical spine that the deployment and monitoring work in later modules depends on.

By the end of this module

- Design a leakage-safe cross-validation procedure in which every preprocessing step that learns from data is fit inside the training folds only, and explain why anything else inflates the reported score.
- Choose evaluation metrics that fit the target type and the asymmetric costs of false positives and false negatives, and justify the choice against a stated business or scientific objective.
- Read a confusion matrix and interpret receiver-operating-characteristic (ROC) and precision-recall (PR) curves, including how moving the decision threshold trades precision against recall.
- Quantify and report uncertainty on any performance metric using a bootstrap or cross-validated confidence interval, and interpret that interval correctly.
- Diagnose subgroup performance collapse, distinguishing an aggregate score that looks strong from disaggregated scores that reveal a failure hidden by the majority class.

Readings

- An Introduction to Statistical Learning (ISLP) (free to read · link), Ch.5 resampling methods, with the accompanying Python labs. Focus on the mechanics of validation-set, leave-one-out, and k-fold cross-validation, and run the labs so you can see how fold assignment changes the estimate. Pay particular attention to the section on the right and wrong way to do cross-validation when feature selection is involved; that example is the core of this module.
- The Elements of Statistical Learning (link), Ch.7 model assessment and selection. Focus on the decomposition of expected prediction error into bias, variance, and irreducible error, on the distinction between the training error and the test error, and on Section 7.10’s worked demonstration of how cross-validation must wrap the entire modeling procedure. Read for conceptual grounding; you do not need every derivation, but you do need the bias-variance decomposition and the leakage warning.

“Advanced Data Science by Michelle Blomberg is licensed under CC BY-NC-SA 4.0.”