Title: Practice and formative deliverable
Course: Advanced Data Science: A Graduate Course > Section: Module 5: Rigorous model evaluation

Practice and formative deliverable

The practice items below are formative and ungraded. They build intuition before the graded assessment. The formative deliverable is checked for completion and returned with feedback, and it earns completion points on submission, credited on completion rather than scored for correctness.

Work through these interactive explainers in order, then produce the short deliverable; the practice earns completion points and none of it is scored for correctness, it is here to build intuition before the summative simulation.

1. MLU-Explain Cross-Validation, step through how folds are formed and how the averaged estimate stabilizes as k changes.
2. ROC & AUC, move the threshold and watch the confusion matrix update in real time; note how the same model traces out an entire curve.
3. Bias-Variance, see how model flexibility shifts you between under-fitting and over-fitting and where test error is minimized.
4. Google MLCC Thresholding, use the imbalanced-data slider to feel how precision and recall respond as the positive class becomes rare.
5. Interpreting Confidence Intervals, watch how intervals from repeated samples behave, and internalize what “95 percent” actually refers to.

◎ Formative deliverable: Take any model you have trained (or a provided baseline), pick one metric appropriate to its target, and report that metric with either a bootstrap or a cross-validated confidence interval. Add one sentence explaining why the point estimate alone would mislead a decision-maker. Checked by a short self-assessment checklist and instructor spot-review in office hours; credited complete or incomplete on submission with a short reflection.

“Advanced Data Science by Michelle Blomberg is licensed under CC BY-NC-SA 4.0.”