Title: Assessment: The Ridgeline Renewables model audit
Course: Advanced Data Science: A Graduate Course > Section: Module 5: Rigorous model evaluation

Assessment: The Ridgeline Renewables model audit

A departing contractor left behind a “production-ready” failure-prediction model for Ridgeline Renewables, a site of five wind turbines and three solar arrays. It scores 0.97 AUC on the evaluation they handed over, with a note recommending immediate deployment. The handoff contains planted defects: target leakage (a feature that encodes the outcome), train/test contamination (records in both partitions, or preprocessing fit on the full data before splitting), a metric mismatched to the stated cost (missing a failure costs far more than a false alarm, yet the model was tuned to overall accuracy), and a subgroup, the solar arrays, on which performance collapses while the aggregate stays high. Your job is to catch each defect, re-estimate the model’s honest performance, and then sign off, condition, or reject deployment, with evidence for every claim.

◆ The graded simulation for this module is the graded, per-student assessment for this module. It covers 86 of the 100 points; the remaining 14 come from the corrected evaluation code and written audit report you submit here.

The interactive simulation is the next item in this module. Open it there.

What you will do, step by step:

1. Reproduce the reported 0.97 score exactly as the contractor computed it, so you have a faithful baseline to interrogate rather than a number you assume is wrong.
2. Audit the train/test split for contamination: check for duplicated or grouped records straddling the partition, and check whether any scaler, imputer, encoder, or feature selector was fit before the split.
3. Audit the feature set for target leakage: trace each suspiciously strong feature back to how it was constructed and its availability at prediction time.
4. Check the chosen metric and threshold against the stated cost structure, and determine whether the reported metric even measures what the business cares about.
5. Run disaggregated subgroup performance across the provided grouping variable to locate the subgroup where the model fails.
6. Quantify the true generalization performance with a leakage-safe resampling procedure, reporting each metric with a confidence interval rather than a point estimate.
7. Correct what can be corrected: rebuild the pipeline so preprocessing is fit inside folds, remove leaking features, and re-select a threshold aligned to the cost structure.
8. Compare the honest re-evaluation against the contractor’s 0.97 and quantify how much of the original score was artifact.
9. Write the sign-off-or-reject decision, tying each conclusion to specific evidence from your audit.
10. Package reproducible code and the audit narrative so a colleague could rerun your analysis and reach the same conclusion.

What to submit: an audit report documenting each defect found and the evidence for it, corrected and runnable evaluation code (with the leakage-safe pipeline and confidence intervals), and a signed one-page sign-off-or-reject recommendation stating your decision and its justification.

How it is graded:

Criterion | What strong work shows | Pts
Contamination found | Identifies the train/test contamination and the preprocessing-before-split leakage, with evidence showing where the boundary was violated. | 15
Target leakage found | Traces the leaking feature to its construction and demonstrates it would be unavailable or outcome-encoding at prediction time. | 15
Correct metric chosen | Selects and justifies a metric and threshold aligned to the stated asymmetric cost, and shows why accuracy or raw AUC was inappropriate here. | 14
Subgroup collapse detected | Disaggregates performance and pinpoints the subgroup where the model fails, quantifying the gap the aggregate score concealed. | 14
Honest re-evaluation | Re-estimates performance with a leakage-safe resampling procedure and reports metrics with defensible confidence intervals, not point estimates. | 14
Defensible decision | Reaches a sign-off-or-reject recommendation that follows logically from the evidence, with no claim unsupported by the audit. | 14
Reproducibility | Submits code a colleague can rerun end to end to reproduce every reported number and reach the same conclusion. | 14
Total | 100

◆ Why this resists generative substitution: the defects are planted and seeded per student, so the leaking feature, the contaminated records, the mismatched threshold, and the failing subgroup differ across submissions; a defensible answer requires diagnosing this particular model’s specific failures and reproducing the corrected numbers, which cannot be produced as generic text.

“Advanced Data Science by Michelle Blomberg is licensed under CC BY-NC-SA 4.0.”