Ridgeline Renewables: audit a model before it ships

AI Tools & Strategy · Learning Design

The graded summative for Module 5 of my graduate Advanced Data Science course. A contractor left a “production-ready” failure-prediction model that scores 0.97 and says deploy now. Your job is to audit it, find every planted defect, re-estimate its honest performance, and decide whether it ships, then defend that call.

What this is

A performance-based summative assessment for this module, worth 100 points. The interactive below takes about 30 to 45 minutes and covers the diagnosis and the defense, and it is self-scoring, so you can run it on your own. In the instructor-led version of the course the full assessment, spread across the two-week module, also includes the corrected evaluation code and the written audit report you submit, which is where most of the hours go, and each student is issued their own version of the model, seeded from their roster identifier, so the specific defects differ from one submission to the next and the instructor can reproduce any of them.

Why you are doing this

A model that fits the data you already have is worthless; a model that generalizes to data you have not seen is the whole point. In the real world nobody hands you a clean model, they hand you a confident number and ask you to sign off. This assessment tests whether you can refuse to trust a number until you have interrogated how it was produced, the single most important habit this course teaches.

What you are being assessed on

100 points total

15Contamination foundTrain/test leakage in the split

15Target leakage foundA feature that encodes the answer

14Right metricMetric and threshold vs real cost

14Subgroup collapseWhere the aggregate hides a failure

14Honest re-evaluationCorrected estimate with an interval

14Defensible decisionShip, reject, or condition, defended

14ReproducibilityGraded on your submitted code

What you submit: in the instructor-led version this interactive covers 86 of the 100 points, and the other 14 come from the corrected evaluation code and a one-page written audit report you upload to the Module 5 assignment in the LMS. Working on your own, the interactive is self-scoring practice you can complete without an instructor.

RIDGELINE RENEWABLES · MODEL REVIEW

THE HANDOFF

Your job: a contractor says this model is ready to deploy. Work through the six steps to prove whether it really is, then decide what to do about it.

This simulation is an open educational resource by Michelle Blomberg, released under CC BY-NC-SA 4.0. Reuse, adapt, and share for noncommercial purposes with attribution, under the same license.