Ridgeline Renewables

Case study · AI Tools & Strategy · Learning Design

A departing contractor left a “production-ready” failure-prediction model that scores 0.97 and a note that says deploy it now. Your job is to find out whether that number is real before anything ships.

Module 5 · Model evaluation · summative simulation

The Ridgeline Renewables site: five turning wind turbines and three solar arrays under a blue sky

Run the simulation

Randomized on every run here, and deterministically seeded per learner when opened with a roster identifier, so an instructor can reproduce a graded attempt exactly. It runs in your browser, stores no personal data, and covers the graded diagnosis and stakeholder defense. Start in the panel below, or open it full screen →. To see the seeding, open it as an assigned learner → and reload: the scenario does not change.

What this simulation is

Ridgeline Renewables is the summative simulation for the model-evaluation module of the graduate data-science course. The learner inherits a model for a site of five wind turbines and three solar arrays, scored at 0.97 AUC on the evaluation the contractor handed over. The handoff hides four planted defects: target leakage (a feature that encodes the outcome), train/test contamination (records in both partitions, or preprocessing fit before the split), a metric mismatched to the stated cost (missing a failure costs far more than a false alarm, yet the model was tuned to accuracy), and a subgroup, the solar arrays, on which performance collapses while the aggregate stays high.

The student audits the model, re-estimates its honest performance with a leakage-safe procedure, and then signs off, conditions, or rejects deployment, with evidence for every claim.

What it assesses

Finding train/test contamination and preprocessing-before-split leakage, with evidence.
Tracing a leaking feature to its construction and showing it would be unavailable at prediction time.
Choosing and justifying a metric and threshold aligned to an asymmetric cost.
Disaggregating performance to locate the subgroup the aggregate score concealed.
Re-estimating honest performance with confidence intervals, not point estimates.
Reaching a defensible sign-off, condition, or reject decision the evidence supports.

Why it resists AI substitution

The defects are planted and seeded per student, so the leaking feature, the contaminated records, the mismatched threshold, and the failing subgroup differ across submissions. A defensible answer requires diagnosing this particular model’s specific failures and reproducing the corrected numbers, which cannot be produced as generic text.