Ridgeline Renewables

Ridgeline Renewables is the summative simulation for the model-evaluation module of the graduate data-science course. The learner inherits a model for a site of five wind turbines and three solar arrays, scored at 0.97 AUC on the evaluation the contractor handed over. The handoff hides four planted defects: target leakage (a feature that encodes the outcome), train/test contamination (records in both partitions, or preprocessing fit before the split), a metric mismatched to the stated cost (missing a failure costs far more than a false alarm, yet the model was tuned to accuracy), and a subgroup, the solar arrays, on which performance collapses while the aggregate stays high.

The student audits the model, re-estimates its honest performance with a leakage-safe procedure, and then signs off, conditions, or rejects deployment, with evidence for every claim.

• Finding train/test contamination and preprocessing-before-split leakage, with evidence.
• Tracing a leaking feature to its construction and showing it would be unavailable at prediction time.
• Choosing and justifying a metric and threshold aligned to an asymmetric cost.
• Disaggregating performance to locate the subgroup the aggregate score concealed.
• Re-estimating honest performance with confidence intervals, not point estimates.
• Reaching a defensible sign-off, condition, or reject decision the evidence supports.

An assessment that varies per student raises three fair questions: can the instructor reproduce a specific attempt, is the variation tied to who received it, and can a student obtain someone else’s version.

What the demonstrations do today. Each run randomizes the scenario in the browser. Nothing is recorded, nothing identifies the person running it, and there is no seed value to retrieve. That is the right design for an open demonstration, where anyone can try it and no one should be tracked. It is not sufficient for grading, and it is not presented as if it were.

What the graded version requires. Three things, in this order.

1. Derive the seed from the learner, not from chance. The seed is a hash of the roster identifier the institution already assigns, so the same student always receives the same scenario and two students almost never share one. It is deterministic, which means it needs no server and no stored record.
2. Surface the seed at submission. The results screen prints the seed and the learner includes it with their work. The instructor pastes it into an instructor view to regenerate the exact scenario that was graded, which is what makes a disputed grade reviewable.
3. Bind the seed to the person. A short signature computed from the identifier and the seed together is printed alongside it. A learner can copy a classmate’s seed, but the signature will not match their own identifier, so a borrowed scenario is visible rather than hidden.

Deriving the seed from identity also closes a quieter problem: with unseeded randomness, a learner can reload until the scenario resembles one they have already seen solved. A deterministic seed removes the reroll without adding surveillance, which is the trade this whole suite is built around.

Status. Built and working in all three scenario-based simulations: Ridgeline, Segments, and the governance board. Opened plainly each runs as open practice: varied, nothing recorded, nobody identified. Add a roster identifier, as in sim-ridgeline.html?id=jrivera, and that learner’s version is fixed, the seed and signature print on the results brief, and reopening with &seed= regenerates the graded attempt exactly, down to the plotted points. Running a classmate’s seed under your own identifier returns a signature mismatch rather than passing quietly. The capstone has no scenario to seed.

One honest limit: the signature is computed in the browser, so it deters casual seed-swapping rather than a determined attacker who reads the source. Making it tamper-proof means computing it where the learner cannot see it, in the LMS or a small service, which is the correct place for it once this is graded rather than demonstrated. The seed derivation does not change, only where the signature is calculated.