Title: Assessment: The Model-Governance Review Board
Course: Advanced Data Science: A Graduate Course > Section: Module 7: Data and AI ethics and governance

Assessment: The Model-Governance Review Board

You will convene a NIST AI RMF review of the module’s case model, a classifier that makes decisions about people, built on administrative records repurposed without consent, whose subgroup-performance table shows real disparate impact. You will assemble a governance dossier, defend it before a rotating three-person board (a proposing data scientist, a compliance officer, and a community representative drawn from your peers), and cast a vote to approve, reject, or condition the model’s deployment, which you then defend live while the board questions you. Roles rotate so that every student sits on two other boards as well as presenting their own.

◆ The graded simulation for this module is the graded, per-student assessment for this module. Your live defense before the instructor-and-peer board is still the graded event.

The interactive simulation is the next item in this module. Open it there.

What you will do, step by step:

1. Quantify the disparate impact in your model: report subgroup selection and error rates and the disparate-impact ratio, with confidence intervals or sample sizes so the board can judge reliability.
2. Complete a Model Card documenting intended use, out-of-scope use, training data, evaluation results disaggregated by subgroup, and known limitations.
3. Complete a Datasheet for the Dataset covering motivation, composition, collection process, consent, and provenance.
4. Map the identified harms onto the four NIST AI RMF functions, naming at least one concrete control under each of Govern, Map, Measure, and Manage.
5. Propose mitigations (for example threshold adjustment, reweighting, data collection, or narrowing the deployment scope) and state the explicit cost or performance tradeoff each one imposes.
6. Take a position: approve, reject, or condition, and if condition, specify the conditions and how compliance would be verified.
7. Prepare a five-minute defense anticipating the board’s hardest question from each role.
8. Deliver the live defense, respond to board questions, and record the board’s vote and rationale.

What to submit: a governance dossier (Model Card, Datasheet for the Dataset, and the RMF harm-to-function mapping), a mitigation plan with named tradeoffs, and a written approve, reject, or condition recommendation, plus the recording or transcript of your live defense and the board’s vote.

How it is graded:

Criterion | What strong work shows | Pts
Disparate-impact analysis | Correct subgroup metrics and disparate-impact ratio, with sample sizes or intervals and honest treatment of small subgroups. | 15
Model Card completeness | Intended and out-of-scope use, disaggregated evaluation, and candid limitations, not marketing language. | 15
Datasheet completeness | Clear provenance, collection process, and a specific account of consent and its limits. | 14
RMF mapping | Each of Govern, Map, Measure, and Manage carries a concrete, non-generic control tied to this model’s actual harms. | 14
Mitigation proportionality | Mitigations match the severity of the harm and each names the metric or capability it degrades. | 14
Defended vote | The approve, reject, or condition position follows from the evidence and survives the board’s questions. | 14
Live defense | Clear, responsive answers under questioning that show ownership of the model’s tradeoffs. | 14
Total | 100

◆ Why this resists generative substitution: The dossier is grounded in this case model’s specific subgroup numbers, the mapping uses the actual RMF functions rather than generic principles, and the vote must be defended live under questioning that reacts to the individual’s answers. A generic AI-generated ethics essay cannot commit to a proportionate mitigation with a named cost and cannot answer the board’s follow-up in real time.

“Advanced Data Science by Michelle Blomberg is licensed under CC BY-NC-SA 4.0.”