Title: Assessment: The RiverCity Engagement
Course: Advanced Data Science: A Graduate Course > Section: Module 8: The RiverCity consulting engagement and defense

Assessment: The RiverCity Engagement

You are a solo consultant retained by RiverCity, a recurring client you have been tracking since Week 1. The one-page problem brief hands you a genuine and ambiguous decision (for example, whether and how to reallocate a fixed operating budget across service districts given uncertain demand) rather than a labeled prediction target, and the data is messy and multi-source: administrative records, a public series pulled from FRED or Census, and an operational export with missingness and inconsistent keys. You scope it, build a leakage-safe model, and reach a defensible interim result. Then, partway through, a new data batch arrives with an unannounced distribution shift that quietly breaks your earlier models, so last month’s validation no longer describes this month’s reality. You detect it with Evidently, diagnose whether it is covariate shift, label shift, or concept drift, adapt your model and possibly your recommendation, and account for the change in your memo and defense. The engagement is graded the way real sponsors and faculty grade a capstone: on the quality and defensibility of the decision, not on a leaderboard number.

What you will deliver (all client-facing):

1. An executive decision memo of at most three pages, non-technical, stating the recommendation, the quantified business impact, the uncertainty around it, and what would change the recommendation.
2. A reproducible repository in a standard structure: a README, an auditable data-lineage record, a pinned environment (for example a lockfile or a container spec), the code, and a completed Model Card describing intended use, data, performance, and limitations.
3. A resampled evaluation with reported uncertainty and a drift analysis (the Evidently report) comparing the pre-shift and post-shift batches at the feature and performance level.
4. A working Streamlit app the client can open and explore, letting them vary key inputs and see the recommendation and its uncertainty respond.
5. A 20-minute live stakeholder defense in which faculty and peers play the executive, the compliance officer, and the engineer and question you in role.

How it is graded:

Criterion | What strong work shows | Pts
Quality of the decision | The recommendation answers the decision the client actually faces, is tied to their cost of being wrong, and states the conditions under which it would flip. | 13
Validity of the evaluation design | The resampling scheme and split boundary prevent leakage, preprocessing happens inside folds, and reported uncertainty is honest rather than a single point estimate. | 13
Drift detection and adaptation | The shift is detected with evidence (the Evidently report), correctly diagnosed as covariate, label, or concept shift, and met with an adaptation whose cost and rationale are documented. | 13
Reproducibility and repository quality | A stranger can clone the repo, restore the pinned environment, follow the data lineage, and regenerate the results; the Model Card is complete and candid. | 13
Business impact and communication | The memo and app make the impact and uncertainty legible to a non-technical decision-maker in minutes, with quantities the sponsor can act on. | 12
The live defense | Answers hold under follow-up from all three roles, concede real limitations, and show that the presenter, not a tool, made and understands the decisions. | 12
Ethics and governance | The work names who could be harmed, checks for disparate impact across districts or groups, respects data provenance and consent, and states governance and monitoring for ongoing use. | 12
Professionalism of the engagement | Intake, scoping, and check-ins were used well; the client contact was kept informed; scope changes were negotiated rather than assumed. | 12
Total | 100

◆ Why this resists generative substitution: Three things a model cannot supply on the student’s behalf sit at the center of this assessment. First, the ground truth is hidden and specific to the seeded engagement, so there is no public answer to retrieve. Second, the mid-project drift arrives after the first model is built and quietly invalidates it, so a defensible result requires noticing and reacting to a change no one announced. Third, the defense is live and unscripted, and follow-up questions from the executive, the compliance officer, and the engineer probe whether the person in the room actually made the decisions. The graded object is fifteen weeks of decisions defended in real time, which is exactly what a generative tool cannot have lived.

“Advanced Data Science by Michelle Blomberg is licensed under CC BY-NC-SA 4.0.”