Title: The engagement, week by week
Course: Advanced Data Science: A Graduate Course > Section: Module 8: The RiverCity consulting engagement and defense

The engagement, week by week

This is the graded capstone workplan. Work through it across Weeks 13 to 15.

1. Week 13, intake and problem brief. Read the one-page RiverCity brief, then confirm your two named points of contact: the sponsor (played by the instructor) and the technical liaison (a rotating peer). Write a half-page intake note that restates the decision in your own words, names the decision-maker, states the unit of analysis and the outcome you will measure, and lists your three biggest open questions. Post it and schedule the first biweekly check-in.
2. Week 13, scoping and a leakage-safe plan. Turn the brief into a scoping document: the target and its measurement, the population and sampling frame, the candidate data sources and their known defects, the evaluation design (the resampling scheme, the split boundary, and exactly which steps happen inside each fold), and the metric tied to the client’s cost of being wrong. Get the plan reviewed at the check-in before you fit anything; leakage caught here is cheap.
3. Week 14, build and interim milestone. Assemble and clean the messy multi-source data, document every transformation into your lineage record, and fit a defensible baseline and one improved model under the plan you scoped. Deliver an interim milestone at the second biweekly check-in: current resampled performance with intervals, the leakage audit, and the risks you are watching. Do not tune to a single point estimate.
4. Week 14–15, the distribution-shift event. Without prior announcement, a new data batch arrives. It is a genuine two-timepoint pull (for example an earlier and a later vintage of the same FRED or Census series) or a per-student drift injector seeded to your ID, and it quietly degrades the model that looked strong at the milestone. You are expected to notice that something changed.
5. Week 15, detection, diagnosis, and adaptation. Use Evidently to produce a drift report comparing the milestone batch against the new batch, at both the input-feature and the prediction and performance level. Diagnose it: is it covariate shift, label shift, or concept drift, and which features moved? Then adapt: re-weight, re-fit, re-scope the population, or change the recommendation, and document why your fix is the right one and what it costs.
6. Week 15, assembling the client-facing deliverables. Write the executive decision memo, finalize the reproducible repository and Model Card, freeze the resampled evaluation and the drift analysis, and deploy the Streamlit app so the sponsor can explore scenarios. Everything a reader needs to trust and act on the recommendation should be in these artifacts, not in your head.
7. Week 15, the live defense. Present and defend for 20 minutes to a panel playing the executive, the compliance officer, and the engineer. Lead with the decision, show the evidence and its limits, and answer follow-ups honestly, including where the drift event changed your mind.

“Advanced Data Science by Michelle Blomberg is licensed under CC BY-NC-SA 4.0.”