Advanced Data Science

How this course works

This course is designed backward from seven learning outcomes: everything you watch, read, practice, and are assessed on traces to one of them (the full list and the week-by-week schedule are just above). Each of the eight modules follows the same rhythm, so the structure never becomes the obstacle: a short recorded introduction (Watch), a vetted open reading or lab (Read), a low-stakes interactive practice (Practice), and a summative simulation you defend on your own work (Assessment), closing with a discussion. Plan about nine hours a week across the fifteen-week term; every task states its estimated time so you can plan.

Tools you will use

• Canvas (or your institution’s LMS): where you submit work, join the weekly discussion, and receive feedback.
• Python, in Jupyter or Google Colab: the language for every analysis. Colab runs in the browser with nothing to install, which is the fastest way to start.
• The course AI tutor, a public Gemini Notebook grounded only in this course and its open resources, answering with citations. It is study support, not your instructor, and it does not replace the weekly instructor interaction.
• Interactive simulations: playable, self-scoring assessments for Modules 5 through 8. The Module 1 to 4 simulations are described in full inside those modules.
• The on-device defense-practice tool, which rehearses the oral defenses by speaking randomized questions one at a time. Your recording stays on your own device; nothing is uploaded.

Where to start

Read this module, skim the seven outcomes and the schedule above, set up Python (Colab is the quickest path), open the course tutor, and choose the one real dataset you will carry through the whole course. Then open Module 1.

Simulations and oral defenses: what is graded

The interactive simulations here are self-scoring practice you can run as often as you like. In the graded version your instructor runs in the LMS, each student is issued a uniquely seeded dataset, and additional written and code artifacts are submitted for the remaining points. Three assessments (Modules 3, 7, and 8) include a live or recorded oral defense, where a role-played stakeholder asks unscripted questions; that defense is the graded event, and the defense-practice tool lets you rehearse it first. The summative simulations and defenses carry the majority of the grade. The formative practice earns completion points: you do the hands-on work, then submit a short reflection or evidence that you ran it, and you are credited for completing it, not scored on getting it right, because practice is where mistakes are supposed to happen.

How your grade is composed

Three parts, weighted so the defended, individually-seeded work carries the most, while the practice that prepares you for it still counts.

AI use and academic integrity

AI tools are permitted and disclosed. Mark where a model helped and reflect on the judgment you exercised. The secured parts of the grade, the seeded-data recovery, the private-test prediction, the planted-defect audit, the drift response, and the live oral defense, are what establish your own achievement, so using AI honestly helps you rather than stands in for you. Never present another person’s or a model’s work as your own, and follow your institution’s academic-integrity policy.

Getting help and accessibility

Ask questions in the course Q&A space and in office hours; the instructor initiates contact every week. The course is built to WCAG 2.1 AA: videos are captioned with a slide or transcript alternative, every interactive is paired with a non-interactive path to the same concept, and most deliverables accept a written or a recorded form. If you need an accommodation, contact your disability-services office and your instructor early.

Overview

Every analysis is only as trustworthy as the data underneath it and the record of how that data arrived. This module builds the acquisition layer of the data-science lifecycle: getting data in from flat files, relational databases, REST APIs, and the live web, while capturing provenance and making the whole pull reproducible by others. You will treat version control, pinned environments, and field-level lineage as first-class deliverables rather than afterthoughts, and you will practice the ethics and mechanics of scraping responsibly. This module serves O1 (acquire data from varied structured and unstructured sources, documenting provenance and access method) and builds the reproducible-workflow foundation that O7 (construct a comprehensive data-science project from inception to presentation, and defend its decisions) depends on.

By the end of this module you will be able to

• Acquire the same dataset from a file, a relational database, and a REST API, and reconcile the three results.
• Build an API client that handles pagination, rate limiting, and retries without losing or duplicating records.
• Detect and document a schema change that occurs mid-collection, and explain how you caught it.
• Record field-level provenance (source, endpoint, parameters, retrieval time, transformations) in a datasheet-style log.
• Package an acquisition workflow so that a peer can reproduce your exact results from a pinned environment and version-controlled code.

1 · Watch

2 · Read

• Software Carpentry, Version Control with Git (CC BY 4.0 · adapt), type-along. Focus on staging versus committing, writing meaningful commit messages, and using .gitignore to keep data and credentials out of the repository; you will use these habits in every deliverable this term.
• Data Carpentry, Databases & SQL (CC BY · adapt). Focus on SELECT, WHERE, joins across tables, and aggregation with GROUP BY; pay attention to how a query is itself a reproducible, parameterized description of an extract.
• Ethical Web Scraping (link). Focus on robots.txt, rate limiting and identifying your client, and the legal and ethical weight of Terms of Service; note the distinction between using a public API and scraping the rendered page.

3 · Practice, formative, low-stakes

Work through four short hands-on run-throughs that each rehearse one acquisition mode you will combine in the assessment.

1. Complete the query chapters of Select Star SQL through joins and aggregation; keep the exact text of two queries you write, because you will reuse that pattern to pull from the database source later.
2. In Hoppscotch, send a GET request to the JSONPlaceholder /posts endpoint, then to /posts?userId=1, and inspect the response headers and status codes; note how query parameters change the returned set.
3. Still in Hoppscotch, page through /comments using the _page and _limit parameters against JSONPlaceholder and record how many total records you retrieve versus how many exist, so you can feel where a naive single call would truncate the data.
4. Practice responsible scraping in the quotes.toscrape.com sandbox: first read its robots.txt, then extract the quotes and authors from the first two pages only, adding a polite delay between requests and a descriptive user-agent.
5. Open a reproducible notebook environment on Binder from a small public repository, run it top to bottom, and confirm you get identical output; this is the environment model your own deliverables must match.

4 · Assessment, summative simulation: The Provenance Incident

A client ticket lands in your queue: the analytics team needs a single reconciled table of weekly engagement metrics assembled from four sources for a board deck due in seventy-two hours. The four sources are a CSV export from finance, a read-only replica of a product database, a partner’s flat monthly extract, and a live, rate-limited, paginated engagement API. Partway through your collection window, the API provider ships a change: a field is renamed and a nested object is flattened, with no version bump and no announcement, so records pulled before and after the change no longer share a schema. Your job is to deliver the reconciled metrics anyway, and to prove exactly where every number came from.

What you will do, step by step:

1. Initialize a version-controlled repository with a pinned environment specification and a .gitignore that excludes data and credentials; commit the empty skeleton first so history starts clean.
2. Ingest the finance CSV and the partner extract, capturing each file’s name, checksum, and receipt time; validate row counts and column types on load.
3. Query the product database replica with a parameterized SQL extract, and store the exact query text as part of the provenance record.
4. Build an API client for the engagement source that walks every page, honors the rate limit (backing off on a 429 rather than hammering), retries transient failures idempotently, and stamps each returned record with its retrieval timestamp and the request parameters that produced it.
5. Instrument the client to fingerprint the response schema on every page, so a change in field names or nesting is caught the moment it appears rather than at analysis time.
6. When the mid-window schema change fires, detect it from the fingerprint diff, quarantine the pre-change and post-change batches, and map both onto one canonical schema with a documented transformation.
7. Reconcile the four sources into one weekly table, resolving key mismatches and de-duplication, and reconcile any metric that appears in more than one source, explaining discrepancies rather than silently averaging them.
8. Validate the reconciled table against sanity checks (totals, ranges, expected weekly cardinality) and record which checks passed.
9. Write a datasheet-style provenance log with one entry per field: origin source, endpoint or query, parameters, retrieval time, and every transformation applied.
10. Write the client memo: what you delivered, that a schema change occurred, how you detected it, and what you did to keep the numbers trustworthy.

What to submit: A reproducible ingestion pipeline (version-controlled code plus a pinned environment that a grader can run from clean); a datasheet-style provenance log covering every field in the delivered table; and a one-to-two page memo that names the schema change, explains how it was detected, and states its effect on the results.

How it is graded:

5 · Discussion, Regular and Substantive Interaction

Consider a scenario from your own field where the same figure could be acquired either through an organization’s public API or by scraping the rendered web page behind it. Argue which route you would take and why, grounding your position in robots.txt, rate limits, Terms of Service, and the reproducibility and provenance costs of each choice. Post your initial position by Wednesday, then reply substantively to two peers, ideally at least one whose recommended route differs from yours, engaging their specific reasoning rather than restating your own. The instructor responds in each thread.

Overview

Real data almost never arrives analysis-ready. It carries missing entries, inconsistent units, duplicated records, mislabeled categories, and values that sit far from the rest. This module reframes cleaning, missing-data handling, outlier treatment, and feature engineering not as janitorial chores but as substantive analytic decisions that shape every downstream result. The organizing principle is that each preprocessing choice must be defensible (justified by an explicit assumption about how the data came to be the way it is), reversible (recorded so that the raw state can always be reconstructed and the transformation re-derived), and documented (logged so a reviewer can audit the reasoning, not just the code). Because dirty data is the norm rather than the exception, mastering these decisions directly serves course outcome O2: producing trustworthy, reproducible analytic pipelines from imperfect inputs. Throughout, we insist that you name the mechanism you believe generated a defect before you choose a fix, because the mechanism, not the symptom, dictates which remedy is honest.

By the end of this module you will be able to

• Distinguish the three canonical missing-data mechanisms, MCAR (missing completely at random), MAR (missing at random given observed variables), and MNAR (missing not at random), and justify which one you believe applies to a given column using evidence from the data and its provenance.
• Choose and defend an appropriate missing-data strategy (listwise deletion, single or multiple imputation, model-based approaches, or explicit missingness indicators) whose assumptions match the diagnosed mechanism.
• Separate genuine outliers from data-entry errors and apply a stated, reproducible rule for treating each, defending the choice rather than deleting by reflex.
• Engineer features in a leakage-safe way, fitting all learned transformations (scalers, encoders, imputers) on training data only and applying them to held-out data through a single reproducible pipeline.
• Maintain a cleaning log in which every transformation records its rationale, its mechanism assumption, and how it can be reversed, so the entire pipeline is auditable and reproducible.

1 · Watch

2 · Read

• Python Data Science Handbook, Chapter 3 (handling missing data, hierarchical indexing, combining datasets with joins, and split-apply-combine grouping), Jake VanderPlas, Python Data Science Handbook (text link, code adapt). Focus on the missing-data sections (the distinction between NaN and None, and detecting, dropping, and filling nulls) and on the merge/join and groupby mechanics you will need to reconcile duplicated and split records. Adapt the code examples rather than copying them; you will reuse these idioms in both deliverables.
• NIST/SEMATECH e-Handbook of Statistical Methods, §1.3.5.17, Detection of Outliers (public domain). Focus on the formal outlier tests (Grubbs, Tietjen-Moore, and the generalized ESD procedure), their distributional assumptions (approximate normality), and why an outlier flagged by a test is a candidate for investigation rather than an automatic deletion.

3 · Practice, formative, low-stakes

Build fluency with the mechanics of cleaning and the visual diagnosis of missingness before the graded simulation.

1. Complete the Kaggle Learn Data Cleaning micro-course (auto-graded), covering missing values, scaling and normalization, parsing dates, character encodings, and inconsistent-entry cleanup.
2. Complete the Kaggle Learn Feature Engineering micro-course (auto-graded), paying particular attention to constructing features without leaking target information.
3. In a notebook, load a small messy dataset and visualize its missingness with missingno, generating the matrix, bar, and heatmap/dendrogram views. Read the correlation structure of the nulls: columns whose missingness co-occurs are evidence pointing away from MCAR and toward MAR or MNAR.

4 · Assessment, summative simulation: Recover the Signal

Each learner receives a personalized messy dataset. The instructor holds privately a clean data-generating process with known parameters; from it, a per-student seed produces a corrupted copy carrying MNAR missingness (values are withheld in a way that depends on the withheld value itself, so naive deletion biases your estimates), unit inconsistencies (a subset of records reported in different units without labels), duplicate records (some exact, some near-duplicates differing by a keyed field), and outliers of a known distribution (a defined fraction drawn from a contaminating distribution, alongside genuine extremes from the true process). Your task is to recover the underlying signal. Cleaning quality is scored numerically by how closely your recovered summary statistics (means, variances, selected quantiles, and a target correlation) match the hidden truth the instructor holds. Because the truth and the answer key are private and unique to your seed, the grade rewards a principled process, not a lucky guess or a borrowed solution.

What you will do, step by step:

1. Profile the corruption: inspect distributions, dtypes, ranges, and null patterns; use missingno to map where and how values are absent before touching anything.
2. Diagnose the missingness mechanism: gather evidence for MCAR versus MAR versus MNAR (co-occurrence of nulls, dependence of missingness on observed covariates, and telltale truncation suggesting the missing value itself drives its absence), and state your conclusion with its support.
3. Reconcile units: detect the inconsistently-scaled subset, infer the conversion, and bring all records onto a common unit, documenting the detection rule.
4. Resolve duplicates: define exact-match and near-match criteria, decide which record survives a near-duplicate collision, and justify the rule.
5. Handle outliers with a defended rule: apply an outlier test from the NIST handbook or a stated robust criterion, then decide per case whether each flagged point is an error to correct/remove or a genuine extreme to retain, and record why.
6. Impute appropriately: choose a strategy whose assumptions match your diagnosed mechanism (for MNAR, this typically means modeling the mechanism or adding a missingness indicator rather than mean-filling), fitting any learned imputer without leaking information across partitions.
7. Document every step as reversible: keep the raw file immutable, transform copies, and log each transformation so the raw state can be reconstructed.
8. Assemble the steps into a single reproducible pipeline that runs end-to-end from raw file to recovered statistics.
9. Report the recovered statistics together with a confidence statement: state which recovered quantities you trust most and least, and why, given the mechanisms you diagnosed.

What to submit: a reproducible cleaning pipeline (a script or notebook that runs from the raw file to the reported numbers on a fresh machine); a decision log defending each choice with its mechanism assumption and reversibility; and the recovered summary statistics with your confidence statement.

How it is graded:

5 · Discussion, Regular and Substantive Interaction

Bring a judgment call from your own practice or reading. Present one missing-data or outlier decision where a reasonable analyst could defensibly go two ways, for example: you find a cluster of implausibly round income values, or a variable whose missingness rises exactly as its plausible true value rises. State the two candidate treatments, the mechanism assumption each rests on, and which you would defend and why. Post your initial contribution by Wednesday of Week 3, then reply substantively to at least two peers by the end of Week 4, challenging or extending their mechanism reasoning rather than merely agreeing. The instructor responds within each thread.

Overview

This module is where analysis stops being for you and starts being for someone else. You will explore an unfamiliar dataset with discipline, and then compress what you found into a defensible, honest narrative that a non-technical decision-maker can act on inside five minutes. The intellectual move is the compression: deciding what to leave out without lying by omission, choosing an encoding because it serves the decision rather than because it looks impressive, and stating what you do not know in plain words. The module serves outcome O5 (communicate findings to a non-technical decision-maker through sound visualization and a defensible data narrative) and sets up O3 (select a method appropriate to a problem and its data), since disciplined exploration is what tells you which method the data can support.

By the end of this module you will be able to

• Run principled exploratory analysis on an unfamiliar dataset: profile distributions, missingness, and skew, and separate signal worth reporting from noise and artifact.
• Choose a visual encoding by communicative intent, matching the chart form to the specific question the audience is asking rather than to the shape of the data alone.
• Spot and avoid misleading figures, including truncated or dual axes, disproportionate ink, and color that implies order or emphasis the data does not support.
• Tell a decision-focused data story: title every figure with its plain-language takeaway and state the residual uncertainty in words a VP can repeat.
• Defend an analysis live under skeptical, unscripted questioning, including answering “what would change your mind?” honestly.

1 · Watch

2 · Read

• Fundamentals of Data Visualization (Wilke) (CC BY-NC-ND · link). Read Ch.4 on color, focusing on when to use sequential versus qualitative scales and why unordered rainbow palettes mislead; then Ch.17 on proportional ink and the ways truncated and transformed axes distort comparison. Focus on the worked before-and-after figures and be able to name, for each, the specific perceptual rule being violated.
• A structured critique of a live stakeholder dashboard: pick one current visualization from Tableau Public and one chart from Our World in Data (OWID CC BY). For each, answer in a short paragraph: what decision does this figure serve, what comparison does the encoding make easiest, and where does color, axis, or ink either help or quietly mislead. Focus on whether a non-technical reader would leave with a correct takeaway. Use ONLY these URLs.

3 · Practice, formative, low-stakes

You will move from raw distributions to a defended encoding choice, then confront how easily the same data can be made to lie.

1. Run interactive EDA with Google Facets: load a dataset, sort features by distribution, and surface missing, constant, or skewed values, noting at least three features whose shape would change how you visualize them.
2. Choose an encoding with the Data-to-Viz chart chooser, working from your question and data type to a defended chart form, then study its misleading-figure catalog and identify which two caveats most threaten your chosen figure.
3. Pick a final encoding by communicative intent with the Financial Times Visual Vocabulary, matching your intent (comparison, change over time, part-to-whole, distribution) to a specific chart and justifying why it beats the runner-up. Use ONLY these URLs.

4 · Assessment, summative simulation: The 10-Minute Executive Brief

A non-technical VP of Operations sends you a real, ambiguous decision with almost no framing: “We can open one new location. East side or west side. You have 48 hours and 5 minutes of my time. Don’t send me a deck, tell me where to go and why.” You receive a messy operational dataset (foot traffic, existing store sales, demographics, costs, with gaps and inconsistencies). You do the exploratory analysis, deliver a one-page memo and a five-minute briefing with no jargon, title every chart with its plain-language takeaway, and state your uncertainty in words. Then you defend it live against a skeptical executive whose central question is “what would change your mind?”

What you will do, step by step:

1. Scope the decision: write the one comparison the VP actually needs (east versus west, on what single criterion), and the alternative you are weighing it against.
2. Do focused EDA: profile the relevant features, expose missingness and skew, and decide which data are trustworthy enough to inform the choice and which you will set aside.
3. Select two or three decision-relevant figures, and cut everything that is interesting but does not move the decision.
4. Title each figure with its takeaway, so the title alone (“West side has 30% more weekday foot traffic”) carries the point.
5. Choose honest encodings: proportional ink, full or clearly labeled axes, ordered or single-highlight color, and confirm each against the misleading-figure caveats.
6. Write the one-page memo: recommendation first, two or three reasons with the supporting figures, and a short honest statement of what you do not know.
7. Rehearse the five-minute brief out loud, cutting jargon and timing yourself, so the recommendation lands in the first thirty seconds.
8. Prepare for the live defense: list the two weakest points in your own analysis before the executive finds them.
9. Prepare a genuine answer to “what would change your mind?” naming the specific evidence that would flip your recommendation.

What to submit: a one-page decision memo (recommendation, reasons, stated uncertainty), the two or three supporting figures with takeaway titles, and a five-minute recorded or live briefing followed by the live defense.

How it is graded:

5 · Discussion, Regular and Substantive Interaction

Find a genuinely misleading figure in the wild (news, a company report, a social post) and post it with a two-paragraph critique: name the specific technique that misleads (truncated axis, disproportionate area, unordered color, cherry-picked window) and redraw or describe the honest version that serves the same reader. Post your initial critique by midweek of Week 5; then reply substantively to two peers, either strengthening their diagnosis with a named perceptual rule or contesting it with evidence from the readings. The instructor responds in each thread, pressing on whether the “fix” would actually change what a non-technical reader concludes.

Overview

Most modeling mistakes are made before a model is ever fit: the wrong problem type is chosen, or the evaluation is set up so that a good score means nothing. This module puts problem framing first. You will decide whether a task is regression (predicting a continuous quantity), classification (predicting a discrete label or its probability), or neither, and only then reach for a method. We work through three model families whose assumptions and failure modes you should be able to state from memory: linear regression, logistic regression, and tree ensembles (bagged trees and random forests). Throughout, interpretation is treated as inseparable from the fit, coefficients, odds ratios, and feature importances are read in the units of the problem, not admired as outputs. The module serves outcomes O3 (select and justify an analytic method that fits the problem and the data) and O4 (fit, interpret, and validate predictive models while stating their assumptions).

By the end of this module you will be able to

• Distinguish regression, classification, and clustering, and diagnose which one a stated business or research question actually requires.
• Select a method that fits both the problem type and the structure of the data (sample size, dimensionality, linearity, class balance), and defend the choice against at least one alternative.
• Fit and interpret linear and logistic regression, reading coefficients, odds ratios, and predicted probabilities in the units of the domain.
• Fit and interpret tree ensembles, explain why they often beat linear models on tabular data with interactions, and read their feature-importance output critically.
• State the core assumptions and failure modes of each model family and design a leakage-safe validation scheme that produces an honest estimate of out-of-sample performance.

1 · Watch

2 · Read & run labs

• An Introduction to Statistical Learning (Python edition, ISLP) (free to read · link). Read Chapter 3 (linear regression, including multiple regression, interaction terms, and diagnostics), Chapter 4 (classification, focusing on logistic regression and the confusion-matrix vocabulary), and Sections 8.1–8.2 (decision trees, then bagging and random forests). Work the Python lab notebook at the end of each chapter, do not just read them; run the cells, then perturb one thing (drop a predictor, add an interaction, change the number of trees) and predict the effect before you execute.
• Harvard CS109A runnable notebooks (MIT · adapt). Use the regression, classification, and tree/ensemble notebooks as a second pass in a different codebase. Compare how CS109A structures a train/validation/test split and a cross-validation loop against the ISLP labs, and note where the two courses make different default choices. Use ONLY these URLs.

3 · Practice, formative, low-stakes

Before you fit anything for a grade, build visual intuition for what each model family is actually doing to the data.

1. Work through MLU-Explain Linear Regression, dragging points to feel how least squares responds to leverage and outliers.
2. Work through MLU-Explain Logistic Regression, and connect the sigmoid to the log-odds interpretation from Chapter 4.
3. Work through MLU-Explain Random Forest, watching how averaging many decorrelated trees smooths the jagged decision boundary of a single tree.
4. Open TensorFlow Playground and, using only the spiral and circle datasets, observe how a purely linear model fails where added features and capacity succeed; this is the bias–variance story made visible.
5. Read R2D3, A Visual Introduction to Machine Learning to see how a tree partitions feature space and how depth trades bias for variance. Use ONLY these URLs.

4 · Assessment, summative simulation: The Private-Leaderboard Competition

You will compete in a hosted prediction competition on a client task, forecasting an outcome from tabular client data. The truth for the test set is hidden. Each submission is scored on two slices of that hidden set: a public leaderboard, updated live, computed on a minority of the test rows, and a private leaderboard, revealed only after the competition closes, computed on the remaining majority. Your grade weights three things: your private-leaderboard score, the gap between your public and private scores (a direct measure of how much you overfit the leaderboard), and a defended written rationale for the model you froze. A team that climbs the public board by chasing its noise and then collapses on the private board earns less than a team with a lower public score and almost no gap.

What you will do, step by step:

1. Establish a leakage-safe validation split first, before any modeling: hold out a validation set (or set up k-fold cross-validation) and fit every transformation, scaling, imputation, and encoding inside the training folds only.
2. Do the exploratory work and build a deliberately simple baseline (a regularized linear or logistic model) to fix a reference score you must beat.
3. Iterate on features, encoding categoricals, constructing interactions and domain-motivated variables, and record each change against your local validation metric, not the public leaderboard.
4. Iterate on models, comparing the linear baseline against tree ensembles, and tune a small number of hyperparameters using cross-validation rather than leaderboard feedback.
5. Submit to the public leaderboard sparingly, and each time compare the public score to your local validation estimate; a widening divergence is an early overfitting alarm.
6. Monitor the public/private risk explicitly: treat the public slice as one more small validation set, not as ground truth, and resist changes that help it but not your cross-validation.
7. Freeze one pipeline before the private test is scored, an end-to-end, seed-set path from raw data to prediction, and stop editing it.
8. Generate the final submission from that frozen pipeline only.
9. Write the one-page rationale defending why this model, at this complexity, on this validation evidence, was the right choice over the alternatives you rejected.

What to submit: reproducible modeling code that runs end to end from the raw data with a fixed random seed, your final competition submission file, and a one-page model-selection rationale citing your validation evidence.

How it is graded:

5 · Discussion, Regular and Substantive Interaction

Post your initial contribution by mid-week: name the model family you selected, the single strongest reason you chose it over one rejected alternative, and your current public/private gap, then argue explicitly whether that gap is telling you the model generalized or that you have started to overfit the leaderboard. By the end of the week, reply substantively to two peers, challenge a validation choice, propose a feature or a regularization change, or defend a smaller model against a flashier one on the evidence they reported. The instructor responds in each thread, pressing on leakage, interpretation, and the meaning of the gap.

Overview

A model that fits the data you already have is worthless; a model that generalizes to data you have not yet seen is the entire point. This module teaches you to evaluate predictive power honestly. You will hold data back through resampling so that your estimate of performance is not contaminated by the data the model trained on, you will choose metrics that match the target type and the real-world cost of each kind of error rather than defaulting to accuracy, and you will locate any model on the bias-variance trade-off so you understand whether it is under-fitting or over-fitting. Crucially, you will stop reporting single numbers: every metric you produce will carry an interval that quantifies how much it could have moved by luck of the split. This module serves course outcome O4, evaluating and communicating model performance with defensible evidence, and it is the analytical spine that the deployment and monitoring work in later modules depends on.

By the end of this module you will be able to

• Design a leakage-safe cross-validation procedure in which every preprocessing step that learns from data is fit inside the training folds only, and explain why anything else inflates the reported score.
• Choose evaluation metrics that fit the target type and the asymmetric costs of false positives and false negatives, and justify the choice against a stated business or scientific objective.
• Read a confusion matrix and interpret receiver-operating-characteristic (ROC) and precision-recall (PR) curves, including how moving the decision threshold trades precision against recall.
• Quantify and report uncertainty on any performance metric using a bootstrap or cross-validated confidence interval, and interpret that interval correctly.
• Diagnose subgroup performance collapse, distinguishing an aggregate score that looks strong from disaggregated scores that reveal a failure hidden by the majority class.

1 · Watch

2 · Read & run labs

• An Introduction to Statistical Learning (ISLP) (free to read · link), Ch.5 resampling methods, with the accompanying Python labs. Focus on the mechanics of validation-set, leave-one-out, and k-fold cross-validation, and run the labs so you can see how fold assignment changes the estimate. Pay particular attention to the section on the right and wrong way to do cross-validation when feature selection is involved; that example is the core of this module.
• The Elements of Statistical Learning (link), Ch.7 model assessment and selection. Focus on the decomposition of expected prediction error into bias, variance, and irreducible error, on the distinction between the training error and the test error, and on Section 7.10’s worked demonstration of how cross-validation must wrap the entire modeling procedure. Read for conceptual grounding; you do not need every derivation, but you do need the bias-variance decomposition and the leakage warning.

3 · Practice, formative, low-stakes

Work through these interactive explainers in order, then produce the short deliverable; the practice earns completion points and none of it is scored for correctness, it is here to build intuition before the summative simulation.

1. MLU-Explain Cross-Validation, step through how folds are formed and how the averaged estimate stabilizes as k changes.
2. ROC & AUC, move the threshold and watch the confusion matrix update in real time; note how the same model traces out an entire curve.
3. Bias-Variance, see how model flexibility shifts you between under-fitting and over-fitting and where test error is minimized.
4. Google MLCC Thresholding, use the imbalanced-data slider to feel how precision and recall respond as the positive class becomes rare.
5. Interpreting Confidence Intervals, watch how intervals from repeated samples behave, and internalize what “95 percent” actually refers to.

4 · Assessment, summative simulation: The Ridgeline Renewables model audit

A departing contractor left behind a “production-ready” failure-prediction model for Ridgeline Renewables, a site of five wind turbines and three solar arrays. It scores 0.97 AUC on the evaluation they handed over, with a note recommending immediate deployment. The handoff contains planted defects: target leakage (a feature that encodes the outcome), train/test contamination (records in both partitions, or preprocessing fit on the full data before splitting), a metric mismatched to the stated cost (missing a failure costs far more than a false alarm, yet the model was tuned to overall accuracy), and a subgroup, the solar arrays, on which performance collapses while the aggregate stays high. Your job is to catch each defect, re-estimate the model’s honest performance, and then sign off, condition, or reject deployment, with evidence for every claim.

What you will do, step by step:

1. Reproduce the reported 0.97 score exactly as the contractor computed it, so you have a faithful baseline to interrogate rather than a number you assume is wrong.
2. Audit the train/test split for contamination: check for duplicated or grouped records straddling the partition, and check whether any scaler, imputer, encoder, or feature selector was fit before the split.
3. Audit the feature set for target leakage: trace each suspiciously strong feature back to how it was constructed and its availability at prediction time.
4. Check the chosen metric and threshold against the stated cost structure, and determine whether the reported metric even measures what the business cares about.
5. Run disaggregated subgroup performance across the provided grouping variable to locate the subgroup where the model fails.
6. Quantify the true generalization performance with a leakage-safe resampling procedure, reporting each metric with a confidence interval rather than a point estimate.
7. Correct what can be corrected: rebuild the pipeline so preprocessing is fit inside folds, remove leaking features, and re-select a threshold aligned to the cost structure.
8. Compare the honest re-evaluation against the contractor’s 0.97 and quantify how much of the original score was artifact.
9. Write the sign-off-or-reject decision, tying each conclusion to specific evidence from your audit.
10. Package reproducible code and the audit narrative so a colleague could rerun your analysis and reach the same conclusion.

What to submit: an audit report documenting each defect found and the evidence for it, corrected and runnable evaluation code (with the leakage-safe pipeline and confidence intervals), and a signed one-page sign-off-or-reject recommendation stating your decision and its justification.

How it is graded:

5 · Discussion, Regular and Substantive Interaction

Describe a real case, from work, a project, or the news, where a single headline metric masked a costly error, and analyze the mismatch between the metric that was optimized and the cost that actually mattered. Post your initial analysis by mid-week (Wednesday of week 10), then reply substantively to at least two peers, either challenging their diagnosis or proposing a metric that would have caught the problem. The instructor responds within each thread to extend the discussion and correct misconceptions.

Overview

This module moves from supervised prediction to the harder question of what structure exists in data when no labels are given. You will apply principal component analysis to compress correlated features into interpretable axes of variance, and apply clustering to propose groupings, and then you will do the part that separates a data scientist from a button-pusher: defend those choices with evidence rather than assertion. The recurring theme is that unsupervised results are hypotheses, not answers, so every claim about “how many groups there are” must be earned through stability, internal validity, and interpretability. This module serves program outcomes O3 (differentiate among machine-learning approaches, including clustering, and select one appropriate to the problem and its data) and O5 (communicate findings to a non-technical decision-maker through sound visualization and a defensible data narrative).

By the end of this module you will be able to

• Apply PCA to a standardized feature matrix and read the loadings, scree plot, and cumulative variance explained to interpret what each component represents.
• Run k-means, explain Lloyd’s algorithm step by step (assignment then update, convergence to a local optimum), and know why initialization and standardization matter.
• Argue for a specific number of clusters using converging evidence, such as bootstrap stability, silhouette width, and the gap statistic, instead of eyeballing an elbow.
• Use t-SNE as an exploratory visualization while resisting over-reading it, correctly treating perplexity as a tuning knob and refusing to interpret cluster size or distance.
• Communicate a segmentation result to a non-technical stakeholder, stating both what the segments are and how confident you are that they are real.

1 · Watch

2 · Read & run labs

• An Introduction to Statistical Learning with Python (ISLP) (free to read · link), Chapter 12 on unsupervised learning with its PCA and clustering labs. Focus on the geometry of principal components and the proportion of variance explained, then work the k-means and hierarchical clustering labs end to end, paying attention to why the authors standardize variables before clustering.
• Harvard CS109B clustering notebooks (MIT · adapt). Focus on the notebooks that compute silhouette scores and compare clustering across k, and adapt their bootstrap or resampling patterns for the summative. Use ONLY these URLs.

3 · Practice, formative, low-stakes

Build intuition first by manipulating each method interactively, then run a short reproducible analysis on a small dataset of your choice.

1. Open Setosa PCA, and drag the points to feel how the principal components rotate to follow the direction of greatest variance and how correlation shows up as a dominant first component.
2. Open Visualizing K-Means Clustering, and step through Lloyd’s algorithm by hand: watch the assignment and update steps alternate, try a bad initialization, and see it settle into a poor local optimum.
3. Open How to Use t-SNE Effectively, and move the perplexity sliders until you have personally reproduced a misleading picture, so you never trust a single t-SNE plot again. Use ONLY these URLs.

4 · Assessment, summative simulation: How Many Segments, Really?

You are handed a customer dataset with genuine underlying structure, correlated noise columns, and non-spherical, unequally sized groups that will punish naive k-means. The marketing team asks a deceptively simple question: how many distinct customer groups actually exist, and how would you describe each one so we can target them? Your work is judged the way segmentation is really defended in industry practice: on cluster stability under bootstrap resampling, on agreement with the instructor’s private ground-truth labels that you never see, and on the rigor of your cluster-count argument. A confident answer with a weak justification scores worse than a cautious answer with strong evidence.

What you will do, step by step:

1. Explore and clean the data, then standardize the features so no single high-variance column dominates the distance metric.
2. Reduce dimensionality with PCA, report cumulative variance explained, and decide how many components to carry forward.
3. Run at least two clustering algorithms (for example k-means and a density- or linkage-based method) so your conclusion does not depend on one method’s assumptions.
4. Sweep a range of k values and record internal indices (silhouette, gap statistic, within-cluster sum of squares) for each.
5. Assess stability by bootstrap resampling: recluster many resamples and measure how consistently pairs of points co-cluster.
6. Use t-SNE or a PCA scatter only as a visual sanity check, explicitly noting what you refuse to conclude from it.
7. Choose a number of clusters and defend it with converging evidence rather than a single plot.
8. Profile each final cluster on the original features and write a plain-language description marketing can act on.
9. Package everything so a peer can rerun it from a fixed random seed and reproduce your figures.

What to submit: a reproducible clustering analysis (notebook or script with a fixed seed), a stability assessment showing bootstrap co-clustering results across candidate k values, and a one-page segment description that states your defended cluster count and characterizes each segment for a non-technical marketing reader.

How it is graded:

5 · Discussion, Regular and Substantive Interaction

Post an initial argument by mid-week on this prompt: why is “the elbow” a weak standalone argument for the number of clusters? Draw on the readings and your practice to explain what the elbow method actually measures, why it is subjective and often ambiguous, and what evidence you would add to make a count credible. Then reply substantively to at least two peers, either strengthening or challenging their proposed evidence. The instructor responds within each thread to push the reasoning further.

Overview

In this module you assess bias, fairness, privacy, and governance using a recognized framework, and you apply it not to a toy example but to a realistic, self-contained case: a classifier used to make decisions about people, built on administrative records repurposed for this use without the affected individuals’ consent. You will quantify who the model helps and who it harms, learn why no model can satisfy every reasonable definition of fairness at once, and place your findings inside the NIST AI Risk Management Framework so that an ethical concern becomes a documented, auditable engineering artifact rather than a private worry. The work culminates in a role-played governance review board where you defend a live approve, reject, or condition decision about the model under review. This module serves outcome O6 (assess the ethical dimensions of a data problem using a recognized governance framework) and O5 (communicate model behavior and limitations to technical and non-technical stakeholders).

By the end of this module you will be able to

• Audit a trained classifier for disparate impact by computing subgroup performance and a formal disparate-impact ratio.
• Explain, with reference to a specific deployment, why common fairness metrics mathematically conflict and cannot be jointly satisfied except in degenerate cases.
• Apply the four functions of the NIST AI Risk Management Framework (Govern, Map, Measure, Manage) to map identified harms to concrete controls.
• Complete a Model Card and a Datasheet for the Dataset that document a model’s behavior, limitations, and data provenance and consent.
• Recommend and defend a proportionate approve, reject, or condition governance decision, naming the tradeoffs each mitigation imposes.

1 · Watch

2 · Read

• Fairness and Machine Learning, Barocas, Hardt, and Narayanan (link). Read the Introduction and the Classification chapter. Focus on the formal definitions of group fairness criteria and the demonstration that independence, separation, and sufficiency cannot generally coexist; this is the mathematical backbone of the whole module.
• Data Feminism, D’Ignazio and Klein (open access). Read “Examine Power” and “Consider Context.” Focus on how the choice of what to count, who is classified, and whose experience is treated as default embeds power relations before any model is trained.
• NIST AI Risk Management Framework (public domain). Focus on the four core functions and their categories; you will map the case model’s harms onto them in the assessment, so read for the structure and vocabulary, not for memorization.
• Data Science Code of Professional Conduct (link). Focus on the sections on competence, confidentiality, and the duty to disclose limitations; these give you professional language for the obligations the technical work implies.

3 · Practice, formative, low-stakes

These four interactive explorables build the intuition you will formalize in the assessment, and then you run a real disparate-impact check on the module’s case classifier, the people-affecting model under review.

1. Open PAIR’s Measuring Fairness, and move the decision threshold to see how improving one group’s outcome degrades a competing fairness metric; note the exact tradeoff you cannot escape.
2. Work through MLU-Explain’s Equality of Odds, to see how equalized true-positive and false-positive rates differ from parity and calibration.
3. Play Survival of the Best Fit, a hiring-bias game that shows how historical data launders past discrimination into an automated system.
4. Audit a model with Aequitas, to see how a bias-and-fairness toolkit reports subgroup disparities against a reference group.
5. Probe predictions with Google’s What-If Tool, to inspect counterfactuals and per-subgroup performance without writing new code.

4 · Assessment, summative simulation: The Model-Governance Review Board

You will convene a NIST AI RMF review of the module’s case model, a classifier that makes decisions about people, built on administrative records repurposed without consent, whose subgroup-performance table shows real disparate impact. You will assemble a governance dossier, defend it before a rotating three-person board (a proposing data scientist, a compliance officer, and a community representative drawn from your peers), and cast a vote to approve, reject, or condition the model’s deployment, which you then defend live while the board questions you. Roles rotate so that every student sits on two other boards as well as presenting their own.

What you will do, step by step:

1. Quantify the disparate impact in your model: report subgroup selection and error rates and the disparate-impact ratio, with confidence intervals or sample sizes so the board can judge reliability.
2. Complete a Model Card documenting intended use, out-of-scope use, training data, evaluation results disaggregated by subgroup, and known limitations.
3. Complete a Datasheet for the Dataset covering motivation, composition, collection process, consent, and provenance.
4. Map the identified harms onto the four NIST AI RMF functions, naming at least one concrete control under each of Govern, Map, Measure, and Manage.
5. Propose mitigations (for example threshold adjustment, reweighting, data collection, or narrowing the deployment scope) and state the explicit cost or performance tradeoff each one imposes.
6. Take a position: approve, reject, or condition, and if condition, specify the conditions and how compliance would be verified.
7. Prepare a five-minute defense anticipating the board’s hardest question from each role.
8. Deliver the live defense, respond to board questions, and record the board’s vote and rationale.

What to submit: a governance dossier (Model Card, Datasheet for the Dataset, and the RMF harm-to-function mapping), a mitigation plan with named tradeoffs, and a written approve, reject, or condition recommendation, plus the recording or transcript of your live defense and the board’s vote.

How it is graded:

5 · Discussion, Regular and Substantive Interaction

Consider a concrete deployment the instructor names each term (for example a pretrial risk score, a loan-approval model, or a hospital readmission flag). Which fairness definition should this deployment prioritize, calibration, equalized odds, or demographic parity, and why does the context justify that choice over the alternatives? Post your initial position by midweek, naming the harm you are most trying to prevent and the tradeoff you accept. Then reply substantively to two peers who chose a different definition, engaging their reasoning rather than restating yours. The instructor facilitates each thread, pressing on unexamined assumptions and surfacing the value judgments hiding inside apparently technical claims.

Overview

The capstone is a single, defended, end-to-end project that mirrors real sponsored capstones such as those at the University of Chicago, Carnegie Mellon Heinz, and Harvard IACS, where students are retained by an external sponsor, work an ambiguous problem with messy data, and present to a review panel. Here you serve as a solo consultant to a recurring client, RiverCity, taking a genuine decision from a one-page brief through scoping, a leakage-safe build, an unannounced distribution-shift event that breaks your earlier models, a detection-and-adaptation cycle, and a live stakeholder defense. This module carries course outcome O7 (conduct and defend a complete, decision-facing data science engagement) and integrates everything from O1 through O6: framing and measurement, data acquisition and cleaning, valid evaluation, modeling, communication, and ethics and governance. Nothing here is a toy; the graded object is the record of fifteen weeks of decisions defended in real time before people who push back.

By the end of this capstone you will be able to

• Run a full consulting engagement from an ambiguous one-page brief to a live defense, negotiating scope with a real point of contact and making the decision the client actually needs made.
• Design a leakage-safe, resampled evaluation whose reported uncertainty survives contact with new data.
• Detect, diagnose, and adapt to a distribution shift that quietly breaks a model already in service, using monitoring evidence rather than intuition.
• Deliver decision-maker-facing artifacts (an executive memo, a reproducible repository with a Model Card, and an explorable app) that a non-technical sponsor can act on.
• Defend your decisions live under adversarial but fair questioning from an executive, a compliance officer, and an engineer.

1 · Watch

2 · Tools & references

• Monitor for distribution shift with Evidently, an open-source library for data and model drift reports and test suites (Evidently) (link). Use it to produce the drift report you will submit as evidence.
• Ship an explorable app the client can open in a browser with Streamlit (link). Use it to let the sponsor test scenarios against your model without reading your code.

3 · The engagement, week by week

1. Week 13, intake and problem brief. Read the one-page RiverCity brief, then confirm your two named points of contact: the sponsor (played by the instructor) and the technical liaison (a rotating peer). Write a half-page intake note that restates the decision in your own words, names the decision-maker, states the unit of analysis and the outcome you will measure, and lists your three biggest open questions. Post it and schedule the first biweekly check-in.
2. Week 13, scoping and a leakage-safe plan. Turn the brief into a scoping document: the target and its measurement, the population and sampling frame, the candidate data sources and their known defects, the evaluation design (the resampling scheme, the split boundary, and exactly which steps happen inside each fold), and the metric tied to the client’s cost of being wrong. Get the plan reviewed at the check-in before you fit anything; leakage caught here is cheap.
3. Week 14, build and interim milestone. Assemble and clean the messy multi-source data, document every transformation into your lineage record, and fit a defensible baseline and one improved model under the plan you scoped. Deliver an interim milestone at the second biweekly check-in: current resampled performance with intervals, the leakage audit, and the risks you are watching. Do not tune to a single point estimate.
4. Week 14–15, the distribution-shift event. Without prior announcement, a new data batch arrives. It is a genuine two-timepoint pull (for example an earlier and a later vintage of the same FRED or Census series) or a per-student drift injector seeded to your ID, and it quietly degrades the model that looked strong at the milestone. You are expected to notice that something changed.
5. Week 15, detection, diagnosis, and adaptation. Use Evidently to produce a drift report comparing the milestone batch against the new batch, at both the input-feature and the prediction and performance level. Diagnose it: is it covariate shift, label shift, or concept drift, and which features moved? Then adapt: re-weight, re-fit, re-scope the population, or change the recommendation, and document why your fix is the right one and what it costs.
6. Week 15, assembling the client-facing deliverables. Write the executive decision memo, finalize the reproducible repository and Model Card, freeze the resampled evaluation and the drift analysis, and deploy the Streamlit app so the sponsor can explore scenarios. Everything a reader needs to trust and act on the recommendation should be in these artifacts, not in your head.
7. Week 15, the live defense. Present and defend for 20 minutes to a panel playing the executive, the compliance officer, and the engineer. Lead with the decision, show the evidence and its limits, and answer follow-ups honestly, including where the drift event changed your mind.

4 · Assessment, summative simulation: The RiverCity Engagement (threaded from week one)

You are a solo consultant retained by RiverCity, a recurring client you have been tracking since Week 1. The one-page problem brief hands you a genuine and ambiguous decision (for example, whether and how to reallocate a fixed operating budget across service districts given uncertain demand) rather than a labeled prediction target, and the data is messy and multi-source: administrative records, a public series pulled from FRED or Census, and an operational export with missingness and inconsistent keys. You scope it, build a leakage-safe model, and reach a defensible interim result. Then, partway through, a new data batch arrives with an unannounced distribution shift that quietly breaks your earlier models, so last month’s validation no longer describes this month’s reality. You detect it with Evidently, diagnose whether it is covariate shift, label shift, or concept drift, adapt your model and possibly your recommendation, and account for the change in your memo and defense. The engagement is graded the way real sponsors and faculty grade a capstone: on the quality and defensibility of the decision, not on a leaderboard number.

What you will deliver (all client-facing):

1. An executive decision memo of at most three pages, non-technical, stating the recommendation, the quantified business impact, the uncertainty around it, and what would change the recommendation.
2. A reproducible repository in a standard structure: a README, an auditable data-lineage record, a pinned environment (for example a lockfile or a container spec), the code, and a completed Model Card describing intended use, data, performance, and limitations.
3. A resampled evaluation with reported uncertainty and a drift analysis (the Evidently report) comparing the pre-shift and post-shift batches at the feature and performance level.
4. A working Streamlit app the client can open and explore, letting them vary key inputs and see the recommendation and its uncertainty respond.
5. A 20-minute live stakeholder defense in which faculty and peers play the executive, the compliance officer, and the engineer and question you in role.

How it is graded:

5 · Discussion, Regular and Substantive Interaction

Interaction is structured as three check-in posts spread across the three weeks, each requiring a substantive peer response and each answered by the instructor acting as the RiverCity client contact. The scoping post (Week 13) shares your intake note and evaluation plan and asks one pointed question about scope or measurement; peers respond by stress-testing the leakage-safe plan, and the instructor responds in role with client priorities and constraints. The milestone post (Week 14) shares interim resampled performance and the risks you are watching; peers respond by probing the weakest assumption, and the instructor responds with a client’s reaction to the emerging recommendation. The post-drift reflection (Week 15) reports what the Evidently analysis revealed, how you diagnosed the shift, and what you changed; peers respond by challenging the diagnosis or the adaptation, and the instructor closes the loop in role and confirms readiness for the defense. Substantive means engaging the specifics of a peer’s decisions, not general encouragement.