Title: Overview
Course: Advanced Data Science: A Graduate Course > Section: Module 2: Preprocessing messy and dynamic data

Module 2: Preprocessing messy and dynamic data

Overview

Real data almost never arrives analysis-ready. It carries missing entries, inconsistent units, duplicated records, mislabeled categories, and values that sit far from the rest. This module reframes cleaning, missing-data handling, outlier treatment, and feature engineering not as janitorial chores but as substantive analytic decisions that shape every downstream result. The organizing principle is that each preprocessing choice must be defensible (justified by an explicit assumption about how the data came to be the way it is), reversible (recorded so that the raw state can always be reconstructed and the transformation re-derived), and documented (logged so a reviewer can audit the reasoning, not just the code). Because dirty data is the norm rather than the exception, mastering these decisions directly serves course outcome O2: producing trustworthy, reproducible analytic pipelines from imperfect inputs. Throughout, we insist that you name the mechanism you believe generated a defect before you choose a fix, because the mechanism, not the symptom, dictates which remedy is honest.

By the end of this module

- Distinguish the three canonical missing-data mechanisms, MCAR (missing completely at random), MAR (missing at random given observed variables), and MNAR (missing not at random), and justify which one you believe applies to a given column using evidence from the data and its provenance.
- Choose and defend an appropriate missing-data strategy (listwise deletion, single or multiple imputation, model-based approaches, or explicit missingness indicators) whose assumptions match the diagnosed mechanism.
- Separate genuine outliers from data-entry errors and apply a stated, reproducible rule for treating each, defending the choice rather than deleting by reflex.
- Engineer features in a leakage-safe way, fitting all learned transformations (scalers, encoders, imputers) on training data only and applying them to held-out data through a single reproducible pipeline.
- Maintain a cleaning log in which every transformation records its rationale, its mechanism assumption, and how it can be reversed, so the entire pipeline is auditable and reproducible.

Readings

- Python Data Science Handbook, Chapter 3 (handling missing data, hierarchical indexing, combining datasets with joins, and split-apply-combine grouping), Jake VanderPlas, Python Data Science Handbook (text link, code adapt). Focus on the missing-data sections (the distinction between NaN and None, and detecting, dropping, and filling nulls) and on the merge/join and groupby mechanics you will need to reconcile duplicated and split records. Adapt the code examples rather than copying them; you will reuse these idioms in both deliverables.
- NIST/SEMATECH e-Handbook of Statistical Methods, §1.3.5.17, Detection of Outliers (public domain). Focus on the formal outlier tests (Grubbs, Tietjen-Moore, and the generalized ESD procedure), their distributional assumptions (approximate normality), and why an outlier flagged by a test is a candidate for investigation rather than an automatic deletion.

“Advanced Data Science by Michelle Blomberg is licensed under CC BY-NC-SA 4.0.”