Title: Assessment: The Provenance Incident
Course: Advanced Data Science: A Graduate Course > Section: Module 1: Acquisition and the reproducible workflow

Assessment: The Provenance Incident

A client ticket lands in your queue: the analytics team needs a single reconciled table of weekly engagement metrics assembled from four sources for a board deck due in seventy-two hours. The four sources are a CSV export from finance, a read-only replica of a product database, a partner’s flat monthly extract, and a live, rate-limited, paginated engagement API. Partway through your collection window, the API provider ships a change: a field is renamed and a nested object is flattened, with no version bump and no announcement, so records pulled before and after the change no longer share a schema. Your job is to deliver the reconciled metrics anyway, and to prove exactly where every number came from.

What you will do, step by step:

1. Initialize a version-controlled repository with a pinned environment specification and a .gitignore that excludes data and credentials; commit the empty skeleton first so history starts clean.
2. Ingest the finance CSV and the partner extract, capturing each file’s name, checksum, and receipt time; validate row counts and column types on load.
3. Query the product database replica with a parameterized SQL extract, and store the exact query text as part of the provenance record.
4. Build an API client for the engagement source that walks every page, honors the rate limit (backing off on a 429 rather than hammering), retries transient failures idempotently, and stamps each returned record with its retrieval timestamp and the request parameters that produced it.
5. Instrument the client to fingerprint the response schema on every page, so a change in field names or nesting is caught the moment it appears rather than at analysis time.
6. When the mid-window schema change fires, detect it from the fingerprint diff, quarantine the pre-change and post-change batches, and map both onto one canonical schema with a documented transformation.
7. Reconcile the four sources into one weekly table, resolving key mismatches and de-duplication, and reconcile any metric that appears in more than one source, explaining discrepancies rather than silently averaging them.
8. Validate the reconciled table against sanity checks (totals, ranges, expected weekly cardinality) and record which checks passed.
9. Write a datasheet-style provenance log with one entry per field: origin source, endpoint or query, parameters, retrieval time, and every transformation applied.
10. Write the client memo: what you delivered, that a schema change occurred, how you detected it, and what you did to keep the numbers trustworthy.

What to submit: A reproducible ingestion pipeline (version-controlled code plus a pinned environment that a grader can run from clean); a datasheet-style provenance log covering every field in the delivered table; and a one-to-two page memo that names the schema change, explains how it was detected, and states its effect on the results.

How it is graded:

Criterion | What strong work shows | Pts
Reproducibility | A grader runs the pipeline from a clean, pinned environment and obtains the submitted table without manual patching; code and data-versus-secrets separation are clean. | 15
API handling | The client walks all pages, respects the rate limit with backoff, retries idempotently, and neither truncates nor duplicates records. | 15
Schema-change detection | The change is caught automatically from response fingerprints, with pre- and post-change batches identified and timestamped, not discovered by accident downstream. | 14
Provenance completeness | Every field in the delivered table traces to source, endpoint or query, parameters, retrieval time, and transformations, with no orphan columns. | 14
Reconciliation correctness | The four sources are joined on correct keys, duplicates resolved, and overlapping metrics reconciled with discrepancies explained rather than masked. | 14
Validation | Sanity checks on totals, ranges, and cardinality are present, run, and reported, including any that failed and how they were handled. | 14
Memo | Clear, honest, and specific about the schema change, its detection, and its impact, in language a client stakeholder can act on. | 14
Total | 100

◆ Why this resists generative substitution: The engagement API is private and moving, and its schema changes at an unannounced moment during each student’s own collection window; the grade rests on the lineage that only that individual run could have produced. A model cannot fabricate retrieval timestamps, page-by-page fingerprints, and a reconciliation that matches this student’s specific pre- and post-change batches, because that record exists only as a byproduct of actually running the pull.

“Advanced Data Science by Michelle Blomberg is licensed under CC BY-NC-SA 4.0.”