Title: Overview
Course: Advanced Data Science: A Graduate Course > Section: Module 1: Acquisition and the reproducible workflow

Module 1: Acquisition and the reproducible workflow

Overview

Every analysis is only as trustworthy as the data underneath it and the record of how that data arrived. This module builds the acquisition layer of the data-science lifecycle: getting data in from flat files, relational databases, REST APIs, and the live web, while capturing provenance and making the whole pull reproducible by others. You will treat version control, pinned environments, and field-level lineage as first-class deliverables rather than afterthoughts, and you will practice the ethics and mechanics of scraping responsibly. This module serves O1 (acquire data from varied structured and unstructured sources, documenting provenance and access method) and builds the reproducible-workflow foundation that O7 (construct a comprehensive data-science project from inception to presentation, and defend its decisions) depends on.

By the end of this module

- Acquire the same dataset from a file, a relational database, and a REST API, and reconcile the three results.
- Build an API client that handles pagination, rate limiting, and retries without losing or duplicating records.
- Detect and document a schema change that occurs mid-collection, and explain how you caught it.
- Record field-level provenance (source, endpoint, parameters, retrieval time, transformations) in a datasheet-style log.
- Package an acquisition workflow so that a peer can reproduce your exact results from a pinned environment and version-controlled code.

Readings

- Software Carpentry, Version Control with Git (CC BY 4.0 · adapt), type-along. Focus on staging versus committing, writing meaningful commit messages, and using .gitignore to keep data and credentials out of the repository; you will use these habits in every deliverable this term.
- Data Carpentry, Databases & SQL (CC BY · adapt). Focus on SELECT, WHERE, joins across tables, and aggregation with GROUP BY; pay attention to how a query is itself a reproducible, parameterized description of an extract.
- Ethical Web Scraping (link). Focus on robots.txt, rate limiting and identifying your client, and the legal and ethical weight of Terms of Service; note the distinction between using a public API and scraping the rendered page.

“Advanced Data Science by Michelle Blomberg is licensed under CC BY-NC-SA 4.0.”