Title: Assessment: The Private-Leaderboard Competition
Course: Advanced Data Science: A Graduate Course > Section: Module 4: Supervised learning: regression and classification

Assessment: The Private-Leaderboard Competition

You will compete in a hosted prediction competition on a client task, forecasting an outcome from tabular client data. The truth for the test set is hidden. Each submission is scored on two slices of that hidden set: a public leaderboard, updated live, computed on a minority of the test rows, and a private leaderboard, revealed only after the competition closes, computed on the remaining majority. Your grade weights three things: your private-leaderboard score, the gap between your public and private scores (a direct measure of how much you overfit the leaderboard), and a defended written rationale for the model you froze. A team that climbs the public board by chasing its noise and then collapses on the private board earns less than a team with a lower public score and almost no gap.

What you will do, step by step:

1. Establish a leakage-safe validation split first, before any modeling: hold out a validation set (or set up k-fold cross-validation) and fit every transformation, scaling, imputation, and encoding inside the training folds only.
2. Do the exploratory work and build a deliberately simple baseline (a regularized linear or logistic model) to fix a reference score you must beat.
3. Iterate on features, encoding categoricals, constructing interactions and domain-motivated variables, and record each change against your local validation metric, not the public leaderboard.
4. Iterate on models, comparing the linear baseline against tree ensembles, and tune a small number of hyperparameters using cross-validation rather than leaderboard feedback.
5. Submit to the public leaderboard sparingly, and each time compare the public score to your local validation estimate; a widening divergence is an early overfitting alarm.
6. Monitor the public/private risk explicitly: treat the public slice as one more small validation set, not as ground truth, and resist changes that help it but not your cross-validation.
7. Freeze one pipeline before the private test is scored, an end-to-end, seed-set path from raw data to prediction, and stop editing it.
8. Generate the final submission from that frozen pipeline only.
9. Write the one-page rationale defending why this model, at this complexity, on this validation evidence, was the right choice over the alternatives you rejected.

What to submit: reproducible modeling code that runs end to end from the raw data with a fixed random seed, your final competition submission file, and a one-page model-selection rationale citing your validation evidence.

How it is graded:

Criterion | What strong work shows | Pts
Private-leaderboard score | Competitive predictive performance on the hidden majority slice, the score that survives when the leaderboard can no longer be gamed. | 15
Public/private gap | A small gap between public and private scores, evidence the model generalized rather than fitting leaderboard noise. | 15
Leakage-safe validation | All preprocessing fit inside training folds; a local cross-validation estimate that tracks the private score closely. | 14
Defensible model selection | A rationale that names the alternatives considered, the assumptions of the chosen family, and the evidence for stopping at the chosen complexity. | 14
Interpretation | Coefficients, odds ratios, or feature importances read correctly in the units of the client problem, with their limits acknowledged. | 14
Reproducibility | Code that reruns from raw data to submission with a fixed seed and produces the reported numbers. | 14
Discipline under pressure | A frozen pipeline and a submission history showing restrained, principled use of the public leaderboard. | 14
Total | 100

◆ Why this resists generative substitution: The private test set is hidden and revealed only after the competition closes, so no produced artifact, copied notebook, or leaderboard hack can be tuned against it in advance. The overfitting gap actively penalizes the exact behavior that copied “winning solutions” encourage, chasing the visible public score, and the grade rewards disciplined validation and a defended rationale, which are processes a student must own rather than outputs a model can hand back.

“Advanced Data Science by Michelle Blomberg is licensed under CC BY-NC-SA 4.0.”