17 Prediction and Generalization
This chapter is part of a book in active development and has not yet been through the author’s review. Content may change as the review advances.
The research decision. Decide whether your question is truly a forecast about cases nobody has seen yet, and if it is, sign a four-part contract before you fit anything: target, baseline, split, metric, plus a leakage check. That contract is what lets you defend a modest honest score and say out loud where your forecast stops working.
17.1 Why this decision matters
The decision on the table: whether you are forecasting unseen cases, and if you are, what score you would accept as honest.
Picture a county public-health officer who must decide, week by week, whether to post a swimming advisory at a lake. Blooms of cyanobacteria can turn a safe beach into a health hazard within days. A vendor arrives with a model that flags harmful blooms “with 94% accuracy.”
“Ninety-four percent compared to WHAT? Most weeks this lake is fine, so if I just say ‘no bloom’ every single week I am already right about eighty-five percent of the time. If your model cannot clearly beat that, and prove it did so on weeks it never studied, I am not closing a public beach on its say-so.” — a public-health officer who has been burned by an impressive-sounding number
That officer asks the two questions this chapter trains you to answer before you trust any forecast. Is it better than the dumbest honest rule, and was it earned on cases the model never saw? Get those wrong and you cry wolf or miss a real hazard.
17.2 The concept
This pathway is the book’s own addition. The four pathways around it adapt RDSS’s design library; prediction extends that library rather than adapting a part of it, because a forecast is judged by how it performs on cases nobody has seen yet (Blair et al. 2023).
A prediction is a best guess about a case whose outcome you cannot see yet. Example: forecasting whether a lake blooms next week, before next week exists. Prediction is descriptive, not causal. It forecasts what will happen, never why.
What makes prediction its own question is its reach, the cases a claim is meant to cover. Prediction’s reach is unseen cases, units not in your data whose outcome is still unknown, such as next month’s samples. Its cousin is generalization, reach from your sample to a broader population, such as from the twelve lakes you measured to every lake in the watershed. Both go beyond the data in hand, and both are honest only when you can name the crossing that licenses them.
A forecast earns trust under a four-part contract, in fixed order. The target is the one column you predict, for example bloom_next_week. The baseline is the dumbest honest rule you must beat, usually “always guess the most common answer”; if 85% of weeks have no bloom, that rule scores 85% for free, so 85% is the bar. The metric is how you keep score, matched to the target; accuracy misleads when blooms are rare, so recall on the bloom class, the share of real blooms the model catches, often matters more. Whichever you choose, score the baseline and every candidate on the same metric, or the comparison means nothing.
The split is where most honest-looking forecasts go wrong, so it gets three parts rather than two. The training set is the data each candidate model learns from. Example: the model works out its coefficients from your first summer. The selection set is separate data you use to choose among candidates. Example: you compare three feature lists and pick one on your second summer. The final holdout is data you lock away and open exactly once, after every choice is settled. Example: only the model you already chose ever touches your third summer.
Run them in that order. Fit on training, choose on selection, then score your one choice on the final holdout. The reason for the middle step is easy to miss. When you try twelve models and keep the best score you saw, that winning score is flattered by luck on average: some model was always going to look best on that particular data, partly because its errors happened to fall kindly there. On any one split the flattery can be large, small, or even reversed; over many splits it points up. Model-selection bias is that flattery, the gap between a winner’s score on the data that crowned it and its honest score on data it has never met (Varma and Simon 2006). Report the crowning score as final and you will promise performance the model cannot deliver.
One consequence is worth stating plainly. If the final holdout makes you go back and change a feature, a threshold, or a model, it has joined development and stopped being an exam. You then need fresh untouched data for the next final check. In the simple workflow this book teaches, seeing the final score seals the analysis; any change after it breaks the seal. Specialists have techniques for careful reuse, and they are beyond this chapter, so hold the simple rule until you have a defended reason not to.
One trap sits above all others. Data leakage is information reaching your model, or your choice of model, that would not be available at the real forecast moment (Kaufman et al. 2011). Example: the toxin concentration measured during the bloom. A leaked feature inflates your score on today’s data and collapses on tomorrow’s, because it will not exist when you truly need the forecast. Choosing your model on the final holdout is the same failure wearing different clothes, since that score was supposed to stand in for data you have never seen.
Time carries its own version of this trap. When your target is next week, your exam weeks should come after the weeks you learned from. Splitting weeks at random hands the model August while asking it to forecast July, which answers how well it fills in scattered past weeks rather than how well it forecasts the next one. Keep the blocks in calendar order, and let the latest block be the one you lock. That is the default that mimics how the forecast will actually be used; specialists relax it in narrow, argued cases, and none of them starts by shuffling the calendar.
17.3 A worked example
Suppose you want to warn swimmers a week ahead. Your target is bloom_next_week, one if the lake exceeds the advisory threshold and zero otherwise. Across three summers of weekly samples, 85% of weeks are clear, so your baseline, “always say no bloom,” scores 0.85 with no model at all.
You keep features genuinely known a week early: water temperature, recent rainfall, upstream phosphorus load. The three summers give you your three roles without any shuffling (Hastie et al. 2009). Summer one trains your candidate models. Summer two chooses between them. Summer three stays locked until the choice is made. You fit a logistic model (a standard yes-or-no classifier) and compare a few feature lists on summer two. Then you open summer three once, for the winner alone. Scored on the same metric as the baseline, accuracy, the model lands around 0.88 against the baseline’s 0.85 (these scores are constructed for the example). That three-point edge feels small, and its smallness is the honest result: a modest true win beats a dramatic fake one. You still report recall beside it, since accuracy alone hides the missed blooms.
Then a well-meaning collaborator adds chlorophyll_a, the pigment reading. The score leaps to 0.97. Exciting, until you ask when chlorophyll is measured: during the very bloom you are forecasting. It is the outcome wearing a different label. Drop it, and the score falls back to 0.88. The timing test, not the accuracy, decides. And because you trained on one shallow lake, you name the boundary out loud: on a deep, cold reservoir the pattern may not hold, so the forecast does not yet generalize there.
The block below builds the three summers, keeps the last one locked, and scores the honest model and the leaking one on the same rows. The scores are constructed for the example; the ordering is the lesson.
import numpy as np, pandas as pd
SEED = 464
rng = np.random.default_rng(SEED)
# 120 weekly samples across three summers, 40 weeks each, six bloom weeks per
# summer (15%). The calendar order is the design: it decides the three roles.
summer = np.repeat([1, 2, 3], 40)
bloom = np.zeros(120, dtype=bool)
for s in (1, 2, 3):
weeks = np.where(summer == s)[0]
bloom[rng.choice(weeks, 6, replace=False)] = True
temp = 22 + 3.0 * bloom + rng.normal(0, 2.0, size=120) # known a week early
rain = 20 + 4.0 * bloom + rng.normal(0, 6.0, size=120) # known a week early
chl = 5 + 30 * bloom + rng.normal(0, 8.0, size=120) # measured DURING it
train, locked = summer == 1, summer == 3
z = lambda v: (v - v[train].mean()) / v[train].std()
acc = lambda pred: float((pred == bloom[locked]).mean())
baseline = acc(np.zeros(locked.sum(), dtype=bool)) # always "no bloom"
score = 0.55 * z(temp) + 0.35 * z(rain)
honest = acc(score[locked] > np.quantile(score[train], 0.85))
leaky = acc(chl[locked] > np.quantile(chl[train], 0.85))
print(f"baseline, always say 'no bloom' : {baseline:.2f}")
print(f"honest features, locked summer : {honest:.2f}")
print(f"with chlorophyll_a added : {leaky:.2f}")
print("\nthe third number is not skill. chlorophyll is measured during the")
print("bloom you claim to forecast, so it cannot exist at prediction time")17.4 A seeded simulation
The chapter claims two things at once, and the code below shows both. A model can get better at data it has already seen while getting worse at cases it has never seen. And the score that crowned your winner flatters it on average. The seed makes the run reproducible: the same code always draws the same worlds.
The first half builds one world (a sine curve plus noise) and runs the protocol you just learned: twelve polynomials fit on training, compared on selection, and the winner alone scored once on a final holdout. The second half measures model-selection bias the way the definition demands, as an average over repetitions. In each of 500 fresh worlds it asks: how does the winner’s true error compare with the selection score that crowned it? Nobody can measure a model’s true error exactly, so the code stands in for it with ten thousand fresh points, which is close enough to trust and far more than any real study would have. Each half restarts the seed, so you can run either one on its own and get these same numbers.
import numpy as np
SEED = 464
degrees = np.arange(1, 13)
def world(rng, n):
x = rng.uniform(-3, 3, n)
return x, np.sin(1.5 * x) + rng.normal(0, .35, n)
def rmse(coefs, data):
x, y = data
return np.sqrt(np.mean((np.polyval(coefs, x) - y) ** 2))
# Left panel: one world, the three roles.
rng = np.random.default_rng(SEED)
training, selection, final = (world(rng, 40) for _ in range(3))
fits = [np.polyfit(training[0], training[1], d) for d in degrees]
sel_err = np.array([rmse(c, selection) for c in fits])
chosen = int(np.argmin(sel_err)) # chosen on SELECTION, never final
print("chosen degree:", degrees[chosen])
print("selection score that chose it:", round(float(sel_err[chosen]), 3))
print("its one final-holdout score: ", round(rmse(fits[chosen], final), 3))
# Right panel: optimism in expectation. Restart the seed so this study
# reproduces on its own.
rng = np.random.default_rng(SEED)
gaps = []
for _ in range(500):
tr, sel, big = world(rng, 40), world(rng, 40), world(rng, 10_000)
f = [np.polyfit(tr[0], tr[1], d) for d in degrees]
se = np.array([rmse(c, sel) for c in f])
pick = int(np.argmin(se))
gaps.append(rmse(f[pick], big) - se[pick]) # true error minus its crown
gaps = np.asarray(gaps)
print("mean optimism:", round(float(gaps.mean()), 3), "RMSE")
print("median:", round(float(np.median(gaps)), 3))
print("winner truly worse than its crowning score in",
round(100 * float((gaps > 0).mean()), 1), "% of worlds")
print("largest single gap:", round(float(gaps.max()), 2))
Read the left panel first. The blue training error only falls: every extra degree of flexibility lets the model bend closer to points it has already seen. The orange selection error falls while the model is learning signal, bottoms out near degree six, and climbs as the model starts memorizing noise. Nothing in the blue line alone would ever tell you to stop at six. Only data the fit never touched can say that.
Now the right panel, and read it as an average, because that is what the bias is. Across 500 worlds the winner’s true error exceeds the score that crowned it by 0.025 RMSE on average (median 0.015), and the winner comes out worse than its crowning score in 64 percent of worlds. In the other third, luck ran the other way, which is why no single split can show you the bias: it is a lean in the pile, not a property of each run.
Then look at the arrow. In 4 of the 500 worlds the gap blew past +0.3, topping out at +1.86, because a wildly flexible fit happened to win selection and then swung hard outside its data. Those worlds are not noise to trim away. They are the procedure’s real risk showing itself, and any average you report has to own them. The honest summary names all three things: the lean (the mean), the typical case (the median), and the tail (how bad, how often). In the companion notebook, rerun the simulation at several noise levels (the .35) and compare the mean, the median, the frequency, and the maximum separately. The average optimism grows with noise; a single seeded extreme, like this run’s +1.86, need not move the same way, because one world’s worst luck is not a summary of the procedure.
17.5 An AI failure case
You paste your feature list and ask an AI partner which features are safe to use. It answers with confidence: “chlorophyll_a is your strongest predictor, keep it.” The reasoning sounds airtight, because chlorophyll correlates almost perfectly with blooms. That near-perfect correlation is exactly the tell. This is the plausible-but-wrong-method failure: the tool ranked the feature on how well it fits the past, not on whether it could exist in time for a forecast. You catch it with one question statistics alone cannot answer. When is chlorophyll measured? During the bloom, so no forecast made a week earlier could have it. Drop it, watch the score fall back to 0.88, and keep the honest model.
17.6 It is your turn
You are working inside Studio 5: Develop the pathway. This lesson serves the prediction pathway. If your declared pathway is different, skim it and work the lesson that matches; the studio page routes you.
Your design is declared and diagnosed. This step decides whether your project is really forecasting cases nobody has seen yet, and if it is, puts that forecast under contract before you fit a single model.
The hands-on half of this section lives in the chapter’s companion notebook: open it in Colab with the badge at the top, and work the steps there.
Commit your own answer first, then delegate. Paste real output, never a number you remember. Expect to run each prompt more than once: read the answer, name the part you do not believe, put that objection back into the prompt, run it again. Tools that run that loop by themselves, fitting and refitting a model across several turns, make the leakage question below more urgent rather than less. Nobody in the loop but you knows when your data were recorded.
Four calls stay yours. The target: what you forecast and why it matters. The baseline: the honest rule the model must beat. The timing of every feature: whether a value would truly exist at the forecast moment, which only you, who knows your data’s timeline, can settle. And the verdict on whether your project should predict at all. A tool that has never seen your timeline cannot make these calls for you.
Ask the blunt version of your question: does your project need to guess an outcome for a case whose outcome does not exist yet? Write yes or no, and the one sentence that justifies your answer.
If yes, sign the four-part contract, in order. Name the target column, the dumbest honest baseline it has to beat, the split, and the metric matched to how rare your outcome is. Give the split all three of its roles: which data fit your candidates, which data choose between them, and which data stay locked for the single final score. If your target is about a later time, order those three blocks by the calendar and lock the latest one.
When you are ready to delegate this step:
Act as a Python tutor. Here is my forecasting workflow: [paste the cells that split, fit, compare, and score]. Audit the three data roles one by one. Which rows fit the candidates? Which rows chose between them? Which rows produced the final score, and had any model already seen them, even indirectly through my choices? If my target is about a later time, check whether every training row is earlier than the rows it is scored on. Then give me one independent way to confirm the final rows never touched training or selection.After running, verify: read the baseline, model, and margin off your own printout and confirm the tool’s numbers match, not ones it guessed. If the audit finds a role missing, that is the finding; do not argue the score. Counters confident fabrication, a fluent walkthrough quoting a margin your code never produced.
List your features and write beside each one the moment its value is settled. Circle anything settled at or after the outcome. That circled set is your leakage list, and it belongs in your write-up whether or not you drop the features on it.
When you are ready to delegate this step:
Here is the outcome I want to predict and the moment I need the forecast: a bloom next week, decided by next week's samples. Here is my candidate feature list with when each value is recorded: [list]. For each feature, say whether its value is settled before, at, or after the outcome, and flag any that could not exist at prediction time.After running, verify: re-derive the timing of the one feature it clears most confidently. If any feature is settled at or after the bloom, reject it whatever the tool says. Counters illusion of completeness, a tidy list omitting the one late-settled feature that dooms the forecast.
Write the boundary in one sentence: the cases your forecast covers, and the ones a reader should not assume it covers. Be specific about what makes them different.
When you are ready to delegate this step:
Act as a hostile reviewer. Attack this headline: "Our model forecasts blooms with 88% held-out accuracy." Name every reason a skeptic would distrust it, including the baseline it beat, the metric, and any hidden leak.After running, verify: check each objection against your printout and keep the ones your numbers support. Counters sycophantic agreement, where an assistant primed to help praises a headline you wrote.
If prediction is not your question, write the line that says so plainly, then run step 3 anyway. A variable settled after the outcome quietly sinks descriptive and causal work too.
Log the step in your AI Research Ledger, and verify at least one output with a named method from the Verification Guide. For a fitted model the method is a locked final holdout: never report a score computed on rows the model trained on, and never report the score that chose the model as though it were the final one. Write down how many candidates you compared, since a reader cannot judge selection optimism without that number. An AI reviewer may run the check with you; the decision to accept or reject stays yours.