11 Uncertainty Before You Need It
This chapter is part of a book in active development and has not yet been through the author’s review. Content may change as the review advances.
The research decision. What quantity you are actually after, what recipe you will use to guess it, and what you will say about how much that guess could have moved. You decide all three before you inspect the outcomes, because each one changes what the others are allowed to mean.
The decision on the table: the quantity, the recipe, and the honest statement of how much the recipe wobbles.
“Your number is not wrong. It is just alone. Show me what else it could have been, and I will know how much to trust it.” — a journal editor, on the first revision of a paper that reported one estimate and no uncertainty
11.1 Why this decision matters
Every result you will ever produce comes from a procedure that could have come out differently. Different households answered. Different weeks got sampled. A coin came up heads for this village and tails for that one. The number you hold is one draw from a process, and the honest question is not “is my number right?” but “how much does a number from this procedure move around?”
That question has an answer you can see. Not guess, not intuit — see, by running the same procedure again and again and looking at the pile of answers it produces. That pile is the whole subject of this chapter. Everything else here is vocabulary for talking about its shape.
You need this before the next chapter, not after. The next chapter asks you to diagnose a design by its bias, its power, and its coverage, and every one of those words is a statement about a pile of repeated answers. Meet the pile first and those words stop being jargon.
A question that often comes up here: “I only get to run my study once. What good is imagining a thousand runs?” You run it once. The uncertainty statement is not about your run, it is about your procedure, and a procedure can be studied by repetition even when your study cannot be repeated. That is exactly why you can report uncertainty from a single sample.
11.2 The concept
Three words get confused constantly, and separating them fixes most of the confusion in this whole area.
An estimand is the quantity you want, defined in the world before you touch the outcome data (Morris et al. 2019). Example: the average monthly rent paid by all 4,000 households in one town, in one month. Notice that it is a number about the world, not about your data. It would have a value even if you never ran a survey.
An estimator is the recipe you apply to data to guess the estimand. Example: draw 60 households at random and take their average rent. The recipe is a procedure, so it is the thing that can be repeated.
An estimate is what one run of the recipe returns. Example: 1,621. One number, from one run, on one day.
The estimand does not move. The estimate does. The distance between those two facts is where uncertainty lives.
One bridge back to the design vocabulary. In RDSS the inquiry is the question your design asks, and the estimand is the value that question has in the world (Blair et al. 2023). This chapter pairs that distinction with the standard estimator-and-estimate vocabulary, which is common statistical usage rather than anyone’s coinage (Morris et al. 2019).
11.2.1 The shape of chance
Run the recipe again on the same town and you get a different estimate. Run it 2,000 times and you get 2,000 estimates, and they form a shape. That shape has a name.
The sampling distribution is the pile of estimates you would get if you repeated your whole procedure many times in the same world. It is a property of the procedure, not of any one run.
Here is that pile, built from a town whose true average rent is 1,568.
import numpy as np
SEED = 464
rng = np.random.default_rng(SEED)
town = rng.lognormal(mean=7.3, sigma=0.35, size=4000) # every household's rent
estimand = town.mean() # the quantity we want
# the recipe: survey 60 households at random, take the mean
means = np.array([rng.choice(town, size=60, replace=False).mean()
for _ in range(2000)])
print(f"estimand (true average rent): {estimand:,.0f}")
print(f"one survey said: {means[0]:,.0f}; another said: {means[1]:,.0f}")
print(f"centre of the pile: {means.mean():,.0f}")
print(f"spread of the pile: {means.std(ddof=1):,.0f}")
print(f"middle 95% of the pile: {np.percentile(means, 2.5):,.0f}"
f" to {np.percentile(means, 97.5):,.0f}")The true average is 1,568. The first survey said 1,621. The second said 1,655. Neither is a mistake. Both are what this recipe does.
The pile centres on 1,569, almost exactly the truth, which tells you the recipe aims true. The pile spreads by about 72, which tells you how far a single survey typically lands from the centre. And the middle 95% of the pile runs from 1,431 to 1,713, which tells you the range this procedure produces almost all of the time.
Read those three facts again, because they are the three things uncertainty language exists to say. Where the pile sits. How wide it is. What range covers nearly all of it.
11.2.2 Standard error and interval
The spread of that pile has a name. The standard error is the typical distance between one run’s estimate and the centre of the pile. For our survey it is about 72.
Notice what it does not tell you. It does not say how wrong today’s estimate is, and a tightly packed pile can still sit centred on the wrong quantity. Width and aim are two different facts, and the next chapter gives the second one its own name.
You cannot see the pile from one survey, but you can estimate its width from one survey, and then build a range around your single estimate. That range has a name too. A confidence interval is a range built by a recipe that catches the estimand a stated share of the time, when the same recipe is repeated.
Read that definition slowly, because the obvious paraphrase of it is false. The interval you actually computed either contains the truth or it does not (Greenland et al. 2016). There is no chance left in it. What has a 95% property is the recipe, and only while its assumptions hold, not your one range.
That distinction sounds pedantic until you check it, which you can do here, because this town is one we built. In an example where the population is known, simulation lets you watch the pile directly. In a real study you never see it: the pile is implied by your sampling plan and your stated assumptions. Simulation checks what those assumptions imply. It does not prove the world obeys them.
rng = np.random.default_rng(SEED)
n, runs, hits = 60, 2000, 0
for _ in range(runs):
s = rng.choice(town, size=n, replace=False)
se = s.std(ddof=1) / np.sqrt(n) # width estimated from ONE survey
lo, hi = s.mean() - 1.96 * se, s.mean() + 1.96 * se
hits += lo <= estimand <= hi
print(f"share of intervals that caught the truth: {hits / runs:.1%}")It catches the truth 94.8% of the time. The recipe does what it promises. Any one interval, though, either caught 1,568 or missed it, and you would never know which.
So the honest sentence about your own study is: “my estimate is 1,621, from a procedure whose 95% intervals catch the true average about 95 times in 100.” The dishonest one, the one you will hear constantly, is “there is a 95% chance the true average is between 1,470 and 1,771.” The first describes a procedure. The second makes a probability claim about a fixed number, and fixed numbers do not have chances.
11.3 A worked example
A business-school team wants the average monthly rent paid by renting households in a mid-sized town this month, to price a housing product. They cannot survey everyone, so they plan to survey 60 households and report an interval.
Their estimand: the average monthly rent across all renting households in the town this month. Their estimator: the mean of 60 households drawn at random from the town’s list of rented dwellings. Their uncertainty statement: a 95% interval built from the spread inside their own sample.
Then the field team makes a reasonable-sounding change. Knocking on 60 doors scattered across town is expensive, so they draw 12 apartment buildings at random from that same list and survey 5 rented flats in each. Still 60 households from the same population, much cheaper. Nothing about the estimand changed. Nothing about the arithmetic changed.
Everything about the uncertainty changed.
11.3.1 Dependence, and what it costs
Flats in one building share a landlord, a neighbourhood, a heating system, a rent schedule. When you learn one flat’s rent, you have already learned a lot about the other four. Dependence means that learning one observation changes what you should expect about another. Here the flats in a building tend to move together, so another flat in the same building adds less than a fresh independent flat would.
The arithmetic does not notice. The procedure’s honesty does.
rng = np.random.default_rng(SEED)
B, K, truth, runs = 12, 5, 2000.0, 2000 # 12 buildings, 5 flats each
naive_hits = cluster_hits = 0
for _ in range(runs):
building_effect = rng.normal(0, 260, B) # landlord/neighbourhood effect
data = np.array([truth + building_effect[b] + rng.normal(0, 120, K)
for b in range(B)])
flats = data.ravel() # pretend: 60 independent flats
se_naive = flats.std(ddof=1) / np.sqrt(60)
naive_hits += (flats.mean() - 1.96 * se_naive <= truth
<= flats.mean() + 1.96 * se_naive)
b_means = data.mean(axis=1) # honest: 12 independent buildings
se_cluster = b_means.std(ddof=1) / np.sqrt(B)
cluster_hits += (b_means.mean() - 1.96 * se_cluster <= truth
<= b_means.mean() + 1.96 * se_cluster)
print(f"treating 60 flats as independent: {naive_hits / runs:.1%} caught the truth")
print(f"treating 12 buildings as the unit: {cluster_hits / runs:.1%} caught the truth")The interval that treats 60 flats as 60 independent observations catches the truth 63.2% of the time (Donner and Klar 2000). It promised 95%. It is not a little optimistic, it is wrong about one case in three, and nothing in the output would have told the team.
Switching to buildings as the unit brings it to 91.7%. Better by a lot, and still shy of 95, which is worth being honest about rather than rounding away. With only 12 buildings, the recipe’s usual multiplier of 1.96 is too narrow. Widening it to the value appropriate for 12 groups gets 94.5%. Surveying 40 buildings instead, at 5 flats each, gets 94.2% with the ordinary multiplier.
That is the practical lesson, and it is not about arithmetic. In this design, sixty flats in 12 buildings carry roughly the information of 14 independent flats. That number comes from these data, not from a rule: it is not “divide by the group size”, and a different dependence structure gives a different answer.
The direction is not automatic either. Here the flats in a building resemble each other, which widens the pile, and the honest fix is a wider interval. A design that deliberately balances within each group can move the pile the other way and produce a narrower one. So “my observations are dependent” does not tell you which way to correct. The direction depends on the sampling design, the estimator, and the pattern of relationships among your observations.
With the variability this example assumes, extra buildings buy far more precision than extra flats inside a building you have already entered. Doubling the flats per building from five to ten moves the spread from about 77 to about 76. Doubling the buildings instead moves it to about 54. Extra flats are not worthless, they just run into sharply diminishing returns, because the part of the wobble that comes from which buildings you picked is untouched by measuring more flats inside them.
A question that often comes up here: “How would I know my data have this problem?” Ask how the data were collected, not what they look like. If units were selected in groups, measured repeatedly, or connected to each other, the dependence is in the design, and the design is something you know before you see a number.
11.3.2 Check yourself
A survey reports an approximate 95% interval for average rent, from 1,470 to 1,771. Which reading is warranted?
A. There is a 95% probability that the true average rent lies between 1,470 and 1,771.
B. About 95% of households in the town pay between 1,470 and 1,771.
C. Under its assumptions, this interval procedure catches the true average in about 95 of every 100 repeated surveys.
D. About 95% of future survey estimates will land inside this particular range.
The answer is C. Option A is the misreading this chapter exists to prevent: it hands the procedure’s repetition rate to one finished interval. Option B confuses uncertainty about a town-wide average with the spread among households, which is far wider. Option D swaps a moving sequence of intervals for one fixed range.
If you want to write the answer out fully: this interval either contains the true average or it does not, and the 95% describes what the recipe does across repeats.
11.4 An AI failure case
Ask an assistant to interpret an interval and you will very often get this, fluently and confidently:
“The 95% confidence interval is 1,470 to 1,771, meaning there is a 95% probability that the true average rent falls within this range.”
The number is fine. The sentence is not. The true average is a fixed quantity, and your computed range is now fixed too, so no probability is left to assign between them. What is 95% is the long-run catching rate of the recipe that produced the range.
This one matters more than it looks. A researcher who believes the wrong version will also believe that a range which “just misses” a value is weak evidence against it, and will start reading single intervals as verdicts. The verification move is small: ask what would be repeated, and if nothing in the sentence could be repeated, the sentence is about the wrong thing.
11.5 It is your turn
You are working inside Studio 4: Declare and diagnose provisionally. Keep what you write here; the studio’s milestone chapter is where it joins the other lessons’ pieces into one artifact you can defend.
Write your project’s uncertainty foundation. This is the piece the next chapter builds its diagnosis on, so make it specific to your own project rather than generic.
The hands-on half of this section lives in the chapter’s companion notebook: open it in Colab with the badge at the top, and work the steps there.
- The estimand. What you are trying to learn is a research decision, not a modelling detail. Nobody can hand it to you.
- The claim your interval makes. You are responsible for the sentence that goes in the paper, including refusing the flattering version of it.
- The independent unit. Only you know how your data were really collected.
State your estimand in one sentence, as a quantity in the world with a population, a setting, and a time. It must be something that would have a value if you never collected data.
State your estimator as a recipe: what you will collect, and exactly what you will compute from it. Someone else should be able to follow it without asking you a question.
💡 AI Prompt: “Here is my research question and how I plan to collect data: [paste yours]. State my estimand in one sentence, as a quantity in the world that would have a value even if I never collected data. Then state my estimator as a recipe someone else could follow. Do not give me any numbers or advice yet.”
After running, verify:Name what would differ on a repeat. Write the two or three things that would come out differently if selection and measurement ran again for the same target population, setting, and time. This describes your sampling distribution in words, before you can see it. Keep the period fixed: a repeat next month would also change the quantity you are chasing, which is a different kind of movement.
Name your dependence structure. List the levels at which your observations were sampled, repeated, or connected, and how many units you have at each level. Then say which units your uncertainty calculation will treat as independent, and the assumption that lets you treat them that way.
💡 AI Prompt: “Here is how my units were selected and measured: [paste yours]. Identify the levels at which my observations were sampled, repeated, or connected. For each level, say what dependence is plausible and what more you would need to know before choosing how to compute uncertainty. Do not infer a single effective sample size from my description.”
After running, verify:Write your uncertainty sentence, and then write the wrong version of it. State what your interval will and will not claim. Then write the tempting false version beside it, and one line on how you would catch yourself saying it.
Log it. Add your AI Research Ledger rows for anything you delegated here, and record which decisions you kept.