14 Observational Descriptive Research
This chapter is part of a book in active development and has not yet been through the author’s review. Content may change as the review advances.
The research decision. Which group your observed data can honestly speak for, and where exactly the line falls past which a description of your sample is no longer allowed to become a claim about a wider population. You draw that line, and you write down the sentence on the far side of it that you refuse to say.
14.1 Why this decision matters
The decision on the table: which group your data can honestly speak for.
Picture a policy stakeholder reading your report before deciding whether it applies to their city. Their first question is never about your math. It is: “Which group did your procedure actually reach?” A result that quietly describes one group while claiming to speak for another is the most common failure in observational work: an estimate that is precise, well-formatted, and about the wrong group. This chapter equips you to name the group your data reach, and to refuse the sentence that swaps it for a group your data never touched.
14.2 The concept
This chapter follows the observational-descriptive pathway of the design library in RDSS (Blair et al. 2023). Observational descriptive research means you sample and measure without assigning anyone a condition, then summarize what you found. Example: you pull the roll call records of every member of a legislature and report how often each one crossed party lines. Because its inquiry and its answer strategy are descriptive, this pathway estimates distributions and relationships, and with care generalizes them to a population. Those estimates do not by themselves identify a causal effect. The next chapter takes the other road: observational data again, no assignment again, but a causal inquiry that has to carry an identification argument to earn its answer.
Four groups keep such a description honest (Groves et al. 2009). They are often drawn as boxes inside boxes, and that picture is worth knowing along with the place it lies.
- Target population: everyone your question is about. Example: every customer a streaming service will ever bill. This is the group you wish you could study.
- Accessible population: the part of the target you could reach in principle, given time and cost. Example: the customers you can actually contact with a survey this quarter.
- Sampling frame: the concrete list you actually draw from. Example: the current subscriber list your survey tool pulls from. This is the box that misbehaves, in three ways worth naming. Undercoverage is people who belong in your target and never make the list. Overcoverage is records on the list for people outside your target altogether. A duplicate is one person listed twice, which quietly gives them two chances of being picked. Treating the frame as a clean slice of the population above it is one of the quietest ways a description goes wrong.
- Sample: the customers you actually survey, drawn from the frame and no wider.
A claim is honest only when the box it describes matches the box it names. Two processes decide how the boxes differ. Selection is any process that decides who lands in your data instead of chance alone. Example: a survey taken outside one dining hall selects for people who eat there at lunch. Nonresponse is the units you drew who never gave you data. Example: of 500 people contacted, 300 reply, and the 200 silent ones are rarely a random slice. When either process relates to the very thing you are measuring, your summary drifts from the truth, and no amount of extra data pulls it back (Bethlehem 2010). Size shrinks the sample-to-sample wobble. It does nothing to a systematic tilt in who you reached (Meng 2018).
14.3 A worked example
A subscription streaming service measures its billing-error rate, meaning the fraction of customers whose monthly charge comes out wrong. Example: 2 wrong charges in 100 customers is a 2% billing-error rate. The analytics team pulls the current subscriber list, emails a short survey to a random draw from that list, and counts how many report a wrong charge in the past year. The survey reports 2%, and the headline writes itself: “the service has a 2% billing-error rate.”
Now walk the boxes. The target population is every customer the service bills (Groves et al. 2009). The sampling frame is the current subscriber list, because only people on that list receive the survey. Here is the trap: the customers hit hardest by billing errors, the ones charged twice or charged after canceling, are also the ones most likely to have quit in frustration. They removed themselves from the frame before anyone surveyed them. This is survivorship selection, a filter that keeps exactly the people least likely to show the problem you are counting. Current subscribers are the contented end of the customer base, not a fair slice of it.
So the 2% is honest about one box and dishonest about another. The licensed claim is: “among customers still subscribed, the billing-error rate is 2%.” The silent upgrade, the sentence to refuse, is: “the service’s billing-error rate is 2%.” The true rate across everyone billed is almost certainly higher, because the worst-hit customers were gone before the survey went out. That gap between the box you measured and the box you named is the subject of this chapter.
The block below builds the whole customer base, lets the worst-hit customers quit, and then reports what a perfect survey of the remaining list would find. Nothing here is a sampling error.
import numpy as np, pandas as pd
SEED = 464
rng = np.random.default_rng(SEED)
n = 50_000
# Everyone the service bills. Wrong charges are what we want to count.
wrong_charge = rng.random(n) < 0.060
# Customers hit by a wrong charge quit far more often — and quitters leave the
# subscriber list, which is the frame the survey draws from.
quit = rng.random(n) < np.where(wrong_charge, 0.70, 0.09)
target_rate = wrong_charge.mean()
frame_rate = wrong_charge[~quit].mean()
print(f"target population (everyone billed) : {target_rate*100:.1f}%")
print(f"sampling frame (current subscribers) : {frame_rate*100:.1f}%")
print(f"customers who left the frame : {quit.mean()*100:.0f}%")
print("\nthe survey can only ever see the second number. no sample size,")
print("and no random draw from that list, closes the gap to the first")14.4 A seeded simulation
The worked example argues that who you reach tilts what you measure, and that a bigger sample does not fix the tilt. You can watch that happen. A seed is a fixed starting point for a random-number generator: the same seed always reproduces the same draws, so the figure below is exactly what you will get when you run this code in the companion notebook. The code builds a population of 100,000 people whose true mean age is known by construction, draws ten honest random samples of 500, then draws one convenience sample of the same size through a channel that mostly reaches younger people.
import numpy as np
import matplotlib.pyplot as plt
SEED = 464
rng = np.random.default_rng(SEED)
population = np.clip(rng.normal(49, 17, size=100_000), 18, 90)
truth = population.mean()
random_means = [rng.choice(population, 500, replace=False).mean()
for _ in range(10)]
weights = np.exp(-(population - 25) ** 2 / (2 * 12 ** 2)) # younger = likelier
convenience = rng.choice(population, 500, replace=False,
p=weights / weights.sum())
fig, ax = plt.subplots(figsize=(7.6, 2.9))
ax.scatter(random_means, np.ones(10), s=60, color="#2a78d6", zorder=3)
ax.scatter([convenience.mean()], [0], s=60, color="#eb6834", zorder=3)
ax.axvline(truth, color="#333333", ls="--", lw=1.2)
ax.text(truth + .5, 1.55, f"true mean = {truth:.1f}", color="#333333",
fontsize=9)
ax.set_yticks([0, 1], ["Convenience\nchannel (n = 500)",
"Random samples\n(n = 500)"], fontsize=9)
ax.set_xlabel("Sample mean age (years)")
ax.set_ylim(-.7, 1.9)
plt.show()
Read it the way a reviewer would. The ten random samples never agree exactly: their means spread between about 47 and 51. But they huddle around the truth, and that spread is honest, quantifiable uncertainty. The convenience channel has exactly the same sample size, and it misses the true mean by about sixteen years. Nothing about the dot itself warns you; it looks as confident as any other. Only the sampling procedure tells you which kind of dot you have. In the companion notebook, move the channel’s center (the 25 inside weights) and rerun: the tilt follows the channel, never the sample size.
14.5 An AI failure case
You ask a chatbot: “My survey of current subscribers shows a 2% billing-error rate on a large panel. Can I report the service’s billing-error rate as 2%?” It answers with full confidence: “Yes. With a sample that large, your estimate is precise and reliable.” This is wrong in two named ways at once. It commits a silent scope change, quietly upgrading “current subscribers” to “the service’s customers,” and it leans on the fallacy that a big sample cures a tilted one.
You catch it because you wrote your own answer first, and yours named the frame: the customers still subscribed, not everyone the service bills, so the swap is visible on sight. Then you check the mechanism against the data. The customers with the worst billing errors canceled before the survey went out, so missingness relates directly to the outcome you count. A larger panel of current subscribers only estimates the current subscribers’ error rate more precisely. It never reaches the people who left.
14.6 It is your turn
You are working inside Studio 5: Develop the pathway. This lesson serves the observational-descriptive pathway. If your declared pathway is different, skim it and work the lesson that matches; the studio page routes you.
You arrive with a declared and diagnosed design. This chapter is the first of the pathway chapters, so your job now is either to build your design out on this pathway or to use it to stress-test the pathway you chose.
The hands-on half of this section lives in the chapter’s companion notebook: open it in Colab with the badge at the top, and work the steps there.
Commit your own answer first, then delegate. Each prompt below leaves the judgment with you and hands the AI only the legwork. Work them as a loop rather than a one-shot: the first list of dropout groups is always the obvious one, and the useful answer usually arrives on the second or third pass, after you have told the tool which group it missed and asked it to try again with that in mind.
Three decisions stay yours alone. First, which population your project is really about: the AI does not know your question’s intent. Second, which frame you can honestly reach, and therefore the coverage gap you must disclose. Third, the boundary line itself: the exact sentence where your description stops and the population claim you refuse to make. You may ask an AI to attack that line, never to draw it for you.
Decide which case you are in. Read your inquiry’s words. If it asks how much, how many, how often, over a group you will observe rather than assign, this is your pathway. If it asks why or what would happen if, it is not, and you run the drill in step 5 instead.
If this is your pathway, write your four boxes, one line each: target population, accessible population, sampling frame, sample. Be concrete about the frame. Name the actual list, database, or feed your data will come from.
Draw your selection diagram: an arrow from each box to the next, with the filter that removes people written on the arrow. Circle the one filter you most fear bends both who you reach and what you measure. That is the one you will have to disclose.
List-to-verify. Turn a vague worry into a checkable list of who drops out.
I am drawing a selection diagram for an observational study. My target population is [X], my frame is [Y], my sample is [Z]. List each group that drops out between target and sample, in a table, with whether their absence pushes my estimate up or down.After running, verify by mapping each named group to one real step in your own diagram, and checking that the one group you feared most is on the list. Counters illusion of completeness: a long, tidy list that quietly omits the group that matters most.
Write the two sentences that define your claim. The first is the population claim your frame actually licenses, stated with its uncertainty. The second is the silent upgrade your design forbids, written out in full so you can recognize and refuse it when a coauthor or an AI hands it back to you.
Red-team. Make the AI attack your boundary sentence instead of praising it.
Here is a boundary sentence I wrote: "[your sentence]". Act as a hostile peer reviewer. Name every place it over-reaches, under-states the bias, or hides the direction of the drift. Do not rewrite it for me.After running, verify by checking each objection against your own numbers and pushing back if every objection is mild. Counters sycophantic agreement: an AI that calls your caveat “well-balanced” has reviewed your ego, not your evidence.
If this is not your pathway, run the classification drill instead. Write your question as if it were an observational descriptive question, name the four boxes it would need, and say precisely what that version could not tell you. The sentence you cannot write here is the reason your chosen pathway has to earn its extra assumptions.
Log the work in your AI Research Ledger, and verify the key claim with a named method from the Verification Guide; simulation fits here, because the claim is about how a sampling procedure behaves rather than about a fixed number. An AI reviewer may attack your boundary sentence with you; the decision to hold the line or move it stays yours.
Locate. Find real methodological backing instead of trusting a paragraph.
Act as a research-methods assistant. List real, retrievable references on survivorship and nonresponse bias in observational studies. For each, give title, authors, year, and venue, and mark any you are not confident exist.After running, verify by opening each source in a library database before you cite it. Counters confident fabrication: a fluent citation for a paper that does not exist looks identical to a real one until you try to retrieve it.