15 Observational Causal Research
This chapter is part of a book in active development and has not yet been through the author’s review. Content may change as the review advances.
The research decision. Whether your observational comparison has earned the word because or must stop at associated with. You settle it by naming the one confounder the comparison turns on and writing the identification argument that closes the back door, or by admitting in writing that you have none.
15.1 Why this decision matters
The decision on the table: whether your comparison has earned the word because.
Picture a thesis advisor reading your first draft. You wrote, “the treated group did better, so the treatment worked.” Her reply is the sentence you will hear for the rest of your research life: “After is not because. Show me the world where they were not treated, or admit you are guessing.” Almost every question worth asking forbids the clean experiment. You cannot assign a person a disease, a country a war, or a wild animal a gut microbe. So the honest researcher’s real craft is deciding, out loud and in writing, when an observed comparison is trustworthy enough to carry a cause. Get it wrong and a paper claims more than its data support. Get it right and a modest, bounded claim becomes credible.
15.2 The concept
This chapter follows the observational-causal pathway of the design library in RDSS: the conditions you compare were set by the world rather than by you, so a causal reading has to be argued rather than assumed (Blair et al. 2023).
Start with the missing world.
- Counterfactual: the outcome that would have happened under the choice that was not made. Example: how long a coffee drinker would have lived if she had not drunk coffee. You never see it, because she did drink coffee.
A causal claim is a claim about a counterfactual. The trouble is a villain that hides inside almost every observational comparison.
- Confounder: a third factor that pushes on both who gets the treatment and the outcome. Example: a health-conscious lifestyle raises both the chance of a daily coffee habit and lifespan, so coffee drinkers look longer-lived even if coffee did nothing.
You can draw the trouble. A causal diagram is a picture of arrows where each arrow means “this directly influences that.” In it, the confounder creates a back door: a sneaky path linking treatment and outcome that runs through the confounder instead of through any real effect. A raw comparison sees both paths at once and blames the treatment for the confounder’s work.
Two moves close the back door without an experiment.
- Selection on observables: identifying an effect by adjusting for a set of measured confounders sufficient to block every back-door path. Example: compare coffee drinkers and non-drinkers within the same lifestyle group, because lifestyle is what pushes on both. It works only for confounders you can actually see, and the word sufficient is doing real work: the goal is the right set, not the longest one.
- Natural experiment: a situation where a chance-like force outside anyone’s control decided who got treated, making assignment as-if random (credibly like a coin flip). Example: a visa lottery decides who makes a trip, so winners and losers differ only by luck. Chapter 41 builds this move into full designs.
One warning before the first move tempts you into adjusting for everything in the file. Adjusting is not free, and adjusting for the wrong variable manufactures bias instead of removing it. A mediator is a variable sitting on the causal path itself: coffee may act on lifespan partly through blood pressure, so adjusting for blood pressure throws away part of the very effect you are estimating. A collider is a variable that treatment and outcome both push on: coffee habits and long life might each raise the chance of appearing in a “healthy aging” study, so adjusting for study membership creates a link between them that exists nowhere in the world. Draw the diagram first; adjust for common causes, and leave mediators and colliders alone (Pearl 2009).
Whichever move you use, you owe a reader an identification argument: your written reason that the comparison recovers a real effect. Every design in this family rests on a short list of load-bearing assumptions the data can never fully confirm, only make plausible (Hernán and Robins 2020). Selection on observables needs the adjustment set to be sufficient and the groups to overlap. A visa-lottery design needs the lottery to be genuinely random and to touch lifespans only through the trip. Name your design’s list; a single sentence per assumption is enough. When you cannot make that argument honestly, you have hit the causal-language boundary, the point past which your evidence stops earning because and must switch to associated with. The most common failure here has a name too. Design mimicry is borrowing the vocabulary (“natural experiment,” “difference-in-differences”) to decorate a comparison whose assumption was never argued. The word is not the argument.
15.3 A worked example
Bring this into a biology lab. You study wild-derived mice and notice a striking pattern: individuals whose gut carries a particular fiber-digesting bacterium are, on average, leaner than mice without it. The headline writes itself: “the microbe causes leanness.” Would you stake a thesis on it?
Draw three boxes first: microbe, leanness, and a third box for anything that could drive both. That third box fills in fast. Diet is a confounder. Mice that forage on high-fiber plants both harbor more of the fiber-digesting bacterium and stay leaner for reasons unrelated to the microbe. Diet points into both boxes, opening a back door, so a naive microbe-versus-no-microbe comparison mostly measures the diet gap and credits it to the bacterium. A colony of ten thousand mice would make that wrong number beautifully precise and no less wrong, because a larger sample shrinks noise, not confounding.
Now the fix. If you recorded each mouse’s diet, use selection on observables: compare carriers to non-carriers within the high-fiber group, then within the low-fiber group, and average the two honest within-diet differences. If the gap collapses toward zero, the microbe was mostly a passenger of diet. The catch is honest: this works only for the confounder you measured. If mice also differ in an unrecorded trait, say a temperament that drives both foraging and metabolism, that trait keeps the back door open and “we controlled for diet” is not identification. Then your defensible claim is that the microbe is associated with leanness, and the causal question waits for a real intervention, such as colonizing genetically identical, identically fed mice. That boundary is what this chapter teaches you to see and to say.
Which variables you must adjust for, and which adjustments open new paths rather than closing them, is decided by the diagram, not by what you happen to have measured (Pearl 2009).
The block below builds mice in which the microbe does nothing at all, and then measures the gap a naive comparison reports. Run it before you trust any carrier-versus-noncarrier number, including your own.
import numpy as np, pandas as pd
SEED = 464
rng = np.random.default_rng(SEED)
n = 10_000
# Diet drives BOTH the microbe and leanness. The microbe itself does nothing.
high_fiber = rng.random(n) < 0.5
microbe = rng.random(n) < np.where(high_fiber, 0.75, 0.25)
leanness = 20 - 3.0 * high_fiber + rng.normal(0, 1.5, size=n) # lower = leaner
df = pd.DataFrame({"high_fiber": high_fiber, "microbe": microbe,
"leanness": leanness})
naive = df[df.microbe].leanness.mean() - df[~df.microbe].leanness.mean()
within = (df.groupby("high_fiber")
.apply(lambda g: g[g.microbe].leanness.mean()
- g[~g.microbe].leanness.mean(), include_groups=False))
print(f"true microbe effect : {0.0:+.2f}")
print(f"naive carrier-vs-noncarrier gap : {naive:+.2f}")
print(f"within high-fiber mice : {within[True]:+.2f}")
print(f"within low-fiber mice : {within[False]:+.2f}")
print(f"average of the two within-diet gaps: {within.mean():+.2f}")
print("\nthe naive gap is the DIET gap wearing the microbe's name;")
print("ten thousand mice make it precise, not true")15.4 Process tracing: causal inference inside one case
A town council votes 6 to 3 to ban single-use plastic bags. A student campaign had spent the spring collecting signatures, and the campaign now says it won the ban. You want to know whether that is true for this one town. You have no comparison group and no lottery. You have one case and an archive.
Process tracing is a way of testing a causal claim inside a single case by asking what evidence each explanation predicts you will find, and then checking the record for it. Example: if the campaign moved the council, members must have heard from it before the vote, so any record of that contact should be dated before the vote, not after. RDSS places process tracing first in its observational-causal library and declares it as a design like any other, with a model, an inquiry, a data strategy, and an answer strategy you can diagnose (Blair et al. 2023). The inquiry is narrower than elsewhere in this chapter. It is a cause-of-effect question, one that asks whether a particular cause produced an outcome that already happened in a particular case. Example: did the campaign cause the ban in this town? That is a different question from how much campaigns change policy on average.
The work starts with rivals. A rival explanation is a different causal story that could have produced the same outcome. Example: the council may have copied a neighboring city’s ordinance, or acted early because a state bill was about to force the issue. Each rival does the job the diet did in the mouse study: it is the story your evidence must beat before because is earned. Write the rivals down before you open a single document, the same way you drew the causal diagram before you adjusted for anything.
Next, sort each clue by how it bears on the claim. The four standard tests come from Van Evera (Van Evera 1997). Collier’s guide lays them out by whether passing is necessary or sufficient for accepting the explanation (Collier 2011).
- A straw-in-the-wind test looks for a clue that fits your explanation but also turns up fairly often without it, so finding it helps a little and missing it hurts a little. Example: a council member later praises “the young activists” in a newspaper interview.
- A hoop test looks for a clue your explanation must produce to stay alive, though passing it settles little. Example: a dated record of the campaign reaching the council before the vote, such as a speaker in the public-comment minutes, a petition logged by the clerk, or an email to a member. If nobody from the campaign ever reached the council, the campaign story is in deep trouble.
- A smoking-gun test looks for a clue that rarely appears unless your explanation is true, so finding it counts for a lot, while missing it counts for little. Example: a staff memo dated before the vote that recommends the ban and names the campaign’s petition as the reason.
- A doubly decisive test looks for a clue that is both required by your explanation and rare without it. Real archives rarely hold one, so when you seem to have found it, check both of its probabilities again before you lean on it.
A hoop test has a trap. The explanation must produce the event, but you only ever search the record. A missing clue counts as a failed hoop only if the event would have been written down, the document survived, and you can get at it. Minutes often skip who spoke in public comment, and an email to a member may never reach a public file. Absence of evidence becomes evidence of absence only to the extent the record would have caught the event.
The tests have a precise core. Each clue carries two probabilities: how likely you are to find it if your explanation is true, and how likely if a rival is true. The ratio between them decides how far the clue should move your belief, and RDSS writes the tests as exactly that kind of updating (Blair et al. 2023). The block below starts from even odds and computes where your belief should land when each clue is found or missing. The probabilities are constructed for the example; in your project they are judgments you write down before you search.
import numpy as np, pandas as pd
SEED = 464
rng = np.random.default_rng(SEED)
# Your judgments, written BEFORE you open the archive:
# P(clue appears | campaign caused the ban), P(clue appears | a rival caused it)
tests = pd.DataFrame({
"test": ["straw-in-the-wind", "hoop", "smoking gun", "doubly decisive"],
"p_if_true": [0.60, 0.95, 0.30, 0.90],
"p_if_false": [0.40, 0.60, 0.03, 0.05],
})
prior = 0.50
def posterior(p_true, p_false, found):
lt, lf = (p_true, p_false) if found else (1 - p_true, 1 - p_false)
return prior * lt / (prior * lt + (1 - prior) * lf)
tests["if_found"] = [posterior(a, b, True) for a, b in zip(tests.p_if_true, tests.p_if_false)]
tests["if_absent"] = [posterior(a, b, False) for a, b in zip(tests.p_if_true, tests.p_if_false)]
print(tests.round(2).to_string(index=False))
# Check the smoking-gun row by simulation: 100,000 imagined towns, half where
# the campaign really caused the ban. Among towns where the clue turns up,
# how often was the campaign the cause?
n = 100_000
campaign_caused = rng.random(n) < prior
clue = rng.random(n) < np.where(campaign_caused, 0.30, 0.03)
print(f"\nsmoking gun found, simulated share true : {campaign_caused[clue].mean():.2f}")
# The same clue, if you judged it less unique (a rival leaves it 10% of the time)
print(f"smoking gun found, rival leaves it 10% : {posterior(0.30, 0.10, True):.2f}")
# A missing hoop clue: the record catches contact only if it was written down,
# kept, and open to you. Suppose that happens half the time under either story.
p_record = 0.50
print(f"hoop clue absent, record always catches it : {posterior(0.95, 0.60, False):.2f}")
print(f"hoop clue absent, record catches it half : "
f"{posterior(0.95 * p_record, 0.60 * p_record, False):.2f}")
print("\nthe posterior is only as good as the two probabilities you wrote first;")
print("the archive supplies the clue, you supply what it would mean")Read the table by rows. Passing the hoop test moves you only from 0.50 to 0.61, but failing it drops you to 0.11. Finding the memo lifts you to 0.91, while missing it leaves you at 0.42, barely below where you started. The straw in the wind nudges you to 0.60 or 0.40 and no further. The simulated towns agree with the formula at 0.91, which is a second route to the same number. The last line is the warning. If a rival story would leave that memo behind 10 percent of the time instead of 3, the same find supports only 0.75. The two hoop lines show why a silent archive is weak evidence. If the record would always have caught the contact, its absence drops you to 0.11. If the record catches it only half the time, the same absence leaves you at 0.43, because most of the silence is the archive’s, not the council’s. The honest uncertainty in process tracing lives in those judgments, so you report them next to your conclusion and show how it changes when a skeptic moves them.
Every clue also needs source assessment, which means asking who produced a document, when, and for what audience, before you let it count. Example: a memo written for internal use before the vote is better evidence than an interview given a year later, when everyone wants credit for a popular ban. Bennett and Checkel’s collection lays out standards for this work, including how to weigh sources that have reasons to shade the story (Bennett and Checkel 2015). Ask also which documents survived and who chose them. Minutes record formal business and leave out hallway conversations, and the campaign’s own files keep the moments that flatter it. An archive filtered that way can make a true story look silent and a false one look loud. A clue whose origin you cannot establish should move your belief very little.
Be exact about the warrant. Nothing here crosses by sampling or by assignment. The causal reading rests on four assumptions you state in writing: your list of rivals contains the true story, your starting odds and your two probabilities for each clue are defensible, your sources are what they claim to be, and clues you combine are separate witnesses. Two clues count as two only if each one’s chance of turning up does not depend on the other under either story. A memo and a later interview by the same staffer are closer to one witness than two. On the compass this is a causal question whose reach is the data at hand, meaning this one case. A strong conclusion about this town says nothing about how often campaigns win bans elsewhere. Calling a clue a “smoking gun” without writing how often a rival would leave it behind is design mimicry at the scale of a single document.
An AI tool is useful at two points here. It can widen your list of rivals, and it can draft, for each rival, the documents that story would leave behind. It fails where the archive begins. Asked to summarize council minutes, a tool may return a fluent quote that appears nowhere in them, and a tool asked whether your clue is decisive tends to agree. Reading the minutes yourself and writing the two probabilities yourself stay human. When your case rests on interviews and documents, the qualitative inquiry chapter shows how to gather and trace that evidence, and the text-as-data chapter covers coding a larger archive.
15.5 An AI failure case
You paste your mouse comparison into a chatbot and ask, “Is this a natural experiment, and can I say the microbe causes leanness?” It answers with a confident, well-formatted yes: your setup “resembles a natural experiment,” the difference “can be read causally,” the write-up “looks rigorous.” This is two named failures at once: plausible-but-wrong-method, attaching a design label to a comparison whose key assumption plainly fails, and silent scope change, upgrading associated with to because while sounding like it settled the question. Fluency is not evidence. You catch it by refusing the label and asking the one question the model cannot answer for you: how was treatment actually assigned? No lottery, no cutoff, no outside force decided which mice carried the microbe. They sorted themselves by diet and behavior, so there is no as-if-random assignment, the back door stays open, and because is not earned. Say associated with, and log why.
15.6 It is your turn
You are working inside Studio 5: Develop the pathway. This lesson serves the observational-causal pathway. If your declared pathway is different, skim it and work the lesson that matches; the studio page routes you.
Your design is declared and diagnosed. This pathway chapter is where a causal question either gets its identification argument in writing, or gets an honest boundary instead.
The hands-on half of this section lives in the chapter’s companion notebook: open it in Colab with the badge at the top, and work the steps there.
Commit your own answer first every time, then delegate. Treat each of these as the opening move of a loop, not a single question. Ask, read, name the one thing the reply got wrong about your setting, and ask again with that correction in hand. On causal questions the second pass matters more than anywhere else in this book, because the first reply almost always accepts your framing, and your framing is exactly what needs attacking.
Three decisions stay yours alone. Whether your design identifies a causal effect is a judgment about how treatment was really assigned, and only you know that. Which confounder most threatens your comparison turns on the mechanism, which a model cannot see in your data. And whether your finding earns because or must stop at associated with is the whole skill of this chapter. A tool will happily call your comparison a “natural experiment” because you typed the words. Earning because is your signature, not the model’s.
Decide which case you are in. If your inquiry asks what would happen if something changed, and nobody randomized the change, this is your pathway. If your inquiry is descriptive, predictive, or genuinely experimental, it is not, and you run the drill in step 6 instead.
If this is your pathway, write your causal question as exactly two things: a treatment (the thing that varies) and an outcome (the thing you measure). One sentence, no hedging.
Draw the causal diagram. Put treatment and outcome at the ends and add every third variable you can think of that pushes on both. Circle the confounder you most fear. Then write, in one line, whether you can measure it.
List the confounders to verify (commit your own list first).
I claim a gut microbe causes leanness in wild mice. List traits that plausibly raise both microbe presence and leanness, in a table with the trait and the direction it would bias a naive comparison. Then, as a hostile reviewer, name the one trait you are most likely wrong to include and the important one you left out.After running, verify (counters illusion of completeness): compare the list to the one you committed first, and map each trait to a back-door path (trait → microbe, trait → leanness). A trait with only one arrow is not a confounder.
Name your leverage and its price. Selection on observables, a natural experiment, difference-in-differences, regression discontinuity, or instrumental variables (Chapter 41, on natural experiments, introduces difference-in-differences, regression discontinuity, and interrupted time series, and the field guide to further designs introduces instrumental variables; name the one your situation seems to fit and say what you would have to learn and defend before using it), and underneath it the short list of load-bearing assumptions it rests on, each in plain language a skeptic could argue with. Start with the one you most doubt. If you have no leverage, write the causal-language boundary sentence instead: what your comparison shows, and the because you are declining to claim.
Locate a real design (commit your own guess at the leverage first).
Act as a research librarian in biology. Find peer-reviewed observational studies that used a natural experiment or a discontinuity to estimate the causal effect of a gut microbe on a host trait. For each: title, authors, year, venue, and the as-if-random source. Only include work you are confident exists; mark anything uncertain.After running, verify (counters confident fabrication): open each source and read the line yourself before citing it, which is primary-source reading. Cut any you cannot find.
A second angle, optional:
Red-team the causal sentence you wrote (write the sentence yourself, do not let AI draft it).
Here is a causal sentence I wrote: "[paste your sentence]." Act as a hostile peer reviewer. Name every place it over-reaches or claims more than my observational design identifies. Do not rewrite it for me; list the weaknesses so I fix them.After running, verify (counters sycophantic agreement): if the objections are all mild, push back with “assume the sentence is wrong; name the single worst flaw.” Praise without an objection is a red flag.
Write down one number you would recompute by a second route to check any estimate a collaborator or an AI hands you, and say what result would make you abandon the causal reading entirely.
If this is not your pathway, run the classification drill. Write the causal version of your question anyway, name the confounder it would turn on, and say what assumption you would have to defend to answer it. Knowing what you are not claiming is worth as much as knowing what you are.
Log all of it in your AI Research Ledger, and verify with a named method from the Verification Guide; the causal diagram is the method built for this chapter, and simulation with a confounder you plant yourself is the strong second check. An AI reviewer may attack the assumption alongside you; the decision to keep because or fall back to associated with stays yours.