15 Observational Causal Research
This chapter is part of a book in active development and has not yet been through the author’s review. Content may change as the review advances.
The research decision. Whether your observational comparison has earned the word because or must stop at associated with. You settle it by naming the one confounder the comparison turns on and writing the identification argument that closes the back door, or by admitting in writing that you have none.
15.1 Why this decision matters
The decision on the table: whether your comparison has earned the word because.
Picture a thesis advisor reading your first draft. You wrote, “the treated group did better, so the treatment worked.” Her reply is the sentence you will hear for the rest of your research life: “After is not because. Show me the world where they were not treated, or admit you are guessing.” Almost every question worth asking forbids the clean experiment. You cannot assign a person a disease, a country a war, or a wild animal a gut microbe. So the honest researcher’s real craft is deciding, out loud and in writing, when an observed comparison is trustworthy enough to carry a cause. Get it wrong and a paper claims more than its data support. Get it right and a modest, bounded claim becomes credible.
15.2 The concept
This chapter follows the observational-causal pathway of the design library in RDSS: the conditions you compare were set by the world rather than by you, so a causal reading has to be argued rather than assumed (Blair et al. 2023).
Start with the missing world.
- Counterfactual: the outcome that would have happened under the choice that was not made. Example: how long a coffee drinker would have lived if she had not drunk coffee. You never see it, because she did drink coffee.
A causal claim is a claim about a counterfactual. The trouble is a villain that hides inside almost every observational comparison.
- Confounder: a third factor that pushes on both who gets the treatment and the outcome. Example: a health-conscious lifestyle raises both the chance of a daily coffee habit and lifespan, so coffee drinkers look longer-lived even if coffee did nothing.
You can draw the trouble. A causal diagram is a picture of arrows where each arrow means “this directly influences that.” In it, the confounder creates a back door: a sneaky path linking treatment and outcome that runs through the confounder instead of through any real effect. A raw comparison sees both paths at once and blames the treatment for the confounder’s work.
Two moves close the back door without an experiment.
- Selection on observables: identifying an effect by adjusting for a set of measured confounders sufficient to block every back-door path. Example: compare coffee drinkers and non-drinkers within the same lifestyle group, because lifestyle is what pushes on both. It works only for confounders you can actually see, and the word sufficient is doing real work: the goal is the right set, not the longest one.
- Natural experiment: a situation where a chance-like force outside anyone’s control decided who got treated, making assignment as-if random (credibly like a coin flip). Example: a visa lottery decides who makes a trip, so winners and losers differ only by luck.
One warning before the first move tempts you into adjusting for everything in the file. Adjusting is not free, and adjusting for the wrong variable manufactures bias instead of removing it. A mediator is a variable sitting on the causal path itself: coffee may act on lifespan partly through blood pressure, so adjusting for blood pressure throws away part of the very effect you are estimating. A collider is a variable that treatment and outcome both push on: coffee habits and long life might each raise the chance of appearing in a “healthy aging” study, so adjusting for study membership creates a link between them that exists nowhere in the world. Draw the diagram first; adjust for common causes, and leave mediators and colliders alone (Pearl 2009).
Whichever move you use, you owe a reader an identification argument: your written reason that the comparison recovers a real effect. Every design in this family rests on a short list of load-bearing assumptions the data can never fully confirm, only make plausible (Hernán and Robins 2020). Selection on observables needs the adjustment set to be sufficient and the groups to overlap. A visa-lottery design needs the lottery to be genuinely random and to touch lifespans only through the trip. Name your design’s list; a single sentence per assumption is enough. When you cannot make that argument honestly, you have hit the causal-language boundary, the point past which your evidence stops earning because and must switch to associated with. The most common failure here has a name too. Design mimicry is borrowing the vocabulary (“natural experiment,” “difference-in-differences”) to decorate a comparison whose assumption was never argued. The word is not the argument.
15.3 A worked example
Bring this into a biology lab. You study wild-derived mice and notice a striking pattern: individuals whose gut carries a particular fiber-digesting bacterium are, on average, leaner than mice without it. The headline writes itself: “the microbe causes leanness.” Would you stake a thesis on it?
Draw three boxes first: microbe, leanness, and a third box for anything that could drive both. That third box fills in fast. Diet is a confounder. Mice that forage on high-fiber plants both harbor more of the fiber-digesting bacterium and stay leaner for reasons unrelated to the microbe. Diet points into both boxes, opening a back door, so a naive microbe-versus-no-microbe comparison mostly measures the diet gap and credits it to the bacterium. A colony of ten thousand mice would make that wrong number beautifully precise and no less wrong, because a larger sample shrinks noise, not confounding.
Now the fix. If you recorded each mouse’s diet, use selection on observables: compare carriers to non-carriers within the high-fiber group, then within the low-fiber group, and average the two honest within-diet differences. If the gap collapses toward zero, the microbe was mostly a passenger of diet. The catch is honest: this works only for the confounder you measured. If mice also differ in an unrecorded trait, say a temperament that drives both foraging and metabolism, that trait keeps the back door open and “we controlled for diet” is not identification. Then your defensible claim is that the microbe is associated with leanness, and the causal question waits for a real intervention, such as colonizing genetically identical, identically fed mice. That boundary is what this chapter teaches you to see and to say.
Which variables you must adjust for, and which adjustments open new paths rather than closing them, is decided by the diagram, not by what you happen to have measured (Pearl 2009).
The block below builds mice in which the microbe does nothing at all, and then measures the gap a naive comparison reports. Run it before you trust any carrier-versus-noncarrier number, including your own.
import numpy as np, pandas as pd
SEED = 464
rng = np.random.default_rng(SEED)
n = 10_000
# Diet drives BOTH the microbe and leanness. The microbe itself does nothing.
high_fiber = rng.random(n) < 0.5
microbe = rng.random(n) < np.where(high_fiber, 0.75, 0.25)
leanness = 20 - 3.0 * high_fiber + rng.normal(0, 1.5, size=n) # lower = leaner
df = pd.DataFrame({"high_fiber": high_fiber, "microbe": microbe,
"leanness": leanness})
naive = df[df.microbe].leanness.mean() - df[~df.microbe].leanness.mean()
within = (df.groupby("high_fiber")
.apply(lambda g: g[g.microbe].leanness.mean()
- g[~g.microbe].leanness.mean(), include_groups=False))
print(f"true microbe effect : {0.0:+.2f}")
print(f"naive carrier-vs-noncarrier gap : {naive:+.2f}")
print(f"within high-fiber mice : {within[True]:+.2f}")
print(f"within low-fiber mice : {within[False]:+.2f}")
print(f"average of the two within-diet gaps: {within.mean():+.2f}")
print("\nthe naive gap is the DIET gap wearing the microbe's name;")
print("ten thousand mice make it precise, not true")15.4 An AI failure case
You paste your mouse comparison into a chatbot and ask, “Is this a natural experiment, and can I say the microbe causes leanness?” It answers with a confident, well-formatted yes: your setup “resembles a natural experiment,” the difference “can be read causally,” the write-up “looks rigorous.” This is two named failures at once: plausible-but-wrong-method, attaching a design label to a comparison whose key assumption plainly fails, and silent scope change, upgrading associated with to because while sounding like it settled the question. Fluency is not evidence. You catch it by refusing the label and asking the one question the model cannot answer for you: how was treatment actually assigned? No lottery, no cutoff, no outside force decided which mice carried the microbe. They sorted themselves by diet and behavior, so there is no as-if-random assignment, the back door stays open, and because is not earned. Say associated with, and log why.
15.5 It is your turn
You are working inside Studio 5: Develop the pathway. This lesson serves the observational-causal pathway. If your declared pathway is different, skim it and work the lesson that matches; the studio page routes you.
Your design is declared and diagnosed. This pathway chapter is where a causal question either gets its identification argument in writing, or gets an honest boundary instead.
The hands-on half of this section lives in the chapter’s companion notebook: open it in Colab with the badge at the top, and work the steps there.
Commit your own answer first every time, then delegate. Treat each of these as the opening move of a loop, not a single question. Ask, read, name the one thing the reply got wrong about your setting, and ask again with that correction in hand. On causal questions the second pass matters more than anywhere else in this book, because the first reply almost always accepts your framing, and your framing is exactly what needs attacking.
Three decisions stay yours alone. Whether your design identifies a causal effect is a judgment about how treatment was really assigned, and only you know that. Which confounder most threatens your comparison turns on the mechanism, which a model cannot see in your data. And whether your finding earns because or must stop at associated with is the whole skill of this chapter. A tool will happily call your comparison a “natural experiment” because you typed the words. Earning because is your signature, not the model’s.
Decide which case you are in. If your inquiry asks what would happen if something changed, and nobody randomized the change, this is your pathway. If your inquiry is descriptive, predictive, or genuinely experimental, it is not, and you run the drill in step 6 instead.
If this is your pathway, write your causal question as exactly two things: a treatment (the thing that varies) and an outcome (the thing you measure). One sentence, no hedging.
Draw the causal diagram. Put treatment and outcome at the ends and add every third variable you can think of that pushes on both. Circle the confounder you most fear. Then write, in one line, whether you can measure it.
List the confounders to verify (commit your own list first).
I claim a gut microbe causes leanness in wild mice. List traits that plausibly raise both microbe presence and leanness, in a table with the trait and the direction it would bias a naive comparison. Then, as a hostile reviewer, name the one trait you are most likely wrong to include and the important one you left out.After running, verify (counters illusion of completeness): compare the list to the one you committed first, and map each trait to a back-door path (trait → microbe, trait → leanness). A trait with only one arrow is not a confounder.
Name your leverage and its price. Selection on observables, a natural experiment, difference-in-differences, regression discontinuity, or instrumental variables (this book does not teach these, so name the one your situation seems to fit and say what you would have to learn and defend before using it, rather than reaching for it now), and underneath it the short list of load-bearing assumptions it rests on, each in plain language a skeptic could argue with. Start with the one you most doubt. If you have no leverage, write the causal-language boundary sentence instead: what your comparison shows, and the because you are declining to claim.
Locate a real design (commit your own guess at the leverage first).
Act as a research librarian in biology. Find peer-reviewed observational studies that used a natural experiment or a discontinuity to estimate the causal effect of a gut microbe on a host trait. For each: title, authors, year, venue, and the as-if-random source. Only include work you are confident exists; mark anything uncertain.After running, verify (counters confident fabrication): open each source and read the line yourself before citing it, which is primary-source reading. Cut any you cannot find.
A second angle, optional:
Red-team the causal sentence you wrote (write the sentence yourself, do not let AI draft it).
Here is a causal sentence I wrote: "[paste your sentence]." Act as a hostile peer reviewer. Name every place it over-reaches or claims more than my observational design identifies. Do not rewrite it for me; list the weaknesses so I fix them.After running, verify (counters sycophantic agreement): if the objections are all mild, push back with “assume the sentence is wrong; name the single worst flaw.” Praise without an objection is a red flag.
Write down one number you would recompute by a second route to check any estimate a collaborator or an AI hands you, and say what result would make you abandon the causal reading entirely.
If this is not your pathway, run the classification drill. Write the causal version of your question anyway, name the confounder it would turn on, and say what assumption you would have to defend to answer it. Knowing what you are not claiming is worth as much as knowing what you are.
Log all of it in your AI Research Ledger, and verify with a named method from the Verification Guide; the causal diagram is the method built for this chapter, and simulation with a confounder you plant yourself is the strong second check. An AI reviewer may attack the assumption alongside you; the decision to keep because or fall back to associated with stays yours.