36 Replication and Reproduction
This chapter is part of a book in active development and has not yet been through the author’s review. Content may change as the review advances.
The research decision. Handed nothing but the files a study shipped, yours or someone else’s, you decide whether the reported number actually comes back out. Then you rank the weaknesses you found by how much each one threatens the headline claim, not by how easy each one is to fix.
36.1 Why this decision matters
The decision on the table: whether a result regenerates from the files alone, and which of its weaknesses most endangers the claim.
“I do not care that your figure is pretty or that the code ran on your laptop. Hand me your files and let me get the same number back out. If I cannot, you do not have a result yet. You have a story.” — a principal investigator reading a first-year researcher’s lab report
In a working lab, a number nobody else can regenerate is a liability, not a finding. A poster can round, a caption can overstate, and a clean-looking notebook can still hide the one choice that holds the whole claim up. The reviewer above is applying the standard every result eventually meets. This chapter puts you on the reviewing side of that standard first, so you learn to see the cracks before someone finds them in your own work.
36.2 The concept
Two words sound alike and mean different things. Reproduction is getting the same number from the same data and code a study shipped (National Academies of Sciences, Engineering, and Medicine 2019). Example: you open another researcher’s notebook, run it top to bottom, and check that the same growth figure comes back. Replication is getting a similar result from a new study or new data. Example: a second lab runs its own experiment and sees the same effect. Most of this chapter is about reproduction: you test the package, not the world. Its last section turns to replication with new data.
What you test is a reproducibility package: the full bundle a study ships so someone else can rebuild its numbers, meaning the data, the code, the run order, any random seed, and the write-up. The one figure the main claim rests on is the headline number. Example: “the supplement raised larval growth by 0.5 mm.” You rebuild that first, using restart-and-run-all, which means clearing the notebook’s memory and running every cell from the top with no manual fixes. If a number only appears when cells run out of order, it is not reproducible (Wilson et al. 2017).
A clean run proves the code executes. It never proves the write-up is true, so you run three audits on top of it. Claims-vs-computation agreement checks that every sentence the write-up asserts is backed by a number the code actually prints. An alternative specification is a different but equally defensible way to compute the same headline, to see whether the answer depends on an undisclosed choice. A hidden assumption is a claim the analysis quietly relies on and never states, which the result would collapse without. The most dangerous one in biology has a name: pseudoreplication, treating repeated measurements from the same animal, plate, or tank as independent data points when the real experimental unit is the animal, plate, or tank (Hurlbert, 1984) (Hurlbert 1984).
36.3 A worked example
A package from another lab tests whether a probiotic added to the water raises the body length of zebrafish larvae. Six tanks: three get the probiotic, three get plain water. Each tank holds about 40 larvae, and body length is recorded for every larva. The poster reads: “The probiotic increased larval body length by 0.5 mm.”
Reproduce it. You restart-and-run-all. The code loads the file, splits larvae by tank type, and differences the two group means. The number that comes back is 0.43 mm, not 0.5 (the figures in this worked example are constructed). The package reproduces, which is a real success. But the write-up says 0.5 and the code says 0.43, and notice that rounding does not explain it: 0.43 rounds to 0.4. This is a claim-output mismatch, your first finding, and it earns you the right to push harder.
Find the choice nobody declared. The package dropped larvae that died before the final measurement and never said so. Death is not an ordinary blank cell. It is a post-treatment event: something that happens after treatment and that treatment itself may cause. If the probiotic changes which larvae survive, the survivors in the two tanks are not the same kind of larva, and their length difference is a comparison between different populations.
You might be tempted to patch the blanks by carrying each dead larva’s last recorded length forward, which produces 0.31 mm. Resist calling that an equally defensible alternative. It quietly assumes a dead larva’s last measurement stands in for a final length it never had, which is a claim about biology, not a neutral default. The two numbers, 0.43 and 0.31, answer different questions with different assumptions, so the gap between them is not a range and reporting it as one would be a third error on top of the first two.
What you report instead is survival by tank type first, then final length among surviving larvae, labeled exactly that way. Say plainly what that pair does and does not do. It is an honest description of what happened in the tanks. It does not deliver an overall causal effect of the probiotic on day-30 length, and no rearrangement of these numbers will, because day-30 length does not exist for a larva that died. Note too that matching survival percentages do not rescue the comparison: two tanks can lose the same fraction and still lose different kinds of larvae. If the study needs one headline outcome, the fix is upstream — predefine a combined outcome before analysis, such as survived-and-reached-a-given-length, and defend why that ordering answers the biological question.
Test the hidden assumption. The package reports its gap as if all 240 larvae were independent, producing a tight interval well above zero. They are not independent. The probiotic was assigned by tank, so larvae sharing a tank share water, food, and crowding. The real experimental unit is the tank, and there are only six. Recompute at the tank level, averaging each tank first and comparing three treated tanks against three controls, and the interval widens until its lower edge touches zero. The point estimate barely moved. The honest uncertainty around it exploded. That is pseudoreplication, and it is the weakness with teeth.
Rank by threat. Pseudoreplication comes first, because it can dissolve the “clear effect” claim. The undisclosed handling of dead larvae comes second, because it decides which larvae the number even describes. The 0.43-reported-as-0.5 mismatch comes third: real, and a genuine reporting failure, but the smallest threat to whether an effect exists.
Treating larvae in a shared tank as independent units is the classic analysis error this example is built to catch (Hurlbert 1984).
The block below builds the package’s data and computes the difference three defensible ways. Run it before you accept any single number, including the one printed on the poster.
import numpy as np, pandas as pd
SEED = 464
rng = np.random.default_rng(SEED)
tanks = pd.DataFrame({"tank": range(6), "probiotic": [True]*3 + [False]*3})
rows = []
for _, t in tanks.iterrows():
n = 40
tank_effect = rng.normal(0, 0.25) # tanks differ, larvae cluster
length = (4.60 + 0.25 * t.probiotic + tank_effect
+ rng.normal(0, 0.35, size=n))
# death is a POST-TREATMENT event, and the probiotic changes who dies
died = rng.random(n) < np.where(t.probiotic, 0.06, 0.18) * (length < 4.6)
rows.append(pd.DataFrame({"tank": t.tank, "probiotic": t.probiotic,
"length": length, "died": died}))
larvae = pd.concat(rows, ignore_index=True)
alive = larvae[~larvae.died]
gap_survivors = (alive[alive.probiotic].length.mean()
- alive[~alive.probiotic].length.mean())
gap_all = (larvae[larvae.probiotic].length.mean()
- larvae[~larvae.probiotic].length.mean())
by_tank = alive.groupby(["probiotic", "tank"]).length.mean()
gap_tank = (by_tank[True].mean() - by_tank[False].mean())
print(f"the write-up claims : 0.50 mm")
print(f"survivors, larva by larva : {gap_survivors:.2f} mm")
print(f"everyone, including the dead : {gap_all:.2f} mm")
print(f"tank means (n = 3 vs 3) : {gap_tank:.2f} mm")
print(f"\ndeaths: {larvae[larvae.probiotic].died.sum()} treated vs "
f"{larvae[~larvae.probiotic].died.sum()} control")
print("three defensible numbers, none of them 0.50, and the unit of")
print("randomization was the TANK, not the larva")36.4 Replication with new data
A package that reproduces has passed one test: its own files return its own number. It has not shown that the effect turns up anywhere outside those files. That second question belongs to replication, the new-data half of the pair this chapter opened with. For the zebrafish study, a replication would mean new tanks, new larvae, and a new batch of probiotic, analyzed with the tank as the unit from the start.
Two kinds are usually named. A direct replication is a new study that repeats the original procedure as closely as it can on new units. Example: a second lab reruns a published pricing experiment with the same instructions, the same prices, and new participants. A conceptual replication is a new study that tests the same claim with a different procedure or measure. Example: checking whether a price anchor moves willingness to pay by running real auctions instead of asking hypothetical questions. Either way, write down first the quantity the original claimed, its estimand, and the population it was about. Example: the average gap in willingness to pay between anchored and unanchored shoppers. A replication can only speak to the quantity it shares with the original. If its participants come from a different population, a smaller effect may mean the claim does not travel to them, not that the original was wrong. Nosek and Errington argue that the label matters less than one test: a study counts as a replication when any outcome it could produce would change your confidence in the original claim (Nosek and Errington 2020). If a disappointing result would only prompt “well, the procedure was different,” the study was probing how far the claim travels. That is useful work, under another name.
The published record gives you reason to take this seriously. When a large team repeated 100 published psychology studies, 97 percent of the originals had reported statistically significant results, but only 36 percent of the replications did, and the replication effects were about half the size of the originals (Open Science Collaboration 2015). A parallel effort in economics repeated 18 laboratory experiments, each with enough participants to detect the original effect at least 90 percent of the time. Eleven of the 18 found a significant effect in the same direction, and the replicated effects averaged 66 percent of the original size (Camerer et al. 2016). Neither says which single original overstated its effect. Both say that one published estimate is often larger than what a new study will find.
So the hard decision is what counts as consistent. Comparing significance labels misleads. “The original was significant and the replication was not” can happen when both studies estimate exactly the same effect, because a small replication has a wide interval. Compare estimates and intervals instead. Here effects are measured as Cohen’s d, which is the gap between the two group averages divided by the typical spread of individual outcomes. Example: d = 0.5 means the treated group’s average sits half of that typical spread above the control group’s average.
Then ask two questions. The first is whether the replication landed where the original said it would. Checking whether the replication’s interval contains the original’s number is not enough, because it treats the original as exact when it is itself one noisy study. Patil, Peng, and Leek use a prediction interval instead: the range a replication of a given size should land in 95 times out of 100, by chance alone, if both studies estimate the same effect (Patil et al. 2016). Its width adds the noise of both studies, so a small original gives a wide range. Example: a 50-person-per-group replication of a 20-per-group original with d = 0.70 should land between about -0.06 and 1.46. You can fix that range from the two sample sizes before the new data arrive. In the psychology project above, 77 percent of the replications fell inside or above their prediction intervals, which says that many “failures” were within ordinary noise, and also that about a quarter fell below, more than chance alone allows.
The second question is whether the replication is precise enough to say the effect is smaller than the original study could reliably have detected. The small-telescopes test is Simonsohn’s answer: it finds the effect size that would have given the original study only a 33 percent chance of detecting it, called d33, and asks whether the replication’s estimate sits clearly below that benchmark (Simonsohn 2015). Example: with 20 people per group, an effect of 0.48 would be detected only a third of the time. A replication showing the effect is clearly below 0.48 says that whatever effect exists is too small for the original’s telescope to have detected reliably. Simonsohn also works out that a replication needs about 2.5 times the original sample to have an 80 percent chance of passing this test when the true effect is zero.
The block below sets up a constructed original, a pricing experiment with 20 people per group that reported d = 0.70. It then runs two replications in a world where the real effect is only 0.10: one the same size as the original, one 2.5 times larger.
import numpy as np
SEED = 464
rng = np.random.default_rng(SEED)
def se_d(d, n): # standard error of d, n people per group
return np.sqrt(2 / n + d**2 / (4 * n))
def cohen_d(a, b):
sd = np.sqrt((a.var(ddof=1) + b.var(ddof=1)) / 2)
d = (a.mean() - b.mean()) / sd
se = se_d(d, len(a))
return d, d - 1.96 * se, d + 1.96 * se, se
# The original lab study (constructed): 20 people per group, d = 0.70.
n_orig, d_orig = 20, 0.70
se_orig = se_d(d_orig, n_orig)
# d33: the effect that would have given the ORIGINAL study only 33% power.
# 1.96 is the usual cutoff; 0.44 is the normal value that leaves 33% power.
d33 = np.sqrt(2 / n_orig) * (1.96 - 0.44)
TRUE_D = 0.10 # the world's real effect is SMALL
print(f"original (constructed) : d = {d_orig:.2f}, "
f"95% interval {d_orig-1.96*se_orig:.2f} to {d_orig+1.96*se_orig:.2f}")
print(f"small-telescope benchmark : d33 = {d33:.2f}\n")
for n in (20, 50): # same size as the original, then 2.5 times
# Prediction interval, fixed BEFORE the new data arrive: where a
# replication of this size should land if both studies estimate the
# same effect. It carries the noise of BOTH studies.
se_both = np.sqrt(se_orig**2 + se_d(d_orig, n)**2)
p_lo, p_hi = d_orig - 1.96 * se_both, d_orig + 1.96 * se_both
treat = rng.normal(TRUE_D, 1, n)
ctrl = rng.normal(0, 1, n)
d, lo, hi, se = cohen_d(treat, ctrl)
z_below = (d - d33) / se # is the effect clearly smaller than d33?
print(f"replication, {n} per group : d = {d:+.2f}, 95% interval "
f"{lo:+.2f} to {hi:+.2f}")
print(f" prediction interval {p_lo:+.2f} to {p_hi:+.2f}, "
f"lands inside: {p_lo <= d <= p_hi}")
print(f" clearly below d33: {z_below < -1.645}")
print("\nboth replications are 'not significant'; only one of them")
print("lands outside what the original predicted, and only that one")
print("shows the effect is too small for the original to detect reliably")The original’s own interval runs from 0.06 to 1.34, so one small study pins down little. The first replication finds d = 0.44, with an interval from -0.19 to 1.06. It is not significant, yet it lands well inside its prediction interval (-0.20 to 1.60), and its own interval holds the benchmark d33 = 0.48. So it is inconclusive: too noisy to support or undercut the original. The second replication finds -0.10, with an interval from -0.49 to 0.29. It lands just below its prediction interval (-0.06 to 1.46) and sits clearly below d33 (the code’s test of “clearly” is a one-sided test at the 5 percent level). That one is informative. Someone counting only significance labels would call both of them failures to replicate, and only the second earns the sentence.
Read even the second with care. It sits only a little outside what the original predicted, so it says the two studies probably disagree, not by how much or which one is off. The prediction interval also takes the original at face value. If the original was published partly because it came out significant, its 0.70 is likely too high, and the interval does not correct for that. Nor does the second replication show the effect is zero, since its interval still allows a modest positive effect up to 0.29. The code uses a shortcut formula (a normal approximation) for the intervals and for d33, so treat the second decimal as approximate.
An AI tool helps with preparation: it can turn a paper’s methods section into a replication checklist and flag where your protocol departs from it. It fails when you ask it for the verdict. Shown the first replication above, a tool will write “the study failed to replicate.” That is plausible-but-wrong-method: it reads a non-significant result as evidence of no effect. It will also paraphrase the original claim into something broader or narrower, a silent scope change that can make any outcome look consistent. So you write the original claim down word for word, decide before you see the new data which outcomes would lower your confidence in it, and let the tool check your arithmetic, never your verdict. When several replications of one claim exist, combining them is a design of its own with its own traps, and the evidence-synthesis lesson takes it up.
36.5 An AI failure case
You paste the whole package into a general AI tool and ask, “Does this study reproduce, and are its limitations complete?” It answers with total confidence: yes, the code runs, and the limitations paragraph looks thorough, covering sample size, measurement noise, and a call for future work. Every sentence is fluent. The tool never once flags that 240 larvae came from only six tanks. It has quietly mistaken a long list for a complete one.
You catch it by refusing to judge the paragraph and judging the design instead. You ask what the experimental unit actually was, then recompute the interval at the tank level. When the lower bound drops to zero, the “clear effect” the tool endorsed turns out to rest on a certainty the data never earned. A green check and a polished paragraph are not a reproduced result. You verify the number and its match to the claim, never the confident story about them.
36.6 It is your turn
You are working inside Studio 11: Reproduce and package. Keep what you write here; the studio’s milestone chapter is where it joins the other lessons’ pieces into one artifact you can defend.
Your project already has claims you can defend and a written disclosure of how AI helped you reach them; this step turns all of it into files built for a stranger to run. Your own cold restart is the solo proxy for that stranger, and it is labeled as such until someone who is not you actually runs it without you in the room.
The hands-on half of this section lives in the chapter’s companion notebook: open it in Colab with the badge at the top, and work the steps there.
Commit your own answer first, then delegate. None of these is a single shot. You send the prompt, read what comes back, interrogate it, sharpen the ask, and run it again. That is the loop, and an audit usually takes two or three turns of it before the tool stops guessing and starts pointing at real lines. Each prompt is a checkable job, not a verdict you hand off.
An AI tool connected to an execution environment can run the code, and asking it to is reasonable. What it cannot do is stand behind the result. So demand the evidence of the run rather than the summary of it: the commands, the environment and package versions, the exact version of the code and data it ran (a commit hash, if the package has one), whether each command exited cleanly or errored, the raw output, and which files it actually touched. A confident “it reproduces” with no execution record tells you nothing about whether the released package ran, a modified copy ran, or nothing ran at all. Then reproduce the headline yourself from a clean start, because the verdict carries your name. You also keep three judgments: which weakness most threatens the claim, how you rank the recommendations, and the final verdict on what the package can and cannot support. A tool proposes suspects. You decide which ones the evidence convicts.
Gather everything into one folder: the data, the code, the run order, the random seed, and a short write-up that states your headline number in words.
Locate every missing input.
Act as a cold reproducibility auditor. Here is everything a peer's package shipped: [paste the README, the data description, the code, and the headline claim]. List every input or step a stranger needs and might not find: the data file and its version, the exact run order, any random seed, any hard-coded path. Put it in a table with the item, whether the package supplies it, and the cell where I should check.After running, verify: map each item to a real cell, then ask “what is the single most important thing you left out?” Counters illusion of completeness (a tidy gap list can still omit the one missing pointer that blocks the whole run).
Reproduce yourself cold. Clear the kernel, run every cell from the top with no manual fixes, and ideally do it somewhere other than the machine you built it on. Write down whether your headline number came back exactly, roughly, or not at all.
Line every sentence of your write-up against a number your code actually prints. Flag each mismatch, including the rounding you did not mean to hide.
Walk the headline cell, then confirm it independently.
Here is the cell that computes the package's headline number: [paste the cell]. Explain line by line what it computes, and name which printed value is the headline a reader should compare against the write-up. Then give me one independent way to confirm that value without rerunning this exact code.After running, verify: confirm the value it names matches the number your run actually printed, then run its independent check yourself. Counters confident fabrication (a fluent walkthrough can describe a number the code never produced).
Change one defensible choice you never disclosed, an exclusion rule or a cutoff or a subset, and record how far your headline moves. Report the spread across those versions, not the friendlier end of it, and check first that each version still answers the same question about the same units. Any version that changes the question, especially one that keeps only units your treatment could have selected, gets reported on its own rather than folded into a range.
Classify every missing value before you fill any of them. Ask whether the number exists and simply was not recorded, or whether an event such as death or dropout means it never existed at all. If treatment could have caused that event, report the event by group first, and label any comparison among the units that remain as exactly that.
Name the assumption your result would collapse without, and rank everything you found by threat to the claim rather than by ease of repair. If you have a peer, swap folders and rerun each other’s cold. The number they get, set beside the number you reported, is your reproduction evidence.
Red-team the hidden assumption.
This analysis treats every measured larva as an independent observation, though the probiotic was assigned by tank. Act as a hostile methods reviewer. Name the one assumption most likely to inflate the reported certainty, say what the honest experimental unit is, and do not reassure me the design is fine.After running, verify: recompute the interval at the tank level yourself and see whether it reaches zero. Counters plausible-but-wrong-method (an independent-observations assumption on clustered data silently shrinks every interval).
Log the audit in your AI Research Ledger, and verify at least one output with a named method from the Verification Guide. An AI reviewer may run the check with you; the decision to accept or reject stays yours.