36 Replication and Reproduction
This chapter is part of a book in active development and has not yet been through the author’s review. Content may change as the review advances.
The research decision. Handed nothing but the files a study shipped, yours or someone else’s, you decide whether the reported number actually comes back out. Then you rank the weaknesses you found by how much each one threatens the headline claim, not by how easy each one is to fix.
36.1 Why this decision matters
The decision on the table: whether a result regenerates from the files alone, and which of its weaknesses most endangers the claim.
“I do not care that your figure is pretty or that the code ran on your laptop. Hand me your files and let me get the same number back out. If I cannot, you do not have a result yet. You have a story.” — a principal investigator reading a first-year researcher’s lab report
In a working lab, a number nobody else can regenerate is a liability, not a finding. A poster can round, a caption can overstate, and a clean-looking notebook can still hide the one choice that holds the whole claim up. The reviewer above is applying the standard every result eventually meets. This chapter puts you on the reviewing side of that standard first, so you learn to see the cracks before someone finds them in your own work.
36.2 The concept
Two words sound alike and mean different things. Reproduction is getting the same number from the same data and code a study shipped (National Academies of Sciences, Engineering, and Medicine 2019). Example: you open another researcher’s notebook, run it top to bottom, and check that the same growth figure comes back. Replication is getting a similar result from a new study or new data. Example: a second lab runs its own experiment and sees the same effect. This chapter is about reproduction. You test the package, not the world.
What you test is a reproducibility package: the full bundle a study ships so someone else can rebuild its numbers, meaning the data, the code, the run order, any random seed, and the write-up. The one figure the main claim rests on is the headline number. Example: “the supplement raised larval growth by 0.5 mm.” You rebuild that first, using restart-and-run-all, which means clearing the notebook’s memory and running every cell from the top with no manual fixes. If a number only appears when cells run out of order, it is not reproducible (Wilson et al. 2017).
A clean run proves the code executes. It never proves the write-up is true, so you run three audits on top of it. Claims-vs-computation agreement checks that every sentence the write-up asserts is backed by a number the code actually prints. An alternative specification is a different but equally defensible way to compute the same headline, to see whether the answer depends on an undisclosed choice. A hidden assumption is a claim the analysis quietly relies on and never states, which the result would collapse without. The most dangerous one in biology has a name: pseudoreplication, treating repeated measurements from the same animal, plate, or tank as independent data points when the real experimental unit is the animal, plate, or tank (Hurlbert, 1984) (Hurlbert 1984).
36.3 A worked example
A package from another lab tests whether a probiotic added to the water raises the body length of zebrafish larvae. Six tanks: three get the probiotic, three get plain water. Each tank holds about 40 larvae, and body length is recorded for every larva. The poster reads: “The probiotic increased larval body length by 0.5 mm.”
Reproduce it. You restart-and-run-all. The code loads the file, splits larvae by tank type, and differences the two group means. The number that comes back is 0.43 mm, not 0.5 (the figures in this worked example are constructed). The package reproduces, which is a real success. But the write-up says 0.5 and the code says 0.43, and notice that rounding does not explain it: 0.43 rounds to 0.4. This is a claim-output mismatch, your first finding, and it earns you the right to push harder.
Find the choice nobody declared. The package dropped larvae that died before the final measurement and never said so. Death is not an ordinary blank cell. It is a post-treatment event: something that happens after treatment and that treatment itself may cause. If the probiotic changes which larvae survive, the survivors in the two tanks are not the same kind of larva, and their length difference is a comparison between different populations.
You might be tempted to patch the blanks by carrying each dead larva’s last recorded length forward, which produces 0.31 mm. Resist calling that an equally defensible alternative. It quietly assumes a dead larva’s last measurement stands in for a final length it never had, which is a claim about biology, not a neutral default. The two numbers, 0.43 and 0.31, answer different questions with different assumptions, so the gap between them is not a range and reporting it as one would be a third error on top of the first two.
What you report instead is survival by tank type first, then final length among surviving larvae, labeled exactly that way. Say plainly what that pair does and does not do. It is an honest description of what happened in the tanks. It does not deliver an overall causal effect of the probiotic on day-30 length, and no rearrangement of these numbers will, because day-30 length does not exist for a larva that died. Note too that matching survival percentages do not rescue the comparison: two tanks can lose the same fraction and still lose different kinds of larvae. If the study needs one headline outcome, the fix is upstream — predefine a combined outcome before analysis, such as survived-and-reached-a-given-length, and defend why that ordering answers the biological question.
Test the hidden assumption. The package reports its gap as if all 240 larvae were independent, producing a tight interval well above zero. They are not independent. The probiotic was assigned by tank, so larvae sharing a tank share water, food, and crowding. The real experimental unit is the tank, and there are only six. Recompute at the tank level, averaging each tank first and comparing three treated tanks against three controls, and the interval widens until its lower edge touches zero. The point estimate barely moved. The honest uncertainty around it exploded. That is pseudoreplication, and it is the weakness with teeth.
Rank by threat. Pseudoreplication comes first, because it can dissolve the “clear effect” claim. The undisclosed handling of dead larvae comes second, because it decides which larvae the number even describes. The 0.43-reported-as-0.5 mismatch comes third: real, and a genuine reporting failure, but the smallest threat to whether an effect exists.
Treating larvae in a shared tank as independent units is the classic analysis error this example is built to catch (Hurlbert 1984).
The block below builds the package’s data and computes the difference three defensible ways. Run it before you accept any single number, including the one printed on the poster.
import numpy as np, pandas as pd
SEED = 464
rng = np.random.default_rng(SEED)
tanks = pd.DataFrame({"tank": range(6), "probiotic": [True]*3 + [False]*3})
rows = []
for _, t in tanks.iterrows():
n = 40
tank_effect = rng.normal(0, 0.25) # tanks differ, larvae cluster
length = (4.60 + 0.25 * t.probiotic + tank_effect
+ rng.normal(0, 0.35, size=n))
# death is a POST-TREATMENT event, and the probiotic changes who dies
died = rng.random(n) < np.where(t.probiotic, 0.06, 0.18) * (length < 4.6)
rows.append(pd.DataFrame({"tank": t.tank, "probiotic": t.probiotic,
"length": length, "died": died}))
larvae = pd.concat(rows, ignore_index=True)
alive = larvae[~larvae.died]
gap_survivors = (alive[alive.probiotic].length.mean()
- alive[~alive.probiotic].length.mean())
gap_all = (larvae[larvae.probiotic].length.mean()
- larvae[~larvae.probiotic].length.mean())
by_tank = alive.groupby(["probiotic", "tank"]).length.mean()
gap_tank = (by_tank[True].mean() - by_tank[False].mean())
print(f"the write-up claims : 0.50 mm")
print(f"survivors, larva by larva : {gap_survivors:.2f} mm")
print(f"everyone, including the dead : {gap_all:.2f} mm")
print(f"tank means (n = 3 vs 3) : {gap_tank:.2f} mm")
print(f"\ndeaths: {larvae[larvae.probiotic].died.sum()} treated vs "
f"{larvae[~larvae.probiotic].died.sum()} control")
print("three defensible numbers, none of them 0.50, and the unit of")
print("randomization was the TANK, not the larva")36.4 An AI failure case
You paste the whole package into a general AI tool and ask, “Does this study reproduce, and are its limitations complete?” It answers with total confidence: yes, the code runs, and the limitations paragraph looks thorough, covering sample size, measurement noise, and a call for future work. Every sentence is fluent. The tool never once flags that 240 larvae came from only six tanks. It has quietly mistaken a long list for a complete one.
You catch it by refusing to judge the paragraph and judging the design instead. You ask what the experimental unit actually was, then recompute the interval at the tank level. When the lower bound drops to zero, the “clear effect” the tool endorsed turns out to rest on a certainty the data never earned. A green check and a polished paragraph are not a reproduced result. You verify the number and its match to the claim, never the confident story about them.
36.5 It is your turn
You are working inside Studio 11: Reproduce and package. Keep what you write here; the studio’s milestone chapter is where it joins the other lessons’ pieces into one artifact you can defend.
Your project already has claims you can defend and a written disclosure of how AI helped you reach them; this step turns all of it into files built for a stranger to run. Your own cold restart is the solo proxy for that stranger, and it is labeled as such until someone who is not you actually runs it without you in the room.
The hands-on half of this section lives in the chapter’s companion notebook: open it in Colab with the badge at the top, and work the steps there.
Commit your own answer first, then delegate. None of these is a single shot. You send the prompt, read what comes back, interrogate it, sharpen the ask, and run it again. That is the loop, and an audit usually takes two or three turns of it before the tool stops guessing and starts pointing at real lines. Each prompt is a checkable job, not a verdict you hand off.
An AI tool connected to an execution environment can run the code, and asking it to is reasonable. What it cannot do is stand behind the result. So demand the evidence of the run rather than the summary of it: the commands, the environment and package versions, the exact version of the code and data it ran (a commit hash, if the package has one), whether each command exited cleanly or errored, the raw output, and which files it actually touched. A confident “it reproduces” with no execution record tells you nothing about whether the released package ran, a modified copy ran, or nothing ran at all. Then reproduce the headline yourself from a clean start, because the verdict carries your name. You also keep three judgments: which weakness most threatens the claim, how you rank the recommendations, and the final verdict on what the package can and cannot support. A tool proposes suspects. You decide which ones the evidence convicts.
Gather everything into one folder: the data, the code, the run order, the random seed, and a short write-up that states your headline number in words.
Locate every missing input.
Act as a cold reproducibility auditor. Here is everything a peer's package shipped: [paste the README, the data description, the code, and the headline claim]. List every input or step a stranger needs and might not find: the data file and its version, the exact run order, any random seed, any hard-coded path. Put it in a table with the item, whether the package supplies it, and the cell where I should check.After running, verify: map each item to a real cell, then ask “what is the single most important thing you left out?” Counters illusion of completeness (a tidy gap list can still omit the one missing pointer that blocks the whole run).
Reproduce yourself cold. Clear the kernel, run every cell from the top with no manual fixes, and ideally do it somewhere other than the machine you built it on. Write down whether your headline number came back exactly, roughly, or not at all.
Line every sentence of your write-up against a number your code actually prints. Flag each mismatch, including the rounding you did not mean to hide.
Walk the headline cell, then confirm it independently.
Here is the cell that computes the package's headline number: [paste the cell]. Explain line by line what it computes, and name which printed value is the headline a reader should compare against the write-up. Then give me one independent way to confirm that value without rerunning this exact code.After running, verify: confirm the value it names matches the number your run actually printed, then run its independent check yourself. Counters confident fabrication (a fluent walkthrough can describe a number the code never produced).
Change one defensible choice you never disclosed, an exclusion rule or a cutoff or a subset, and record how far your headline moves. Report the spread across those versions, not the friendlier end of it, and check first that each version still answers the same question about the same units. Any version that changes the question, especially one that keeps only units your treatment could have selected, gets reported on its own rather than folded into a range.
Classify every missing value before you fill any of them. Ask whether the number exists and simply was not recorded, or whether an event such as death or dropout means it never existed at all. If treatment could have caused that event, report the event by group first, and label any comparison among the units that remain as exactly that.
Name the assumption your result would collapse without, and rank everything you found by threat to the claim rather than by ease of repair. If you have a peer, swap folders and rerun each other’s cold. The number they get, set beside the number you reported, is your reproduction evidence.
Red-team the hidden assumption.
This analysis treats every measured larva as an independent observation, though the probiotic was assigned by tank. Act as a hostile methods reviewer. Name the one assumption most likely to inflate the reported certainty, say what the honest experimental unit is, and do not reassure me the design is fine.After running, verify: recompute the interval at the tank level yourself and see whether it reaches zero. Counters plausible-but-wrong-method (an independent-observations assumption on clustered data silently shrinks every interval).
Log the audit in your AI Research Ledger, and verify at least one output with a named method from the Verification Guide. An AI reviewer may run the check with you; the decision to accept or reject stays yours.