37 Open and Reusable Research Packages
This chapter is part of a book in active development and has not yet been through the author’s review. Content may change as the review advances.
The research decision. Decide what goes into the reproducibility capsule that ships with your research note, and defend the claim that a stranger can rerun your work and get your numbers. You own what the package includes and whether “it reproduces” is honest, and you never let a clean run stand in for a correct one.
37.1 Why this decision matters
The decision on the table: what goes inside the capsule that ships with your work, and whether the sentence “this reproduces” is honest when you write it.
“I don’t want to hear that it runs on your laptop. Hand me the folder, let me clear everything, and let me press run. If your headline number does not come back on my machine, you do not have a result yet. You have a memory of one.” — a replication editor, opening the folder attached to your submission
A research note argues your claim in prose. The capsule proves the claim survives without you in the room. If the only place your analysis runs is the laptop where you built it, in the click-order you happen to remember, then no one can check you, and a result no one can check is a rumor with a chart. This chapter is how you package the work so the checking is easy, and how you stay honest about what “reproducible” really means.
37.2 The concept
A reproducibility capsule is everything a stranger needs to rebuild your numbers and nothing they would have to guess (Wilson et al. 2017). Example: a folder holding your notebook, your data, and a short record of every by-hand choice. It has five parts, each defined once.
- A runnable notebook that passes restart-and-run-all, meaning you clear the kernel and run every cell top to bottom with no memory of earlier clicks. Example: Runtime, Restart and run all, and your headline number reappears.
- A data-provenance note: where each dataset came from, its version, and how it may be used (Wilkinson et al. 2016). Example: the download URL, the date, and the licence.
- A fixed seed, a starting number that makes every random step return the same values on every run (Sandve et al. 2013). Example:
SEED = 464feeding each resample. - A decision log: the by-hand choices that shaped the result, each with its reason. Example: every row you dropped, and why.
- An AI-use ledger: every tool, its task, and how you verified its output.
Capsules break in boringly predictable ways, which is good news, because a predictable failure is a catchable one. Named, they are the five package sins: a hard-coded path that exists only on your machine, a missing seed that moves every run, a by-hand edit no clean run reproduces, an undocumented exclusion with no logged reason, and stale data a reader cannot reobtain. The lab’s auditor scans for all five. But hold onto the one line that carries the whole chapter: a capsule with zero flags is runnable, never proven correct. The scan gets you to the starting line. A person who reruns you cold is the race.
37.3 A worked example
You collected precinct-level turnout for twelve counties in one state’s midterm election, and your headline is the share of precincts where turnout fell below 40%. Watch the same analysis shipped two ways.
The sinful capsule loads ~/Desktop/turnout_clean.csv, a file only your laptop has. It bootstraps a confidence interval for that share with no seed, so the interval shifts on every run. It patches one duplicated precinct name by hand in a cell no clean run repeats. It silently drops every precinct whose registered-voter count was missing, with no note on why. And its data was downloaded “sometime last spring,” before the state certified the count, with no version recorded. Five sins, and every one breaks the rerun.
The clean capsule loads the certified file through its public URL, with the source, the download date, and the terms of use in the provenance note. It fixes SEED = 464 before the bootstrap. It records the one relabel in the decision log with its reason. It keeps the precincts with missing registration counts, or drops them by a logged, pre-declared rule. Now a stranger clears the kernel, runs top to bottom, and your share of precincts below 40% comes back. Same number, no guessing.
Here is the honesty check the auditor cannot do for you. The clean capsule can still be wrong: your seed might be fixed on the wrong subset, or dropping the missing-registration precincts might be a bad rule cleanly logged. Runnable is not correct. That gap is exactly why your capsule is exercised by a person, not only by a script.
Every repair in the clean capsule, pinned seed included, is on the published list of rules for reproducible computational work (Sandve et al. 2013).
The block below is the clean capsule’s analysis cell. Every repair the sinful version did by hand happens here in code, and the seed makes the interval the same on every run.
import numpy as np, pandas as pd
SEED = 464 # sin four, fixed: the seed is pinned
rng = np.random.default_rng(SEED)
# Twelve counties of precinct turnout, with the two data problems the sinful
# capsule handled silently: a duplicated precinct name and missing registrations.
precincts = pd.DataFrame({
"county": np.repeat([f"county {i:02d}" for i in range(1, 13)], 25),
"precinct": [f"P{i:04d}" for i in range(300)],
"turnout": np.clip(rng.normal(0.44, 0.09, size=300), 0.05, 0.95),
})
precincts.loc[17, "precinct"] = precincts.loc[16, "precinct"] # the duplicate
precincts.loc[rng.choice(300, 14, replace=False), "turnout"] = np.nan
deduped = precincts.drop_duplicates("precinct")
missing = deduped.turnout.isna().sum()
below = (deduped.turnout < 0.40).sum() / deduped.turnout.notna().sum()
boot = [(rng.choice(deduped.turnout.dropna(), deduped.turnout.notna().sum())
< 0.40).mean() for _ in range(2000)]
lo, hi = np.percentile(boot, [2.5, 97.5])
print(f"precincts loaded : {len(precincts)}")
print(f"duplicate names removed : {len(precincts) - len(deduped)}")
print(f"registrations missing : {missing} (reported, not silently dropped)")
print(f"share below 40% turnout : {below*100:.1f}% [{lo*100:.1f}%, {hi*100:.1f}%]")
print("\nrerun this cell: the interval does not move, because the seed is")
print("pinned and every repair above happens in code a clean run repeats")37.4 An AI failure case
You paste your notebook into your AI and ask, “will this run cold and produce my low-turnout share?” It walks through every cell and answers, with total confidence, “Yes, this runs top to bottom cleanly and returns your headline number.” It sounds like a green check. It is not. The AI never executed anything. It read the code and narrated it. Cell 3 loads ~/Desktop/turnout_clean.csv, a path only your laptop has, and the fluent walkthrough described reading that file as if it were sitting there.
You catch it the only way that counts: you actually run restart-and-run-all on a fresh kernel, ideally in Colab on a machine that is not yours. The FileNotFoundError appears on cell 3 in seconds. A narrated run is not a run. You verify the notebook by executing it, not by reading a paragraph about executing it.
37.5 It is your turn
You are working inside Studio 11: Reproduce and package. Keep what you write here; the studio’s milestone chapter is where it joins the other lessons’ pieces into one artifact you can defend.
Your project already has a research note and a folder that runs; this step turns the two into one capsule someone could pick up a year from now and reuse.
The hands-on half of this section lives in the chapter’s companion notebook: open it in Colab with the badge at the top, and work the steps there.
Commit your own answer first, then delegate. Packaging is a loop you will run more than once: you ask, you fix what comes back, you rerun the capsule cold, and the rerun surfaces the next gap. Some tools will now run that loop themselves, editing files and rerunning until nothing errors. A loop that ends with no errors has proved the code executes. It has not proved the numbers are right, and only you can tell the difference. Each prompt is a checkable job, not a verdict.
You decide what goes in the capsule and whether the sentence “this reproduces” is honest. The tool can list gaps and pin versions, but it cannot record your true data provenance, judge whether an exclusion’s logged reason is a good reason, or certify that your analysis is correct rather than merely runnable. The name on the folder, and the claim that a stranger can trust it, are yours.
Assemble the five parts for your own project: the runnable notebook, the data-provenance note, the fixed seed, the decision log, and your AI-use ledger. Anything a reader would otherwise have to guess belongs in one of the five.
Locate the standard tool.
Act as a reproducibility assistant. Name the standard file and format for recording the exact package versions a notebook needs in order to rerun, and cite the official documentation. Only name tools you are confident exist.After running, verify: open the official docs and confirm the file and its syntax exist. Counters confident fabrication (an invented tool name arrives as confidently as a real one).
Run the chapter’s audit on your own key lines and fix what it flags, starting with the sin you were most tempted to leave alone. Hard-coded paths and missing seeds are the two that break strangers most often.
List so you can verify (the cold replicator).
Here are the key lines of my capsule: [paste]. Playing a replicator who has only these lines and none of my memory, list every input, file, or by-hand step you would need to rerun my headline number and might not find here.After running, verify: match each named gap against what your own
audit_capsulerun actually flagged, and drop any gap that maps to no real missing line. Counters illusion of completeness (a tidy capsule that looks whole while one input is missing).Write the README a stranger reads first: what the project asks, what the headline number is, which file produces it, and in what order to run things.
Rerun it cold in a second environment, a clean Colab session or a machine that is not yours, and check the headline returns within rounding. A number that travels is reproducible. A number that does not means something you carried by hand was doing quiet work you never packaged.
Red-team the claim.
My claim is: "a stranger can rerun this capsule and get my number." Act as a hostile replication reviewer and name every way it could fail on a machine that is not mine. Do not fix it for me.After running, verify: if it only reassures you, push back and demand the single worst failure. Counters sycophantic agreement (praise that reviews your ego, not your package).
Write one honest sentence about the limit of all this: your capsule is runnable, which is not the same as correct, and name the choice inside it you would most want a reviewer to question.
Log the packaging round in your AI Research Ledger, and verify at least one output with a named method from the Verification Guide. An AI reviewer may run the check with you; the decision to accept or reject stays yours.
Milestone next. This was the last lesson of Studio 11. Milestone 11: Your reproducible package is where the lessons’ pieces become the studio’s versioned artifact. Produce it before you move on.