23 AI as Analytical Assistant
This chapter is part of a book in active development and has not yet been through the author’s review. Content may change as the review advances.
The research decision. You decide which analytical tasks the assistant runs and which numbers you re-derive yourself before any of them reach a claim. The assistant supplies labor by the bucket. Which checks count, which flags are real, and what the surviving number means: those stay on your side of the desk.
23.1 Why this decision matters
The decision on the table: which analytical work you hand off, and which numbers you personally re-derive before you believe them.
“I do not care that a model ran your robustness checks. I care which ones you chose, which numbers you re-derived yourself, and which flags you confirmed against the data. Show me that trail and I trust your judgment. Show me the model’s transcript and I trust neither.” — a discussant at a labor economics seminar, reading your results section
An AI tool can write your analysis code, run a dozen robustness checks, and hand you a clean table in seconds. That speed is exactly why a loose habit is dangerous. Paste the table into your paper and you have signed your name to numbers you never checked. The discussant above wants the short list of decisions that stayed yours. This chapter gives you the habit that survives that question.
23.2 The concept
An AI analytical assistant is an AI tool you hand a well-specified analytical task, never your judgment (Vaithilingam et al. 2022). Example: you say “recompute this gap after dropping the three people who enrolled with an offer already in hand,” and it writes and runs that one check. The task is checkable. The verdict is not its job.
The unit you attack is your headline estimate, the single number that stands in for your whole finding. Example: “people who went through the program received a first offer about 0.7 standard deviations sooner.” Before you defend that number, you try to break it, and breaking it is the labor you delegate.
Two attacks do most of the work. A robustness check re-runs the same finding under a different but equally defensible choice and asks whether the answer holds (Simonsohn et al. 2020). Example: you reported a mean time to first offer, so you also compute the median and check the story survives. A placebo test runs your exact analysis where the effect cannot exist and asks whether what comes back is ordinary for a world with nothing in it (Lipsitch et al. 2010). Example: you shuffle the “program” and “no program” labels at random, many times, and look at the pile of fake gaps your machinery produces. Random labels always differ a little, so the pile is not zeros; what matters is whether your real gap looks ordinary in it or sits far outside.
23.2.1 The assistant works in a loop, and the loop needs a log
You will not get your analysis from one prompt. You prompt, read the output, interrogate it, refine, and run it again, and agentic tools now run those cycles on their own, writing and executing and patching until something looks finished. Verification attaches to each cycle, not to the last one. A number that survived turn four says nothing about the code that produced it on turn seven.
There is a second reason to care, and it is the one that bites. Every re-prompt is a fork in your analysis. If you keep re-asking until the estimate looks better, you have run a specification search wearing a friendly face, and you have run it without telling anyone, including yourself. The fix is cheap: keep a running log, one line per cycle, saying what you asked, what came back, and what you changed and why. An assistant that ran ten silent variants and reported the nicest one is not an assistant. It is an unlogged search.
So here is the division of labor. The assistant proposes checks, writes code, and prints numbers, cycle after cycle. You decide which checks count, you log the cycles, and you verify every number before it reaches your claim. A tool that ran without an error has not proven its result is correct. You verify the number, not the paragraph about the number.
23.3 A worked example
You are studying whether a free job-search assistance program, a four-week service that helps people target applications and rehearse interviews, gets participants to a first job offer faster. Your outcome is weeks from enrollment to first offer. Your headline estimate: participants reached an offer about 0.7 standard deviations sooner. You use an analytical assistant to attack it.
Delegate. You ask the assistant to recompute the gap across a small grid: all enrollees versus dropping the three who already had an offer in hand when they signed up (the sample), mean versus median weeks to offer (the measurement), and the raw gap versus one adjusted for each person’s prior work experience (the specification).
Inspect and verify. The assistant returns eight numbers. You do not trust them yet. You recompute two rows by hand with a second, simpler expression, and they match. The direction holds across all eight, and the magnitude ranges from about 0.6 to 0.7 standard deviations. Your honest headline is a direction plus a range, not one flattering number. Three cycles went into getting that grid to run, and all three are in your log.
Placebo and red-team. You ask the assistant to shuffle the program labels and re-run many times. Your real gap sits far out in the tail of the fake ones, which lowers your worry that the machinery alone is manufacturing the effect, as far as this check can see. Then you ask it, as a hostile reviewer, for the single worst flaw. It says with total confidence that the most experienced workers are driving the result. You already dropped the already-employed group in the sample check and the gap held, so the data refute that flag. The flaw it missed is the one that matters: weeks to an offer is not the quality of the offer, so your claim must stay about speed of placement, not wages or fit.
The assistant did the arithmetic. You decided which checks counted and which flags the data confirmed.
Reporting the whole grid of defensible choices rather than one flattering cell is the discipline specification-curve analysis was built to enforce (Simonsohn et al. 2020).
The block below builds the grid the assistant would return, then runs the placebo shuffle yourself. Recompute two rows by hand before you believe any of them, exactly as the section says.
import numpy as np, pandas as pd
SEED = 464
rng = np.random.default_rng(SEED)
n = 300
experience = rng.normal(0, 1, size=n)
enrolled = rng.random(n) < 0.5
weeks = (18 - 6.0 * enrolled - 1.6 * experience
+ rng.normal(0, 6.5, size=n))
had_offer_already = np.zeros(n, dtype=bool)
had_offer_already[rng.choice(np.where(enrolled)[0], 3, replace=False)] = True
weeks[had_offer_already] = 1.0
df = pd.DataFrame({"enrolled": enrolled, "weeks": weeks,
"experience": experience, "pre_offer": had_offer_already})
def gap(d, stat): # in standard deviations, sign flipped
f = getattr(d.groupby("enrolled")["weeks"], stat)()
return (f[False] - f[True]) / d.weeks.std()
grid = []
for sample, d in (("all enrollees", df), ("dropping pre-offers", df[~df.pre_offer])):
for measure in ("mean", "median"):
raw = gap(d, measure)
adj_resid = d.weeks - np.polyval(np.polyfit(d.experience, d.weeks, 1),
d.experience)
adj = ((adj_resid[~d.enrolled].agg(measure)
- adj_resid[d.enrolled].agg(measure)) / d.weeks.std())
grid += [{"sample": sample, "measure": measure, "spec": "raw", "gap (sd)": raw},
{"sample": sample, "measure": measure, "spec": "adj. experience",
"gap (sd)": adj}]
g = pd.DataFrame(grid)
print(g.round(2).to_string(index=False))
print(f"\nall eight point the same way; magnitude runs "
f"{g['gap (sd)'].min():.1f} to {g['gap (sd)'].max():.1f} sd")
# placebo: shuffle the labels and re-run
fake = [gap(df.assign(enrolled=rng.permutation(df.enrolled)), "mean")
for _ in range(2000)]
beat = int((np.array(fake) >= gap(df, "mean")).sum())
print(f"placebo: of 2000 label shuffles, {beat} produced a gap this large")23.4 An AI failure case
You hand an assistant your grid and it returns a polished eight-row table, every row near 0.7, and declares the result “fully robust.” The run threw no error. Here is the trap: its “median” rows silently call the same mean function as the “mean” rows, so four of the eight numbers are duplicates wearing different labels. The table agrees with itself because it never varied the handle it claims to vary. This is the illusion of completeness, a thorough-looking output missing the one thing that matters, shading into confident fabrication, a number stated with certainty that never came from the code path it names.
You catch it by recomputing one “median” row by hand. The true median gap is 0.55, not 0.70. The grid overstated how much your choices agreed. A green check is not a correct result.
23.5 It is your turn
You are working inside Studio 7: Produce a reproducible first analysis. Keep what you write here; the studio’s milestone chapter is where it joins the other lessons’ pieces into one artifact you can defend.
Your project has verified code and a first number. This step runs the real analysis with an assistant beside you, and keeps the receipts for every turn of the loop so your analysis stays one analysis instead of a quiet search.
The hands-on half of this section lives in the chapter’s companion notebook: open it in Colab with the badge at the top, and work the steps there.
Commit your own expectation first, then delegate. Each prompt is a checkable job, not a request for a verdict.
Three calls never leave your hands. You decide which checks count for the claim you want to make, which flagged problems the data actually confirm, and the final claim you defend, with its boundary and its range. The assistant proposes and computes. The evidence decides, and you are the one who reads the evidence.
Before the assistant touches anything, write your headline estimate in one sentence and the answer you expect. You are committing on the record so a surprising result surprises you.
Hand over one well-specified task at a time. Ask for the number, the exact columns, and the filter behind it, never for a verdict on what it means.
Keep a cycle log as you go: one line per turn with what you asked, what came back, and what you changed and why. If an agentic tool ran turns you did not see, make it list them and add them to your log.
Make the loop confess.
Before I accept this table: list every version of this analysis you ran while producing it, in order, with what changed each time and why. If you tried a variant and set it aside, it goes on the list too.After running, verify: compare its list against your own cycle log and re-run any variant it mentions that you never saw. Counters an unlogged specification search (the loop quietly trying options until one looks good).
Re-derive at least two numbers yourself, by hand or with a second simple expression, before any of them enter a claim.
Delegate the computation, keep the numbers checkable.
Recompute my time-to-offer gap under each row of this grid and return a table: row label, the gap, and the exact columns and filter used for that row. Do not summarize; show one number and its code path per row.After running, verify: recompute two rows by hand with a second expression and confirm they match the table. Counters plausible-but-wrong-method (a row whose label and code quietly disagree).
Run a placebo: shuffle your group labels, re-run the same code unchanged, and confirm the fake gap lands in the ordinary part of that pile.
Close with the milestone’s three checks: restart and run everything from a clean state and confirm the headline numbers match, record your environment (versions, packages, data files), and trace your provisional claim to the exact output that supports it.
Log the analysis in your AI Research Ledger, cycle log attached, and verify at least one output with a named method from the Verification Guide. Alternative code fits a computed gap well. An AI reviewer may run the check with you; the decision to accept or reject stays yours.
Red-team the surviving claim.
Here is my claim: "the program gets people to a first offer about 0.7 SD sooner." Act as a hostile reviewer of labor-market program evaluations. Name the single worst way this result could mislead, and the exact data check that would confirm or kill it. Do not rewrite my claim.After running, verify: run the check it names, and treat the flaw as real only if your own output confirms it. Counters sycophantic agreement (praise that reviews your ego, not your evidence).
Milestone next. This was the last lesson of Studio 7. Milestone 7: Your first reproducible analysis is where the lessons’ pieces become the studio’s versioned artifact. Produce it before you move on.