26 AI as Adversarial Reviewer
This chapter is part of a book in active development and has not yet been through the author’s review. Content may change as the review advances.
The research decision. When a reviewer hands you a list of flaws in your own analysis, you decide which flaws are real, and you decide it by running the one data check that confirms or refutes each, never by going with whichever reviewer sounded most certain. Confidence is not evidence, and a panel of confident reviewers is not three pieces of evidence.
26.1 Why this decision matters
The decision on the table: which of the flaws a reviewer names are real, settled by a check you run rather than by how sure the reviewer sounded.
Picture the review of a proposal to roll out a store change chain-wide. An operations manager is the person whose sign-off lets your change reach every store, and who answers for it when the checkout lines tell a different story. Example: the manager who approved your new self-checkout layout is the one fielding complaints when the queues back up on the first busy Saturday. Their concern is blunt.
“Do not tell me the numbers got better. Tell me which check you ran that would have caught it if they hadn’t. I trust the check, not the graph.”
A confident measurement, a fluent AI critique, and a clean-looking table can all be wrong in the same convincing way. The manager is asking for what sounds least impressive and matters most: the check you committed to before you looked, and the flag you actually verified against the data.
26.2 The concept
An adversarial reviewer is a reader whose job is to attack your result and find where it breaks, not to praise it. Example: a peer who tries to show your measured improvement was luck rather than a real change. The reviewer can be a colleague, one AI tool, or a panel of models, and each fails in its own way.
You defend a result by trying to break it first. A robustness check re-runs the same finding under a different but equally defensible choice and sees whether the answer holds. Example: you measured the improvement during one time of day, so you also measure it in two other realistic time blocks. A placebo test runs your exact procedure where the effect cannot exist and asks whether what comes back is ordinary for a world with nothing in it. Example: you compare two stretches of the old layout against each other. Repeat that over many such pairs, since any two stretches differ a little by chance, and an “improvement” far larger than the rest of that pile is your measurement lying.
The habit that keeps an attack honest is order. Specification searching means trying many versions of an analysis and reporting only the one that gave the answer you wanted, without disclosing the search (Simmons et al. 2011). Its everyday name is p-hacking. Example: you measure ten different hours and show only the hour where your change wins. The cure is a set of pre-listed checks, the full list you will run, committed to before you see any result (Nosek et al. 2018). Example: you write down three time blocks and one placebo, then run all four and report every number, including the ones that did not help.
One more failure earns a name because AI reviewers make it constantly. Correlated error is two reviewers wrong in the same way, so their agreement is an echo, not a confirmation (Peker 2023). Example: two AI models share a blind spot and both flag the same non-issue with equal confidence.
26.2.1 An agentic reviewer still does not get a vote
You can now point a tool at your whole analysis and let it run its own loop: read your code, rerun it, poke at variants, and come back with a ranked list of problems. That is a genuine upgrade over a single prompt, and it will find real errors you missed. It changes nothing about who adjudicates. Every item on that list is still a proposal, and each one still needs the same treatment: name the check, run the check, read your own output. The list is longer and better organized than it used to be, which makes it more tempting to accept wholesale, so hold the line harder. A reviewer proposes a flaw, and the data decide whether it is real. Confidence, human or machine, tells you nothing about whether the flaw exists.
26.3 A worked example
Your store installed a new self-checkout layout, and your data say customers got through the line faster. Before you defend that, you attack it. First, one term. p95 wait time is the wait that 95 out of 100 customers come in under, a fairer summary than the average because it captures the long waits people actually complain about. Example: a p95 of 210 seconds means only the slowest 5 percent waited longer than that. Your headline is a p95 drop from 210 seconds to 137.
You hand the result to three adversarial reviewers, and each names a different fatal flaw. The first says the win is driven by a single unusually quiet morning. The second says you cherry-picked the one customer mix where the layout helps. The third says you compared a fully staffed week against a short-handed one, so you measured staffing, not the layout. All three are confident, and all three disagree.
You do not trust the loudest voice. You turn each flaw into a check. Leave-one-out drops the quiet morning and recomputes the p95: it barely moves, so the first flaw is refuted. Your pre-listed specification grid reruns the drop across three customer mixes, and it holds in all three, so the second is refuted too. The placebo compares two stretches of the old layout against each other. An improvement between two identical stretches would prove the third reviewer right, but it comes back near zero, so the machinery is clean.
The two loudest flags dissolved the moment a check touched them. The flaw that survives is one no further measurement can remove and none of the three raised: this ran at one store on weekdays, not across the chain. So the honest claim is bounded. You measured a p95 improvement for these three customer mixes at this store, not a guarantee for every location your company runs.
Committing to the checks before seeing the results is what keeps this from becoming a search for the version you liked (Nosek et al. 2018).
The block below runs each reviewer’s objection as a check. Notice that the checks, not the reviewers’ confidence, decide which flaw survives.
import numpy as np, pandas as pd
SEED = 464
rng = np.random.default_rng(SEED)
def waits(n, scale, staffed):
return rng.gamma(2.2, scale * (1.0 if staffed else 1.18), size=n)
mornings = 20
old = pd.DataFrame({"morning": np.repeat(np.arange(mornings), 60),
"wait": np.concatenate([waits(60, 40, m % 5 != 0)
for m in range(mornings)]),
"mix": np.tile(["commuter", "shopper", "family"], 400)})
new = pd.DataFrame({"morning": np.repeat(np.arange(mornings), 60),
"wait": np.concatenate([waits(60, 27, m % 5 != 0)
for m in range(mornings)]),
"mix": np.tile(["commuter", "shopper", "family"], 400)})
p95 = lambda d: np.quantile(d.wait, 0.95)
print(f"headline p95: {p95(old):.0f} s -> {p95(new):.0f} s")
# flaw 1: one unusually quiet morning is carrying the win
loo = [p95(new[new.morning != m]) for m in range(mornings)]
print(f"\nleave-one-out p95 range : {min(loo):.0f}-{max(loo):.0f} s "
f"(the drop survives dropping any single morning)")
# flaw 2: it only holds for one customer mix
by_mix = pd.DataFrame({"old p95": old.groupby("mix").wait.quantile(0.95),
"new p95": new.groupby("mix").wait.quantile(0.95)}).round(0)
print("\n" + by_mix.to_string())
# flaw 3: the placebo — two stretches of the OLD layout against each other
placebo = p95(old[old.morning < 10]) - p95(old[old.morning >= 10])
print(f"\nplacebo, old layout vs itself : {placebo:+.0f} s "
f"({'clean' if abs(placebo) < 25 else 'the machinery manufactures gaps'})")
print("three confident reviewers, three checks, and only the checks decide")26.4 An AI failure case
You paste your wait-time summary into an AI reviewer, and it answers with total certainty: your improvement is an artifact of one unusually quiet morning, drop that morning and it vanishes. The claim is specific, mechanistic, and stated without a hedge. It reads exactly like a reviewer who has seen this mistake a hundred times.
Here is how you catch it. You do not act on the verdict. You run the leave-one-out yourself, dropping the quiet morning and recomputing the p95. The improvement barely moves. The confident flaw was fabricated, a real-sounding mechanism bolted onto a problem your data do not have. Trust the certainty and you throw away a genuine result on a hunch. Pointing the check at your own numbers is the only thing that settled it.
26.5 It is your turn
You are working inside Studio 8: Stress-test and adjudicate. Keep what you write here; the studio’s milestone chapter is where it joins the other lessons’ pieces into one artifact you can defend.
Your project has an estimate, a range around it, and a negative test behind it. This step hands the whole thing to a reviewer whose only job is to break it, and makes you the one who decides what broke.
The hands-on half of this section lives in the chapter’s companion notebook: open it in Colab with the badge at the top, and work the steps there.
Commit your own read of the result first, then let a reviewer attack it. Each prompt below hands out a task you can check, with a verify note naming the failure it defends against.
These stay yours, no matter how fluent the reviewer sounds. Which robustness checks you commit to before you look, because only those carry confirmatory weight. A check you think of after seeing the result is still worth running; it is exploratory, and it counts only if you label it that way and report it with the rest. Which flagged flaws your data actually confirm, decided by the measurement and not by the confidence. Whether a flaw is a claim-boundary problem that no further measurement can fix and only a narrower claim can answer. And the one bounded result you will defend, with its uncertainty stated. A reviewer proposes; you verify, and the evidence decides.
Write a one-paragraph summary of your design, your headline claim, and the checks you have already run. A reviewer who does not know what you did will invent flaws you ruled out last week.
Commission the review: ask for the single most serious flaw and the exact measurement that would confirm or refute it. Do this at least twice, with different framings or different tools, and treat two matching answers as one candidate to check rather than as two confirmations.
Red-team the result (you run the check it names).
Act as a hostile operations reviewer, not a cheerleader. Here is a one-paragraph summary of my wait-time study and its headline claim: [paste your summary]. Name the single most serious flaw you can find, and state the exact measurement I could run that would confirm or refute it. Do not rewrite my study and do not list more than one flaw.After running, verify (counters sycophantic agreement): if it calls your setup “sound” or offers only mild praise, push back and ask for the worst problem assuming the result is fake. A flaw is real only once you run the named measurement and your own numbers confirm it.
Take the three hardest points you got back. For each, write the one data check that would settle it, in a single line.
Collapse the panel’s flags (you keep the verdict).
Three reviewers each named one "most serious flaw" in my analysis: [A], [B], [C]. For each, give the single data check that would confirm or refute it, in one line. Then tell me which two of the three are most likely the same underlying concern in different words, and which one is genuinely separate.After running, verify (counters correlated errors): three reviewers echoing one blind spot can feel like three confirmations. Run each check against your own output before believing any flag, and treat unanimous agreement as a candidate to check, never as the check itself.
Run all three checks against your own data. Mark each flag confirmed or refuted by your own output, never by how certain the reviewer sounded. Save the most confident wrong flag you caught; it tells you something about the tool you will use again tomorrow.
Find the flaw that no check can fix, the one that is a boundary problem rather than a measurement problem, and narrow your claim until the claim is true.
Log the review and every adjudicated flag in your AI Research Ledger, and verify at least one output with a named method from the Verification Guide. Peer reasoning belongs here: walk your narrowed claim past a person and see whether it survives their first question. An AI reviewer may run the check with you; the decision to accept or reject stays yours.