25 Diagnostics and Negative Tests
This chapter is part of a book in active development and has not yet been through the author’s review. Content may change as the review advances.
The research decision. You decide which negative test your design actually needs, a check aimed at a place where the answer must be zero, and you decide what a clean pass does and does not license you to say. Picking the test is the research. Running it is the easy part.
25.1 Why this decision matters
The decision on the table: which check aimed at a guaranteed zero you run, and what passing it earns you.
“Before you tell me the extract works, show me the plate with no extract on it. If the blank disk also cleared the bacteria, you have not discovered a drug. You have discovered that your solvent is toxic.” — a microbiology lab advisor, on the first plate they ask to see
A striking result is the easiest thing in the world to produce by accident. Your procedure has many moving parts: how you handled the samples, how you measured, how you coded the groups. Any one of them can manufacture a signal that has nothing to do with the thing you set out to study. A negative test is how you find out whether the world produced your result or your own pipeline did. Skip it, and you can spend months defending an artifact.
25.2 The concept
A negative test is a check that runs your exact analysis on a situation where the true answer has to be zero. Example: measuring your “effect” in a group that received no treatment at all. Be precise about what “zero” means there, because this is where the test is most often misread. The TRUE answer is zero; the number your sample returns will almost never be exactly zero, because different samples wobble. What a clean pass looks like is a reading small enough to sit comfortably among the readings your own procedure produces when nothing is happening. What should worry you is a reading far out from that pile, big enough that ordinary wobble is a strained explanation. Notice the hedge in both sentences: chance does occasionally produce a large reading, so no cutoff turns this into a verdict. Demand exact zeros and you will “fail” perfectly healthy machinery, or worse, tinker until a readout prints 0.00 and call the tinkering a fix.
A good negative control has to satisfy three conditions, and all three matter (Lipsitch et al. 2010). Your proposed cause must not be able to reach it, or it is not negative. The artifact you fear must be able to reach it, or it is not a test: a control your pipeline never touches cannot catch your pipeline. Example: if the worry is that your solvent kills bacteria, the control disk must carry the solvent, prepared and measured exactly like the real ones, minus only the extract. And the check must be sensitive enough to show an artifact that would actually matter: run too few plates, or measure too coarsely, and a control sits near zero no matter what is going on. A quiet control from an insensitive check is not evidence of a clean pipeline; it is no evidence at all. Three members of this family show up in almost every audit.
- Placebo test: you replace the real cause with a fake one that cannot act, and confirm the effect disappears. Example: swap the real treatment and control labels for a coin flip, and check that the estimate collapses toward zero.
- Falsification test: you check a consequence that must be false if your explanation is right, and confirm it is false. Example: your compound can only act after you add it, so a reading taken before you added it should show nothing.
- Negative control: you point the same machinery at an outcome your cause could not possibly touch. Example: an antibiotic should not change the diameter of the agar dish, so a “shrinking dish” reading would mean the measurement, not the drug, is talking.
One term earns its own line. A vehicle control is the negative control matched to how you delivered the cause: the carrier with the active ingredient left out. Example: if you dissolved your extract in ethanol, the vehicle control is a disk soaked in ethanol alone. What every one of these tests hunts for is an artifact, a signal produced by your procedure rather than by the thing you study. Example: a clear ring caused by the solvent, not the extract.
Alongside them sits the cheapest diagnostic in research, the leave-one-out check: drop each case in turn, recompute, and see whether one observation is quietly carrying your whole finding. It answers a different question from a negative test. The negative test asks whether your procedure invents signal. Leave-one-out asks whether your result rests on a single point.
The negative tests all share one move: aim the machinery where the answer must be zero, and check that what comes back is ordinary for a world where it is (Rosenbaum 2002). Here is the boundary you must never blur. A passed negative test reduces your concern about one specific artifact, as far as that check could have detected it. It does not prove the artifact is absent, it does not prove your effect is real, and it never turns a correlation into a cause. Report what the check could and could not have caught, not just that it passed.
25.3 A worked example
You are testing whether an extract from a medicinal plant kills a common bacterium. You soak a small paper disk in the extract, lay it on a lawn of bacteria growing on an agar plate, incubate overnight, and next morning measure the zone of inhibition, the clear ring where bacteria failed to grow around the disk. A wide ring looks like a real antibacterial effect, and it is tempting to write that down.
Before you do, you run the negative test. You dissolved the extract in 70% ethanol, so your vehicle control is a disk soaked in 70% ethanol alone, placed on an identical plate and treated exactly the same way. If that disk leaves a bare, uninhibited lawn, the test passed: ethanol on its own did nothing, so the ring around your extract disk is not a solvent artifact, and your effect survives one specific worry. If the vehicle disk grows its own clear ring, the test failed. The ethanol is killing the bacteria, and you can no longer tell how much of the extract’s ring is the extract and how much is the solvent. The finding is an artifact until you redo it at a lower concentration.
Before you trust that pass, ask one more thing: could this check have caught a problem worth catching? A positive control is a condition you know produces a signal, run through the same machinery. Example: a disk soaked in a standard antibiotic should clear a wide ring. Read what that does and does not tell you. If even the antibiotic disk reads flat, your assay is broken today and the quiet vehicle told you nothing. If it clears a ring, you have learned that the system responds to a strong inhibitor, which is not the same as learning it would notice a small one.
The gap between those two matters, because the artifact you fear is usually small. So decide first how big a solvent effect would actually change your conclusion, then show the assay can see something that size: run a dilution series down toward that magnitude and check that the rings still track the dose and that repeat plates agree closely enough to tell that difference from noise. An assay that detects a powerful antibiotic and loses a one-millimetre shift has not cleared your vehicle; it has only failed to notice.
Notice what a clean pass still does not buy you. It does not tell you the extract works inside a living animal, or which molecule is responsible. It lowered your worry about one artifact, as far as this assay could detect it, and that modesty is the point.
A negative control earns its name only under stated conditions: it must share the suspected bias and be unable to show the real effect (Lipsitch et al. 2010).
The block below reads all four plates at once, because a negative control means nothing until you have also seen that the assay can detect something.
import numpy as np, pandas as pd
SEED = 464
rng = np.random.default_rng(SEED)
# Zone of inhibition (mm) on identical plates, ten replicates each.
conditions = {
"plant extract in 70% ethanol": 14.0, # what you want to claim
"vehicle only (70% ethanol)": 0.0, # negative control: must be bare
"standard antibiotic": 22.0, # positive control: must show a ring
"extract in 20% ethanol": 11.5, # the redo at lower solvent
}
plates = {c: np.clip(rng.normal(mu, 1.2, size=10), 0, None)
for c, mu in conditions.items()}
print(pd.DataFrame({"condition": list(plates),
"mean zone (mm)": [v.mean().round(1) for v in plates.values()],
"max zone (mm)": [v.max().round(1) for v in plates.values()]})
.to_string(index=False))
vehicle, positive = plates["vehicle only (70% ethanol)"], plates["standard antibiotic"]
print(f"\nnegative control ring : {vehicle.mean():.1f} mm -> "
f"{'PASS' if vehicle.mean() < 1 else 'FAIL, the solvent is doing work'}")
print(f"positive control ring : {positive.mean():.1f} mm -> "
f"{'PASS' if positive.mean() > 5 else 'FAIL, the assay cannot detect anything'}")
print("\nboth controls must be read before the extract's ring means anything.")
print("a negative control that passes on a broken assay proves nothing")25.4 A seeded simulation
So what does “the vehicle did nothing” actually look like on paper? Not a row of clean zeros. The code below builds a world where the vehicle truly has no effect at all, then runs the experiment 2,000 times. Each run measures twelve matched vehicle and blank disks, with a shared plate-to-plate wobble and ordinary measurement noise, and reports the average difference.
import numpy as np
import matplotlib.pyplot as plt
SEED = 464
rng = np.random.default_rng(SEED)
reps, pairs = 2000, 12
plate = rng.normal(0, 0.4, size=(reps, pairs)) # shared plate effect
blank = 6.3 + plate + rng.normal(0, .35, size=(reps, pairs))
vehicle = 6.3 + plate + rng.normal(0, .35, size=(reps, pairs))
nulls = (vehicle - blank).mean(axis=1) # true vehicle effect: exactly zero
print("exactly zero:", int((nulls == 0).sum()), "of", reps, "experiments")
print("rounds to 0.00 at two decimals:", int((nulls.round(2) == 0).sum()))
print("typical spread (middle 95%):", np.percentile(nulls, [2.5, 97.5]).round(2))
print("this run's reading:", round(float(nulls[0]), 3), "mm")
Count them: in two thousand experiments, in a world built so the vehicle does nothing whatsoever, not one reading came back exactly zero. Sixty-two of them round to 0.00 when you print two decimals, which is worth knowing, because a readout showing “0.00” is a rounded number and not a measurement of nothing. The readings pile up around zero, with the central 95 percent running from about -0.29 to +0.27 mm and rarer runs reaching roughly -0.48 and +0.47; the first run happened to return -0.03. A researcher demanding a true zero would have called all two thousand of these honest experiments failures.
That pile is what you compare against, and building your own is the useful habit: simulate your null world at your own sample size, look at the readings it produces, and then ask where your real control sits among them. A reading of -0.03 is unremarkable there. A reading of +1.2 mm sits far outside anything this null world produced, and now you have something to investigate.
Two limits keep this honest. The pile is only as trustworthy as the assumptions that built it, and it is an illustration of ordinary variation, never a pass line. Do not turn “inside the middle 95 percent” into a rule: that band excludes one honest experiment in twenty by construction, and drawing a cutoff there is a statistical decision, and no single cutoff settles it. Ask the three questions first. Could my cause have reached this control? Could the artifact I fear have reached it? Could this check have detected an artifact big enough to matter? Then use the standard error and the interval you already met to say how unusual this control is, and report that evidence with its assumptions rather than a verdict.
25.5 An AI failure case
You ask an AI to design a negative control for your ethanol-based assay. It answers, with complete confidence, “use a disk soaked in sterile water.” You run it, the water disk comes back perfectly clean, and the tidy conclusion writes itself: control passed, effect confirmed. Here is the trap. Water is not the vehicle you used. You delivered the extract in ethanol, so the artifact you needed to catch is ethanol toxicity, and a water disk can never show it. The clean result told you nothing at all.
You catch it with one question the tool never asked itself: which artifact could this control actually have detected? A valid vehicle control matches how you delivered the cause, solvent for solvent. The moment you see the mismatch you swap water for 70% ethanol and run the test that can genuinely fail. This is a plausible-but-wrong-method: the advice sounded like textbook practice and was wrong for your case.
25.6 It is your turn
You are working inside Studio 8: Stress-test and adjudicate. Keep what you write here; the studio’s milestone chapter is where it joins the other lessons’ pieces into one artifact you can defend.
Your project has a headline estimate with a range around it. This step points two checks at places where the answer must be zero, so you find out whether the world produced your result or your own pipeline did.
The hands-on half of this section lives in the chapter’s companion notebook: open it in Colab with the badge at the top, and work the steps there.
Commit your own expectation first, then delegate a checkable job. Expect to work in cycles: propose, interrogate, refine, ask again. One judgment never improves by re-prompting, though, and that is whether your control is genuinely null. You settle that once, from the mechanism, and no amount of looping settles it for you.
You decide which negative test your design actually needs, whether the control is genuinely null (a place your cause truly cannot reach), and what a clean pass licenses you to claim. A tool can list candidate controls, but only you know the mechanism well enough to certify that your cause cannot touch the one you chose. The judgment about what a passed negative test bought you, and how far its sensitivity reached, stays yours.
Name the artifact you are most afraid of: the one way your procedure, rather than your subject, could have manufactured your result. One sentence, written down before you go looking.
Choose the negative test that could catch it, whether that is a placebo label, a falsification outcome measured where the effect cannot yet exist, or a negative control your cause could not possibly touch. Then confirm the control is genuinely null. If your cause can reach it, it is not a negative test and it will fail quietly.
Locate the standard control.
Act as a lab-methods assistant. I ran an antibiotic disk-diffusion assay with a plant extract dissolved in 70% ethanol. Before any advice, name the standard negative (vehicle) control for this assay and cite a protocol or textbook where it appears. Only name controls you are confident are standard practice.After running, verify: open the cited protocol and confirm the control is real and standard before you use it. Counters confident fabrication (an invented “standard” control arrives as confidently as a real one).
A second angle, optional:
List candidate negative tests, then verify each.
Here is my analysis: [one-sentence design, your cause, your outcome]. Propose three negative tests: a placebo label, a falsification outcome measured where the effect cannot yet exist, and a negative-control outcome my cause could not plausibly affect. For each, name the quantity that is truly zero, and say what reading would count as ordinary versus large enough to worry about at my sample size.After running, verify: for each test, confirm two things. Your cause genuinely cannot touch the negative control, or it is not null. And the artifact you fear genuinely can touch it, or the test cannot catch anything. Reject any answer that asks for a reading of exactly zero. Counters plausible-but-wrong-method (a “negative control” that is not actually null, or that your pipeline never touches).
Before you run anything, write down two things. First, how big an artifact would actually matter for your claim: a shift of what size would change what you conclude? Second, what ordinary looks like: simulate your null world at your own sample size and see the range of readings it produces, so you are comparing against a pile rather than against 0.00. Then commit to reporting your result whichever way it lands.
Run one diagnostic beside it. Leave-one-out is the cheapest: drop each case in turn, recompute, and find out whether a single observation is carrying your finding.
Record what you got against what you predicted, and if the estimate moved, say by how much. Then write your boundary in one line: the artifact this pass makes less likely, how far its sensitivity reached, and the causal or general claim it still does not establish.
Red-team a clean pass.
Here is a negative test I ran and its result: [describe the control and its near-zero outcome]. Act as a hostile reviewer. Name every reason this test could read as zero for the WRONG reason: too few samples to detect anything, a control too weak to carry the artifact, a zero I would have gotten no matter what. Do not reassure me.After running, verify: if it only praises the clean result, push back and demand the single way the test is uninformative. Counters sycophantic agreement (praise that reviews your relief, not your evidence).
Log both tests in your AI Research Ledger, and verify at least one output with a named method from the Verification Guide. Simulation is the one useful tool for a negative test: build a dataset by hand where no effect exists, run your test machinery on it many times, and see the readings scatter around zero. Note how large they get by chance, then judge your real control against that pile rather than against 0.00. Pair it with a positive control to show the system responds at all, and then with a check aimed at the smallest artifact that would change your conclusion, since responding to a strong signal does not prove the assay would notice a weak one. An AI reviewer may run the check with you; the decision to accept or reject stays yours.