12 Declaring and Diagnosing a Research Design
This chapter is part of a book in active development and has not yet been through the author’s review. Content may change as the review advances.
The research decision. Whether the design you wrote down is strong enough to run, decided before you collect a single data point: you simulate it, read how often it lands on the truth, and name the one change that most improves it. Then you either run it, fix it, or narrow what you promised.
12.1 Why this decision matters
The decision on the table: whether this design is strong enough to run, judged before you spend anything collecting data.
Picture a methodologist on your committee reading your plan. She asks one question: “How often would this exact design catch the real effect if you ran it a thousand times?” If your answer is “I never checked,” she stops there. A design can read beautifully, with a sharp question and a clean comparison, and still come back “nothing here” nine times out of ten, or land confidently on the wrong number every time. This chapter lets you find that out cheaply, in simulation, so the months you would have spent collecting doomed data go instead to a design that can actually answer your question.
12.2 The concept
You have already built the four parts of a design. This chapter puts them to the test with a three-step loop this book borrows from RDSS (Research Design in the Social Sciences: Declaration, Diagnosis, and Redesign) (Blair et al. 2023).
Declare means writing your four design parts as something a computer can run: a model that generates fake data, an inquiry it can compute on that fake data, a plan for sampling and assigning conditions, and a rule for turning data into an estimate. Example: code that invents 40 shoppers, flips a coin to give each the new checkout screen, and subtracts the two group averages.
Diagnose means running that declared design many times on fresh simulated worlds and watching how its estimate behaves. Example: run the coin-flip study 2,000 times and collect all 2,000 estimates. Three numbers start the story, and each answers a different worry.
Bias is the average of your errors across the runs, counting direction: each estimate minus the truth, averaged. A design is unbiased when its estimates center on the true answer. Example: the truth is 2, and five runs return 0, 1, 2, 3, and 4. The errors are -2, -1, 0, +1, +2, which average to zero, so bias is zero even though only one run landed on 2. Direction is what makes bias different from plain distance: misses that cancel tell you the design is aimed correctly, while misses that all lean one way tell you it is tilted.
Variance is how much the estimate wobbles from one run to the next. Example: estimates swinging between -5 and +9 around a true value of 2 have high variance. Collecting more of the same kind of data, from the same design, usually shrinks this wobble. It is the one problem size reliably helps.
Power is how often the design detects a real effect, and it means nothing until you say how you would decide (Greenland et al. 2016). A statistical test is the rule you use to decide whether your data conflict enough with “no effect” to call the result real. Example: in the checkout study, a difference-in-means rule: declare an effect when the gap between the two groups is more than about twice its typical run-to-run wobble. The test’s threshold is how demanding that rule is, and it is read across imagined repetitions: at the usual 5 percent setting, a design whose true effect is ZERO would still cry effect in about 5 runs out of 100. The threshold is a property of the rule over many runs, never the chance that your one finding is false. Example: a design that clears that rule in 8 of 100 runs, when the true effect is 2, has 8% power, which is almost useless. Power moves with everything in the declaration: the test, the threshold, the effect you planted, the sample size, and how the world makes noise. Report the whole card beside the number.
Bias, variance, and power are all the same kind of thing: a property of the design read across many simulated runs, never off a single study. Blair, Cooper, Coppock, and Humphreys introduced the term diagnosand for a property of a design you want to diagnose (Blair et al. 2019, 2023). Bias, variance, and power are three of them.
Those three do not exhaust the pile, and fuller diagnoses add more. Two worth knowing by name. Root mean-squared error is the typical size of a miss, with big misses weighted extra. Example: in the five-run example above, the plain average distance is 1.2, but the RMSE is about 1.4, because the two 2-point misses count for more than their share. Coverage is how often the range you report around your estimate actually contains the truth. Example: if you report a range meant to be right 95 times out of 100, coverage checks whether it really is. A design can be unbiased and still miss badly every time, which is exactly why one number is never the whole diagnosis (Morris et al. 2019). The last lesson built these ranges properly; here you only need to know that a diagnosis can ask about them.
Redesign means changing exactly one part in response to the diagnosis, then diagnosing again. Low power from high variance calls for more data. A tilt built into the design, such as who ends up in which group, calls for a different design: more data makes a tilted estimate steadier, not truer.
12.3 A worked example
You work with an online grocery retailer that wants to know whether a one-tap reorder button, which rebuilds a shopper’s last cart in a single press, gets people through checkout faster. Your outcome is seconds-to-complete. Your true effect, in the world you simulate, is that the button shaves 8 seconds off.
You declare it: build shoppers whose baseline checkout times vary a lot, because some browse and some race; flip a coin to assign the button; subtract the two group averages. You diagnose a pilot of 12 shoppers per version. The result is sobering. Bias is near zero, so the design aims true, but the spread is huge because per-person times vary so much, and power comes back under 10%. The design is honest and useless at once. It points at the 8-second gain on average and almost never gets close enough on any single run to prove it.
Now redesign. Variance is the problem, so you raise the sample to 400 per version and diagnose again. Power jumps past 90% and the spread collapses toward the truth. That is variance cured by size. But run one more design as a warning: drop the coin flip and let shoppers opt into the new button themselves. The shoppers who opt in are the frequent, practiced ones who check out faster anyway, so the button gets credit for speed that habit delivered. This confounded design is just as tight at 400 per version, yet it centers on a 14-second gain instead of 8, and no amount of extra data moves it. That gap names the last idea. A randomized design can identify the button’s effect, meaning pin the time gap on the button and rule out other causes, while the confounded one shows only an association, where the two move together but a common cause could explain it.
Diagnosing a design by simulating it many times before running it once is standard practice for evaluating statistical methods (Morris et al. 2019).
The block below declares all three designs and diagnoses each one, which is the whole loop this chapter teaches. Read the bias column and the power column as two different questions.
import numpy as np, pandas as pd
SEED = 464
rng = np.random.default_rng(SEED)
TRUE_EFFECT = -8.0 # seconds the button really saves
def run(n_per_arm, randomize, runs=2000):
"""Declare a design once, then diagnose it by running it many times."""
out = []
for _ in range(runs):
base = rng.normal(150, 32, size=2 * n_per_arm) # people differ a lot
if randomize:
treated = rng.random(2 * n_per_arm) < 0.5
else: # shoppers opt in — the fast ones do
treated = rng.random(2 * n_per_arm) < 1 / (
1 + np.exp((base - 150) / 170))
secs = base + TRUE_EFFECT * treated + rng.normal(0, 8, size=2 * n_per_arm)
out.append(secs[treated].mean() - secs[~treated].mean())
est = np.array(out)
se = est.std(ddof=1)
return {"estimate": est.mean(), "bias": est.mean() - TRUE_EFFECT,
"power": float((np.abs(est) > 1.96 * se).mean())}
designs = {"pilot, randomized, 12/arm": run(12, True),
"randomized, 400/arm": run(400, True),
"opt-in, 400/arm": run(400, False)}
print(pd.DataFrame(designs).T.round(2).to_string())
print("\nsize cured the variance. it did nothing for the opt-in design's bias")12.4 An AI failure case
You paste your confounded observational design and ask, “My estimate is noisy. What should I do?” The tool answers with confidence: “Collect more observations to tighten your estimate.” That sounds obviously right and is exactly wrong for your case. Your problem is bias from self-selection, not variance, and more data only makes a biased estimate a more precise wrong number. You catch it the way this chapter teaches: run the confounded design in simulation at a large sample and watch the estimates cluster tightly around 14 seconds when the truth is 8. The spread shrank; the error did not. That is the plausible-but-wrong-method failure, and your own diagnosis unmasks it.
12.5 It is your turn
You are working inside Studio 4: Declare and diagnose provisionally. Keep what you write here; the studio’s milestone chapter is where it joins the other lessons’ pieces into one artifact you can defend.
You have four design parts on paper. This step asks whether they would survive contact with the world, while a bad answer is still free. The declare-and-diagnose loop, and the word “diagnosand” for the properties you read out of it, come from Blair, Cooper, Coppock, and Humphreys, developed at book length in RDSS.
The hands-on half of this section lives in the chapter’s companion notebook: open it in Colab with the badge at the top, and work the steps there.
Commit your own expectation first, then delegate. Each prompt below leaves you a draft you still have to check. Simulation is where the loop gets fastest and most dangerous at once: modern coding assistants will write the script, run it, read the error, and patch it without waiting for you, and they will happily converge on code that runs cleanly while declaring a different design than yours. Let the tool own the typing. Keep ownership of the true effect you plant and the numbers you read out.
Three calls stay yours. Which world your question assumes, the model you declare, is a claim about reality no tool can make for you. Which single quantity you name as your inquiry decides what “success” even means. And the honest call of whether to run a design your own diagnosis says is too weak is a judgment you sign your name to. An AI can propose threats and draft simulation code, but it cannot decide that your under-powered study is worth months of your life anyway.
Declare your design as something that can be run. If you can code it, write the smallest script that builds a world where you set the true effect, draws your sample, applies your answer strategy, and prints one estimate. If you cannot code it yet, declare it in words precise enough that someone else could: state the true effect you are assuming, the number of units, and the exact arithmetic.
When you are ready to delegate this step:
Act as a simulation assistant. Here is my declared design in words: [model, inquiry, data strategy, answer strategy]. Write the smallest Python that builds a world where I set the true effect, runs the design many times, and reports bias, variance, and power. List every assumption your code makes about my design that I did not state.After running, verify (counters plausible-but-wrong-method): re-run it with the true effect set to zero and confirm it “detects” an effect only about as often as your significance threshold allows. A simulation that fires on a zero effect is declaring a different design than yours.
Diagnose it. Run it a few thousand times and read three numbers: bias (do the estimates center on the truth you planted?), variance (how wide is the spread?), and power (how often does it clear your stated threshold?). Write down the test and threshold you used, or the power number means nothing. If you cannot simulate yet, record each number as “not estimated” and reason in words about direction instead: which way you expect the tilt, and what would widen the spread. Label that reasoning as reasoning. A number you did not compute is not a diagnosis.
Name the worst of the three, and say which kind of problem it is. Wobble and low power often yield to a bigger sample, though not only to it: sharper measurement, better-balanced groups, and a stronger treatment contrast all buy power too. A tilt in who ends up where is a design problem, and a bigger sample makes it more precise rather than less wrong.
When you are ready to delegate this step:
Here is my diagnosed design and its numbers: [bias, variance, power]. List the threats to any conclusion from one run as a table: threat, whether it is a bias / variance / power problem, and the single redesign that reduces it most.After running, verify (counters illusion of completeness): check the table against your own printed diagnosis. The killer your numbers already proved must be the top row; a tidy six-threat list that never names it missed the point.
Redesign once. Change exactly one thing, the one your diagnosis points at, and diagnose again. Record both diagnoses side by side so the improvement is visible.
Make the honest call in one sentence: run it as redesigned, redesign again, or narrow the claim to what this design can actually deliver. All three are respectable outcomes. Pretending you never checked is not.
When you are ready to delegate this step:
Argue against my claim that this design is strong enough to run. As a hostile methodologist, name the one diagnosand I am most likely fooling myself about, and the observation that would expose it.After running, verify (counters sycophantic agreement): if it praises the design without a single objection, discard the answer and diagnose again yourself.
Log both diagnoses in your AI Research Ledger, and verify the key number with a named method from the Verification Guide; simulation is the method for a claim about how a procedure behaves, and a causal diagram is the second check if your inquiry uses the word causes. An AI reviewer may run the diagnosis with you; the decision to run, fix, or narrow stays yours.