19 Hybrid and Complex Designs
This chapter is part of a book in active development and has not yet been through the author’s review. Content may change as the review advances.
The research decision. When your answer has to be stitched together from several measurements or stages, decide whether the combined design is still one aligned thing you can diagnose before you run it. Stacking careful pieces does not automatically produce a careful result, and deciding to cut a piece you cannot characterize is as much a design decision as adding one.
19.1 Why this decision matters
The decision on the table: whether the extra moving piece in your design earns its place, or just adds an error you cannot account for.
“Do not hand me the headline number. Hand me the error budget: every quantity you measured, how you measured it, and how each one’s uncertainty flowed into the final answer. A precise result built on one uncharacterized step is a precise wrong answer.” — an antitrust economist reviewing a report that claims to have measured how concentrated a market is
A complex design fails in a quiet way. Each piece looks careful on its own, so the whole looks careful too. But an answer assembled from several measurements is only as trustworthy as its worst-characterized piece, and the assembly can amplify a small mistake into a large one. This chapter gives you the discipline to take such a design apart and diagnose the whole before you trust its number.
19.2 The concept
Every research design has four parts, named MIDA by Blair, Cooper, Coppock, and Humphreys (Blair et al. 2019) and developed in RDSS (Blair et al. 2023): the Model (your written picture of how the world could work), the Inquiry (the one exact quantity you want from it), the Data strategy (every procedure that makes the data, from sampling to measurement), and the Answer strategy (the whole procedure that turns those data into an answer, uncertainty included). The Colab lab builds all four from scratch. This chapter takes them somewhere harder.
A complex, or hybrid, design is a research design whose four MIDA parts have more than one moving piece: several measurements, stages, or sub-questions stitched into a single answer (Blair et al. 2023). Example: computing a market’s concentration score from two separate measurements, each drawn from its own source and carrying its own error.
Two ideas decide whether such a design holds together. The first is alignment, meaning all four parts point at the same single quantity. Complexity breaks it in a fresh way: two carefully built sub-measurements can each be fine yet quietly target slightly different quantities that never combine into the one you named. The second is error propagation, meaning each measured input’s uncertainty flows into the final answer, sometimes amplified (Joint Committee for Guides in Metrology 2008). Example: if your answer depends on the square of a measured share, a 1% error in that share becomes roughly a 2% error in the result.
Keep the lab’s two failure words close. Bias is a systematic tilt that more data does not shrink; variance is run-to-run wobble that more data does. Complexity lets a small systematic error hide behind an impressively small random one.
19.3 A worked example
You want to know how concentrated your city’s grocery market is: does a handful of chains dominate it, or is it split among many sellers? The standard answer is a concentration index, a single score that rises as a market’s sales pile up in fewer hands (U.S. Department of Justice and Federal Trade Commission 2023). You compute it by squaring each seller’s market share and adding the squares, where a seller’s share = its revenue / total market revenue. That one formula is a complex design in miniature. It stitches two sub-measurements into one answer: each chain’s revenue, which you take from a monthly industry survey, and the total market revenue in the denominator, which you have to measure separately.
Watch the anatomy. A single month of reported revenue is noisy, because a holiday week or a late filing pushes any one chain’s figure up or down. So you make a smart data-strategy move: average twelve months of revenue for each chain before you compute shares. That averaging crushes the variance, and your index looks beautifully repeatable, moving a point or two at most when you swap which months you use.
Here is the trap the lab trains you to see. Repetition does nothing for bias. Your industry survey covers the chains, but not the independent corner stores or the online grocery deliveries, so the total market revenue you divide by is systematically too small. Every share is therefore systematically too large, and because shares are squared, that tilt is amplified straight into the index. Your data strategy and your model have drifted apart. An index that reads 2,862 and never wobbles is not a correct index, only a confidently wrong one, and it can push a market across the line competition authorities treat as highly concentrated. The fix is a redesign of the data strategy, taking the denominator from a source built to cover every seller, such as a government economic census, not one more year of the same survey.
The block below computes the index both ways, with the survey’s total and with the true one. Rerun it with different months and watch how little the wrong number moves.
import numpy as np, pandas as pd
SEED = 464
rng = np.random.default_rng(SEED)
months = 12
chains = {"chain A": 38.5, "chain B": 29.5, "chain C": 21.0, "chain D": 13.5}
uncovered = 22.0 # independents + online delivery: real sales, never surveyed
avg = {c: rng.normal(rev, rev * 0.09, size=months).mean()
for c, rev in chains.items()}
surveyed_total = sum(avg.values())
def index(total):
return sum((rev / total * 100) ** 2 for rev in avg.values())
print(pd.DataFrame({"avg monthly revenue ($m)":
pd.Series(avg).round(1)}).to_string())
print(f"\nindex using the SURVEYED total : {index(surveyed_total):,.0f}")
print(f"index using the true total : {index(surveyed_total + uncovered):,.0f}")
print("\naveraging twelve months bought precision, not correctness. the")
print("denominator is missing sales, so every share, and the index, tilts up")19.4 An AI failure case
You paste your revenue table into an AI tool and ask for the concentration index. It returns, with total confidence, index = 2,862 ± 18. The number looks plausible for a market with a few big chains, the interval is tiny, and the code ran without an error. It is wrong. The tool propagated only the month-to-month revenue variance, the part you already crushed by averaging twelve months, and never modeled the systematic bias from a denominator that leaves out every seller your survey does not reach. That tight ± 18 describes wobble it could see and stays silent about the tilt it could not.
You catch it in two moves. First, check the assumption the number rests on: does the total your shares divide by actually cover the whole market? Second, run a falsification move: recompute the index from a deliberately narrower survey, dropping the smallest chains too, and watch the “precise” index drift systematically upward. A quantity that shifts well outside its own stated interval had an uncertainty that was never honest. Verify the number, not the tidy interval beside it.
19.5 It is your turn
You are working inside Studio 5: Develop the pathway. This lesson is an optional overlay: work it when your project needs it, and skip it without guilt when it does not.
You have studied your primary pathway and at least one useful contrast. This step decides whether your own project needs more than one moving piece, and makes you justify the answer either way. The four MIDA parts you are taking apart here are Blair, Cooper, Coppock, and Humphreys’ framework, developed at book length in RDSS.
The hands-on half of this section lives in the chapter’s companion notebook: open it in Colab with the badge at the top, and work the steps there.
Commit your own answer first, then delegate. Each prompt is a checkable job, not a request for a verdict. Run them as a loop rather than a single shot: the second pass, where you hand back the piece the tool skipped and ask again, is usually the one that finds the uncharacterized step. Agentic tools that chain those passes themselves will assemble a complete-looking error budget in one go. A complete look is the failure mode this chapter is about.
Three calls stay yours. You decide what single quantity your inquiry names (which market, over what geography and time window), whether each sub-design actually targets that same quantity so the parts align, and whether to trust a combined number your diagnosis says a systematic error dominates. A tool can propagate the arithmetic, but it cannot decide that a precise number is a correct one. That judgment is the whole point of the chapter, and it is yours.
Write your design as an ordered list of the measurements and stages it takes to get from raw data to your answer. One line means a simple design. Two or more means you are running a hybrid, whether you meant to or not.
For each line, name which MIDA part it lives in and the single quantity it targets. Then hold the list against your inquiry. Do all the pieces point at the same quantity, or has one quietly drifted to a neighboring one?
Decompose the design into pieces you can check.
Here is my combined design: [paste your model, inquiry, and how you measure each seller's revenue and the total market revenue]. Break it into sub-designs. For each sub-measurement, name which MIDA part it lives in, the one assumption it depends on, and how its error enters the final index. Return a table.After running, verify: set the table beside your own decomposition. Any sub-design or assumption it skipped is exactly the gap to chase. Counters illusion of completeness (a tidy table that omits the one uncharacterized step).
Pick the piece whose error would hurt most and trace how its uncertainty reaches your final answer. Flag any step that squares, divides, or ratios a measured value, because a small tilt gets amplified exactly there.
Locate the real rule.
Act as an industrial-organization research assistant. For a market concentration index computed as the sum of every seller's squared market share, name two things and cite a real, retrievable source for each: (1) the index thresholds competition authorities use to call a market moderately or highly concentrated, and (2) the standard rule for propagating the uncertainty in the measured revenues and the market total into the index. Only state values you are confident exist.After running, verify: open the cited source and confirm the thresholds and the propagation rule actually appear there. Counters confident fabrication (an invented threshold arrives as confidently as a real one).
Decide, and write the one-sentence justification. Either “this piece buys me something a single measurement cannot, and I can characterize its error,” or “this piece adds a step I cannot characterize, so I am cutting it and narrowing my claim to match.”
Name the systematic error in what remains that more data will never shrink, and what you would have to change instead: a different source, a wider sampling frame, a better instrument.
Red-team the combined result.
Act as a hostile antitrust economist. Here is my measured concentration index and its stated uncertainty: [paste]. Name the single systematic error that more months of data can never fix and the place my four MIDA parts fail to point at the same quantity. Do not fix it for me.After running, verify: if it only praises your precision, push back and demand the one worst systematic error. Counters sycophantic agreement (praise that reviews your ego, not your error budget).
Log the step in your AI Research Ledger, and verify at least one output with a named method from the Verification Guide. Simulation is the natural check for a stitched design: plant a known bias in one piece, run the whole thing many times, and watch the variance shrink while the bias sits exactly where you put it. An AI reviewer may run the check with you; the decision to accept or reject stays yours.
Milestone next. This was the last lesson of Studio 5. Milestone 5: Your pathway, declared is where the lessons’ pieces become the studio’s versioned artifact. Produce it before you move on.