16 Experimental Descriptive Research
This chapter is part of a book in active development and has not yet been through the author’s review. Content may change as the review advances.
The research decision. A coin flip does not decide what kind of question you asked. You decide whether your randomized study is measuring a property the world already has or testing an intervention you would deploy, and you name the artifacts your measurement still has to guard against before the number means anything.
16.1 Why this decision matters
The decision on the table: whether the number your randomized study reports is a property the world already has, or an effect you would go out and cause.
Picture your advisor stopping at one line of your methods page. “You randomized,” they say, “so you wrote causes. Show me the question first.” Random assignment feels causal by reflex, and that reflex is the trap this chapter defuses. What fixes the kind of a question is the question, not the procedure. Point a coin flip at “what would change if we deployed this?” and you have a causal study. Point the same flip at “how much of this property sits in the world right now?” and you have a measurement. Mislabel the second as the first and you overclaim. You promise a lever your evidence never tested, and a careful reader stops trusting the rest of your work.
16.2 The concept
RDSS calls this the experimental-descriptive pathway: random assignment does the work of a measurement design, revealing a characteristic units do not hand over directly (Blair et al. 2023). Because the inquiry stays descriptive, randomizing does not by itself license a causal claim.
Three terms carry the idea. Meet them one at a time.
- Latent characteristic: a real property you cannot read off directly, so a design has to reveal it. Example: how much a search interface’s position pulls clicks, apart from how good each result is (Joachims et al. 2005).
- Controlled stimulus: a version of a prompt, item, or setting that you fix on purpose and assign by chance, so any difference in response pins to the version rather than to who received it. Example: the same result shown in slot 1 for some sessions and slot 5 for others, chosen at random.
- Experiment as a measurement instrument: using randomly assigned controlled stimuli to expose a latent characteristic and report it as a plain description. The number you report is the estimand, the exact quantity the design is built to state, and here it describes the world as it stands.
A measurement can be confident and still wrong, because the setup itself is one of the ingredients. Two artifacts do most of the damage. A demand effect is when the people being measured guess what you are looking for and drift toward it; example: users who notice they are in a study click more carefully than they would at home (Orne 1962). An instrument effect is when the measuring device or wording moves the reading on its own; example: slot 1 renders a larger thumbnail, so the logger credits position with attention the picture earned. Both threaten construct validity, the degree to which your number estimates the concept you meant rather than something adjacent (Cronbach and Meehl 1955). Randomizing who sees what does nothing about a bias that hits every unit alike, which is why you design these out separately.
16.3 A worked example
A music-streaming team wants to pin down a latent property of its “Recommended for you” shelf: how much of a track’s play count comes from where it sits versus how well it matches the listener. Nobody prints that number anywhere, so they build an instrument. For a random slice of sessions they insert one fixed probe track, identical audio and identical artwork, and assign its shelf position at random across six slots, then record the play rate by slot. Because the track never changes and the slot is dealt by chance, the spread in play rate across slots isolates the pull of position alone. The estimand is the shelf’s attention curve: how much each rank commands, described for these sessions.
Here is where the decision bites. Inside the study there truly is a causal effect of slot on clicks, but the team’s question was descriptive, and the honest report says so: “for these sessions, slot 1 drew about three times the plays of slot 5, at equal quality.” The forbidden sentence is “promoting tracks to slot 1 will triple their plays,” because that is a causal, deployable claim the design never tested. And before trusting even the descriptive number, the team guards against two artifacts by design: a demand effect if the probe looks like an ad, and an instrument effect if slot 1’s larger art or a slot-keyed logger inflates the count. The redesign renders every slot identically and logs impressions server-side.
Randomizing position while holding the item identical is the standard way to separate presentation from quality in ranked interfaces (Joachims et al. 2005).
The block below deals the probe track’s slot at random and reports the attention curve. The randomization is real; the claim it licenses is still descriptive.
import numpy as np, pandas as pd
SEED = 464
rng = np.random.default_rng(SEED)
n_sessions = 12_000
# The probe track is identical every time; only its slot is dealt by chance.
slot = rng.integers(1, 7, size=n_sessions)
attention = {1: 0.150, 2: 0.104, 3: 0.081, 4: 0.066, 5: 0.055, 6: 0.043}
played = rng.random(n_sessions) < np.array([attention[s] for s in slot])
by_slot = (pd.DataFrame({"slot": slot, "played": played})
.groupby("slot")["played"].agg(sessions="size", play_rate="mean"))
by_slot["play_rate"] = (by_slot.play_rate * 100).round(1)
print(by_slot.to_string())
r1, r5 = by_slot.play_rate[1], by_slot.play_rate[5]
print(f"\nslot 1 vs slot 5 : {r1/r5:.1f}x, at identical audio and artwork")
print("descriptive, for these sessions. it does not say what promoting a")
print("DIFFERENT track to slot 1 would do")16.4 An AI failure case
You paste your attention-curve result into an AI tool and ask it to write the findings sentence. It returns something fluent: “Moving tracks to the top slot causes a 3x lift in plays, so the team should promote its best matches to slot 1.” It reads like a win, but it is a silent scope change: you asked how attention is distributed, and the tool quietly answered a causal, deployable question about what promoting would do. You catch it by laying its sentence beside yours, word for word. Yours asked how much position pulls; its answer asked what to do. The tells are “causes,” “should,” and the imagined “lift” from an intervention you never ran. Reject it, and rewrite the claim bounded to the units you observed.
16.5 It is your turn
You are working inside Studio 5: Develop the pathway. This lesson serves the experimental-descriptive pathway. If your declared pathway is different, skim it and work the lesson that matches; the studio page routes you.
Your design is declared and diagnosed. This step asks a narrower question of it: is your study measuring a property the world already has, and if so, can it do that honestly?
The hands-on half of this section lives in the chapter’s companion notebook: open it in Colab with the badge at the top, and work the steps there.
Commit your own answer first, then delegate. Each prompt below ends with a check you run yourself. And treat each one as the opening move of a loop, not a one-shot request: read the output, push back on the weakest part, sharpen the prompt, run it again. Agentic tools now run several turns of that loop on their own, which does not retire your check. It just means more output arrives between your checks.
Three calls stay yours. Whether your inquiry is descriptive or causal, because the kind lives in your question’s words and no tool can read your intent. Whether the design actually measures your construct rather than something adjacent, which is a judgment about meaning, not about code. And which artifacts you are willing to stake your name on having ruled out. An AI partner can draft, locate, and attack. You declare and defend.
Read your lead question aloud and underline the verb. Does it ask how much of something already sits out there, or what would change if you intervened? Write the answer in one line. That line decides whether this pathway is yours.
If it is yours, name the latent characteristic your project has to reveal, the property nobody publishes anywhere, and the controlled stimulus you will assign by chance to reveal it.
When you are ready to delegate this step:
Act as a literature scout. List three real, published experiments used to MEASURE a latent characteristic (position bias in a ranking interface, a hidden prevalence, a preference weight) rather than to test a deployable intervention. For each give authors, year, venue, and the single quantity measured, in a table. Only include work you are confident exists; mark any row you are unsure about.After running, verify: open a database and retrieve each study yourself; any row you cannot find in a minute is a confident fabrication, so cut it.
Write your estimand in one sentence: the exact quantity, for which units, described as it stands. Underneath it, write the causal sentence you forbid yourself, the one that starts with “if we”.
When you are ready to delegate this step:
Here is code that estimates an interface's attention curve as the difference in play rate across randomly assigned slots: [paste]. Name every assumption this estimator needs to be valid, and for each, one concrete way it could fail in my logs.After running, verify: check each assumption against your actual pipeline; an estimator that sounds standard but assumes something your data break is a plausible-but-wrong method.
Name the one demand effect and the one instrument effect you most fear, each with the redesign that rules it out.
When you are ready to delegate this step:
Act as a hostile measurement reviewer. Here is my construct and how I measure it: [plan]. List every demand effect and instrument effect that could produce my number even if the true value were different, and the redesign that rules each one out.After running, verify: compare the list against the one artifact you wrote down first; if yours is missing, the tidy output just failed the illusion of completeness test, so add it back.
If this pathway is not yours, run steps 2 to 4 as a drill anyway, treating your question as if you had to measure it rather than intervene. What breaks is your evidence that the pathway you chose fits better. Write that in two lines.
Log the step in your AI Research Ledger, and verify at least one output with a named method from the Verification Guide. A falsification test fits this chapter well: set the true effect to exactly zero in a toy version of your design and confirm your instrument reports about zero. An AI reviewer may run the check with you; the decision to accept or reject stays yours.