35 Difficult Questions and Uncertainty
This chapter is part of a book in active development and has not yet been through the author’s review. Content may change as the review advances.
The research decision. Prepare to answer the hardest reasonable question in the channel your venue actually uses: live discussion, a reviewer report, an editor query, or a written response. The answer does the same work in every channel — name the concern, state what the evidence supports, acknowledge what it does not resolve, and show what changes as a result — and you prepare it in advance, never improvise it.
35.1 Why this decision matters
The decision on the table: how you will state your study’s uncertainty and its limits out loud, before a stranger forces the question.
“When a presenter tells me, unprompted, exactly what their study can’t show, I trust everything else they said more. When they insist it’s airtight, I start looking for the crack myself.” — a policy stakeholder who has heard a thousand poster pitches
Your adaptation is finished and honest, whether it is a poster, a talk, or a page. The danger now is your own mouth. Under pressure, a nervous presenter does one of two things, and both misstate the evidence. One buries a real finding under apology. The other, caught in the moment, upgrades a careful “went together” into a confident “caused.” The stakeholder above is not testing your confidence. She is testing whether you know where your claim stops. This chapter gives you the answer that survives her hardest question.
35.2 The concept
The move that turns a limit into a strength has a name: the ULN move, a three-beat way to speak a boundary. It runs uncertainty, then limitation, then next step, and each beat is a term worth defining once.
An uncertainty statement is a sentence saying how much your number could wobble, and why. Example: “with 812 people, my estimate could sit a few points higher or lower.” A limitation is a true sentence about what your design cannot show (Wasserstein and Lazar 2016). Example: “I measured that two things went together, so I cannot say one caused the other.” A next step is the study that would resolve that limitation. Delivered together, ULN sounds like expertise, because you are naming the edge of your evidence on purpose.
The failure ULN prevents is the apology spiral: burying a real, defensible finding under self-erasure like “honestly this is probably all wrong, my sample’s tiny.” The fix is to choose precision, naming the exact boundary, over hedging, vague self-protection that blurs where the claim stops. “This is one campus sample” is precision. “Sort of, take it with a grain of salt” is hedging. Precision informs the listener; hedging only protects you (Bles et al. 2019).
The second instrument is the question bank: a short written list of the hardest fair questions a visitor could ask, each with a prepared answer (Bourne 2007). Five covers the space, one per recurring type (method, alternative explanation, so-what, generalization, AI-use). When you cannot fully answer one, you deliver an honest don’t-know: an admission with content, naming what you cannot say, the thing you can stand on, and the method that would settle it.
35.3 A worked example
You work with usage logs from a small software company’s app. Comparing customers who chose the compact toolbar against those who kept the expanded one, compact users found the target page a median of six seconds faster. At the poster, a visitor asks the question you feared: “so the compact toolbar causes faster navigation?”
Your study cannot say that. Users picked their own toolbar, so power users may have chosen compact and would have been fast with any layout. That self-selection is a confound, a third factor that shapes both the choice and the outcome. So you deliver the two-beat reply. First the truth: “I can’t say it causes the speed, because compact users may already be power users.” Then the interest: “the study that would answer you is randomly assigning the toolbar to new users, and that’s what I’d run next.” You refused the word “causes,” and handed the visitor a real research question rather than a wall.
Stating uncertainty plainly rather than hiding it does not cost you the audience’s trust, which is what the evidence on communicating uncertainty finds (Bles et al. 2019).
The block below builds users in which the toolbar does nothing at all and then measures the six seconds anyway. That is the honest version of the answer you owe the visitor.
import numpy as np, pandas as pd
SEED = 464
rng = np.random.default_rng(SEED)
n = 1200
power_user = rng.random(n) < 0.35 # fast with ANY layout
compact = rng.random(n) < np.where(power_user, 0.72, 0.28) # they self-select
seconds = (34 - 13.5 * power_user
+ rng.normal(0, 6, size=n)) # the toolbar itself does nothing
observed = np.median(seconds[~compact]) - np.median(seconds[compact])
within = [np.median(seconds[~compact & g]) - np.median(seconds[compact & g])
for g in (power_user, ~power_user)]
print(f"observed median gap : {observed:.0f} s faster for compact")
print(f"true effect of the toolbar : 0 s")
print(f"gap among power users only : {within[0]:+.0f} s")
print(f"gap among everyone else : {within[1]:+.0f} s")
print(f"share of compact users who are power users : "
f"{power_user[compact].mean()*100:.0f}%")
print("\nthe six seconds are real and the toolbar did not cause them. that is")
print("exactly the sentence the two-beat reply has to carry")35.4 An AI failure case
You paste your limitation into an AI tool and ask it to “make this sound more confident for the poster.” It returns a fluent, polished sentence: “The compact toolbar improves navigation speed, with strong results across users.” It runs clean, it reads well, and it is wrong. The tool quietly swapped “was associated with faster times” for “improves,” and “improves” is a causal verb your observational logs cannot support. This is a silent scope change dressed as helpful editing.
You catch it by reading the AI’s verb against your poster’s headline, word for word. Your headline says “associated with.” The rewrite says “improves.” The kind of claim moved while the sentence got prettier. You keep your own verb, and log that the polish inflated the claim.
35.5 It is your turn
You are working inside Studio 10: Prepare to publish or present. Keep what you write here; the studio’s milestone chapter is where it joins the other lessons’ pieces into one artifact you can defend.
You can say your claim at two lengths now; this step prepares the part of the conversation you do not control. If your venue has no live questions, the drill still runs: collect the hard questions in writing, from a peer or an AI reviewer, and answer them on paper.
The hands-on half of this section lives in the chapter’s companion notebook: open it in Colab with the badge at the top, and work the steps there.
Commit your own answer first, then delegate. Each prompt below is a checkable job, not a request for approval.
These work best as a conversation. Ask, read, then name what the answer avoided and ask again. A question bank built over three rounds is noticeably harder than one built in a single pass. When a tool runs those rounds itself, the questions arrive already polished, which is why you check the list against the question you actually dread rather than the other way around.
Three decisions never leave your hands: which claim your evidence actually supports, where that claim must stop, and how you acknowledge its uncertainty and limitations. The tool can draft a question or flag a word, but only you decide the compass position your design can bear and the one word you must refuse aloud. Your honest don’t-know is yours to mean, not to recite.
Write the five hardest fair questions someone could ask you, one for each type: method, alternative explanation, why-it-matters, generalization, AI use.
List your hardest questions, to verify against your own fear.
Play a skeptical but fair conference visitor. My project: "[one-paragraph summary of question, design, and headline finding]". Write the single hardest FAIR question for each type: method, alternative explanation, why-it-matters, generalization, AI-use. For each, flag whether an honest two-sentence answer exists.After running, verify: compare the five against the one question you wrote down that you most dread. If it is missing, the set is not complete. Counters illusion of completeness (a tidy five that omits the exact question you fear).
Draft a two-sentence answer to each and mark it defend or concede. A concede answer names the narrower claim you would retreat to on the spot.
Red-team your reply.
Here is my two-sentence reply to "does it cause it?": "[paste]". Act as a hostile reviewer. Name every word that still claims more than an association supports. Do not rewrite it for me.After running, verify: if it only praises the reply, push back and demand the single worst word. Counters sycophantic agreement (praise that reviews your ego, not your evidence).
Write your uncertainty-and-limitations statement in ULN form: how much your number could wobble and why, the one thing your design cannot show, and the study that would settle it.
Force the uncertainty into the open.
Here is my headline number and how I got it: "[paste]". Do not restate it. List what is missing before I can state its uncertainty honestly: the sample size, the source of wobble, and the one limitation a reviewer would raise first.After running, verify: confirm each item against your own data before you speak it. Counters missing uncertainty (a clean number reported with no bound).
Read it aloud and cut every self-erasing word, replacing each with the exact boundary it was hiding. Then write one honest don’t-know in full, for the question you dread most.
Log it in your AI Research Ledger, and verify at least one prepared answer with a named method from the Verification Guide, such as peer reasoning. An AI reviewer may run the check with you; the decision to accept or reject stays yours.
Milestone next. This was the last lesson of Studio 10. Milestone 10: Your artifact, ready to publish or present is where the lessons’ pieces become the studio’s versioned artifact. Produce it before you move on.