9 Finding and Verifying Prior Evidence
This chapter is part of a book in active development and has not yet been through the author’s review. Content may change as the review advances.
The research decision. Which of your candidate leads survive retrieval and reading, and therefore become evidence you may build on, and whether the quiet spot they leave on your map is a genuine gap or a search you stopped too early.
9.1 Why this decision matters
The decision on the table: which leads become verified evidence, and whether the quiet spot on your map is a real gap.
Picture a thesis advisor handing your first draft back across the desk. “Show me the conversation your work joins,” she says, “not a pile of summaries. Which studies agree, which fight, and where exactly is the hole you say you are filling?” That question decides whether your project has a place to stand. There are two ways to get it wrong, and both are expensive. Cite a source that does not exist, or does not say what you claim, and every result resting on it is compromised. Announce a gap a paper already filled, and you have promised a contribution you cannot deliver. This chapter is where you make both calls, on the record.
9.2 The concept
A field is a cumulative conversation: a running argument where each study answers, extends, or challenges the ones before it. One team reports that a soil treatment raises crop yield; the next tests whether the effect holds in a different climate. Your job is not to invent an island. It is to find the conversation your question belongs to and add one honest sentence to it.
Every source you keep passes through one habit. The retrieval-verification loop is four steps: ask any tool to surface leads, retrieve the actual source yourself, verify it exists and says what was claimed, and document where you found it (Snyder 2019). A lead is a candidate source you have not confirmed yet. A lead stays a lead until retrieval and verification promote it to a source.
Two named failures are what the loop exists to catch. A citation hallucination is an AI inventing a source that does not exist, dressed in a real-sounding author pair, a recent year, and a plausible journal. A mischaracterized source is a real paper cited for a claim it never makes, such as a study that reports a correlation being cited as proof of a cause. A source can be real and still be mischaracterized, so “does it exist” and “does it say this” are two separate checks (Ji et al. 2023). Run both.
Once you hold one verified source, you grow the rest by snowballing: walking its links in two directions. A backward citation search reads the source’s own reference list to find the older work it built on. A forward citation search finds the newer works that cite it, through a “Cited by” trail. Two solid sources, snowballed both ways, become ten.
That is also how you earn the word gap. A genuine gap is a question your verified sources actually leave unanswered, confirmed by looking. An unfinished search is a question you simply stopped chasing. They feel identical from your desk, and only one is a real contribution. The test is retrieval. You have earned the word gap only after real sources, followed forward and backward, still stay silent on your question (Greenhalgh and Peacock 2005).
9.3 A worked example
You are studying crop science, and your question is whether adding biochar (charcoal made from crop waste and mixed into soil) raises maize yield in sandy fields. You ask an AI tool for evidence. It returns a confident citation: a named author pair, a 2021 date, a real-sounding soil-science journal, and a tidy finding that biochar lifted yield by 18 percent. Perfect. So you go to read it, and it is nowhere. Not in the library portal, not on Google Scholar, not on the publisher’s site.
Here is the decision on the table. Is the missing study proof that “nobody has tested biochar in sandy soil, so my gap is real”? No. Three things could be true. The AI fabricated the study whole, and there is nothing to find. It garbled a real source, right idea but wrong author or year, so you search the claim itself and the likely real authors. Or the paper is real but paywalled or obscure, reachable through your library or by emailing the author. Your next move is the same in all three: the lead stays out of your verified column until retrieval succeeds.
So you do the work. You retrieve four real soil-science studies through the library database, read each one, and snowball. Backward from one 2019 field trial you find the older greenhouse study it built on. Forward through “Cited by” you find two newer trials that tested the same idea in loam and clay. Now you can see the conversation: biochar helps yield in some soils, the size of the effect varies, and none of the four you retrieved ran the trial in the sandy soil your county farms. That quiet spot, ringed by sources you opened yourself, is a gap you can defend.
The failure mode this example is built around, a fluent citation to something that was never published, is a known property of these systems (Ji et al. 2023).
The block below tallies what happened to twelve leads. It carries no citations at all, real or invented; the statuses are the evidence, and the tally is the number that decides whether you have earned the word gap.
import numpy as np, pandas as pd
SEED = 464
rng = np.random.default_rng(SEED)
# Twelve leads returned by a search, each with what happened when you tried to
# retrieve it. The statuses are anonymous: no citation appears here, real or
# invented. Four opened; the rest are still leads.
status = rng.permutation(["retrieved"] * 4
+ ["garbled (real, wrong metadata)"] * 3
+ ["paywalled (request sent)"] * 2
+ ["nothing found"] * 3)
leads = pd.DataFrame({"lead": range(1, 13), "retrieval status": status})
print(leads.to_string(index=False))
counts = pd.Series(status).value_counts()
print("\n" + counts.to_string())
print(f"\nverified column : {counts.get('retrieved', 0)}")
print("everything else stays OUT until retrieval succeeds — a lead you could")
print("not open is not evidence that nobody has studied your question")9.4 An AI failure case
The failure to expect here is confident fabrication. Ask a general-purpose tool for “the most-cited study on biochar and maize yield in sandy soil” and it will often produce one, complete with a DOI-shaped string and a real journal name, because producing a fluent citation is exactly what it is built to do. The detail is not evidence of existence. You catch it the moment you try to retrieve it: paste the exact title in quotation marks into the library database and Google Scholar. If nothing resolves anywhere it should live, the citation is fabricated until proven otherwise, and it never enters your ledger. The fluency was the trap; the empty search result is the tell.
9.5 It is your turn
You are working inside Studio 3: Ground it in verified evidence. Keep what you write here; the studio’s milestone chapter is where it joins the other lessons’ pieces into one artifact you can defend.
You came in with a candidate list of leads. You leave with an evidence base you have personally opened, and a gap you can defend out loud.
The hands-on half of this section lives in the chapter’s companion notebook: open it in Colab with the badge at the top, and work the steps there.
Commit your own answer first, then delegate. Verification is not one prompt either. It is a loop: you ask, you read the output against the paper, you tell the tool exactly where it overreached, and you ask again. The second pass is usually where the honest answer shows up, because the first one is still trying to please you.
Two calls stay yours alone, because your name goes on the map. First, whether a source counts as verified: you retrieved it, opened it, and read the claim yourself, so no AI summary substitutes for the paper. Second, whether your gap is genuine: you walked one backward and one forward link from each verified source and the question still stands unanswered. An AI can point; only you can vouch.
Try to retrieve every candidate on your list. Search the exact title in quotation marks in your library database and in Google Scholar. Anything that does not resolve stays a lead and never enters your evidence base. Note why it failed.
When you are ready to delegate this step:
My research question is: [paste it]. The claim I need sources for is: [paste the claim]. List the five published studies most relevant to this claim, each with authors, year, exact title, and venue, so I can retrieve and verify each one myself.After running, verify: search each exact title in quotes before believing it exists. This counters confident fabrication (a real-sounding citation for a paper that was never written).
Open the ones that resolve and read them. At minimum: the abstract, the design section, and the results. Then write each source’s central claim in one sentence in your own words. If you cannot write that sentence, you have not read it yet.
When you are ready to delegate this step:
Here are the five sources I retrieved and read: [paste them]. For each, state the single claim its evidence actually supports, and flag any where my one-line summary reaches further than the study does.After running, verify: read each flagged claim against the paper’s own results section. This counters the mischaracterized source and silent scope change (a correlation quietly reported as a cause).
Criticize each one. Name the population it studied, the design it used, and the one thing you would not conclude from it. A source you cannot criticize is a source you have not understood.
Draw your evidence map: one node per verified source, lines between the ones that agree, and a marked line between any two that disagree. Contradictions get drawn, never smoothed over.
Find the quiet spot on the map and write your gap as one bounded sentence: “Across the sources I retrieved and verified, ___ has not been established for ___.” If the honest version is “my search is still unfinished,” write that instead. That is a finding about your search, not a failure.
When you are ready to delegate this step:
Here is my research question and the gap I placed: [paste them]. Argue the gap is NOT real: name up to three published studies that may already fill it, with authors, year, title, and venue. Do not reassure me.After running, verify: retrieve every study it names; a source you cannot open closes nothing. This counters sycophantic agreement (a partner that praises your gap has flattered you, not tested it).
Close the loop on your declaration: rewrite your declared question in light of the map, or defend leaving it unchanged with a reason tied to a specific node, and record the resulting version with its reason.
Log the verification work in your AI Research Ledger, naming the method you used from the Verification Guide; primary-source reading is the one that fits every citation. An AI reviewer may try to argue your gap away with you; the decision to keep it or drop it stays yours.
Milestone next. This was the last lesson of Studio 3. Milestone 3: Your evidence base is where the lessons’ pieces become the studio’s versioned artifact. Produce it before you move on.