39 Conflicting Agents and Human Escalation
This chapter is part of a book in active development and has not yet been through the author’s review. Content may change as the review advances.
The research decision. Your loops have come back and they do not match, or they match a little too well. You decide which of three things you are looking at (real disagreement, correlated error, or manufactured consensus), and you decide the exact moment to stop the loops and take the call into your own hands.
39.1 Why this decision matters
The decision on the table: whether agreement among your AI reviewers is evidence or an echo, and the moment you halt the loops and decide for yourself.
“Don’t tell me your reviewers agreed. Tell me whether they could have agreed for the same wrong reason. Agreement is the easiest thing in the world to manufacture, and the most dangerous to trust.” — a skeptical peer at your practice defense
By now you can run a small team of AI roles on your work, which means you can run several loops at once. The seductive next thought is that a team is safer than one assistant, so three roles blessing your draft must mean three confirmations. It does not. Three loops built the same way, reading the same draft with the same instructions, tend to make the same mistake, so their agreement is one error repeated three times. Worse, a loop that keeps running can talk itself into confidence: each turn polishes the last one’s answer instead of testing it, and what returns to you is a well-defended version of the first mistake. This chapter is about seeing through that, and about the moment you stop counting votes and decide for yourself.
39.2 The concept
When your roles do not match, the first move is not to pick a winner. It is reconciling conflicting outputs: deciding what to believe by first diagnosing why the roles differ. Example: before you side with the reviewer over the methodologist, you ask what would make each of them right. There are three cases, and they call for opposite responses.
Real disagreement is when the roles genuinely read the evidence differently. Example: your reviewer calls a sentence clear and strong while your methodologist flags that same sentence as reaching past your data. This is the good case. The clash points straight at the claim you need to check.
Correlated error is when the roles agree, but on the same wrong assumption, so the agreement proves nothing. Example: three reviewers built from the same base model all miss the same flaw because they share the blind spot. The question to ask is simple: could they be wrong the same way? If yes, their consensus is one signal, not three (Peker 2023).
False consensus is when the roles agree only because your prompt framed the task so they had to. Example: you ask “confirm my draft is ready,” and every role obliges. You led all the witnesses. Reframe the ask neutrally and run it again.
The tool that tells these apart is an independence check: a test that makes agreement worth more than a head-count, either an independent method you run yourself or a way you force two roles to be genuinely independent (Hong and Page 2004). Example: two roles bless a cited number, so you open the source and recompute it yourself before trusting either.
Then comes the decision only you can make. Escalation to human judgment is the moment an AI output would settle rather than inform a decision that is yours, so you stop and decide it yourself (Bansal et al. 2021). Example: a role proposes calling your result “causal,” which would settle your claim boundary, so it escalates to you. What you do at that moment is a human override: you take the pen back and make the call in your own words, backed by your own check, even when the roles agreed on something else. A decision that just counted the roles is not a decision. It is a vote, and votes among look-alike models are meaningless.
Escalation is also the one moment where the loops have to stop. An agentic tool left running will keep looking for a next step, and if nobody halts it, it will take the decision by default, not because you delegated it but because the loop had nowhere else to go. So write your triggers down before you start the round: the specific decisions where the loops pause and wait for you. A trigger you decide in the moment, with three tidy reviews in front of you, is not a trigger. It is a mood.
39.3 A worked example
You estimated the unemployment rate in your town (the share of people who want paid work and cannot find it) from a survey you ran yourself. You asked 250 adults outside the public library, over three weekday afternoons, whether they were working or looking for work, then divided one count by the other. Your draft reports “unemployment here is 4.0%, in excellent agreement with the official rate.”
You run three AI reviewer roles. All three return the same verdict: “Looks solid. Matches the published figure.” Is that three confirmations?
Diagnose before you celebrate. Every one of those roles already expects a number near 4%, because that is the range the familiar headline rates sit in, so each one pattern-matches your estimate to the value it walked in with and blesses it without checking how you got there. That is correlated error wearing the mask of confirmation: they share one blind spot, the anchor to the familiar figure, and it hides your actual mistake. You asked only 250 people, so the luck of who happened to walk by is worth a couple of percentage points on its own, and you asked them on weekday afternoons outside a library, where almost nobody who was at work all day could appear. The tidy “4.0%” was luck riding on an honest-looking method.
This is where you escalate. The reviewers’ agreement would settle your headline claim, and that claim is yours, so you take the pen back. You recompute the rate from your raw responses yourself and carry the uncertainty of a 250-person sample through the arithmetic, and the honest result is 4.0% ± 2.4 percentage points, from a group that could not include most of the people who were working. Consistent with the official rate, but nowhere near as precise as you wrote. You override three “looks great” verdicts and rewrite the claim with its real uncertainty. The roles counted the answer as correct. You checked whether it was earned.
Agreement is worth something only when the agreeing parties can fail independently, which is why diversity of approach, not headcount, is what makes a group better than its members (Hong and Page 2004).
The block below builds the survey and computes the rate two defensible ways. Neither number is the one the reviewers blessed, and both come from the same afternoon outside the library.
import numpy as np, pandas as pd
SEED = 464
rng = np.random.default_rng(SEED)
# 250 adults asked outside the library on three weekday afternoons.
n = 250
employed_ft = rng.random(n) < 0.58 # daytime library sample skews non-working
wants_work = np.where(employed_ft, False, rng.random(n) < 0.24)
looking = wants_work & (rng.random(n) < 0.20) # only some are actively looking
labour_force = employed_ft.sum() + looking.sum()
your_rate = looking.sum() / labour_force
official_style = wants_work.sum() / (employed_ft.sum() + wants_work.sum())
print(f"respondents : {n}")
print(f"in your labour force : {labour_force}")
print(f"counted as unemployed : {looking.sum()}")
print(f"your rate : {your_rate*100:.1f}%")
print(f"same people, counting everyone who WANTS work: "
f"{official_style*100:.1f}%")
print("\ntwo defensible definitions, two different numbers, from one sample")
print("that never met anyone at work on a weekday afternoon. three reviewers")
print("who only checked whether the answer LOOKED right caught none of it")39.4 An AI failure case
You send your draft to four AI roles and all four reply “no major problems.” It feels like overwhelming confirmation, and it is confidently wrong. The four share a base model and read the same draft with the same instructions, so they share a blind spot. Whenever your one real flaw falls inside that blind spot, all four miss it together. Their unanimous “fine” is close to a single voice, not four. Running each loop longer does not rescue you either. Four loops with the same blind spot, given more turns, return the same verdict with better paragraphs around it.
You catch it by refusing to treat agreement as evidence. You run an independence check: re-read the flagged claim against your own inquiry declaration, or hand the draft to a role with a different framing and different context. In the lab you will see the size of this trap measured. Four correlated reviewers miss a real flaw about seven times as often as four independent ones, and piling on more correlated roles cannot close the gap.
39.5 It is your turn
You are working inside Studio 12: Special topic: agentic AI, release, and the next cycle. Keep what you write here; the studio’s milestone chapter is where it joins the other lessons’ pieces into one artifact you can defend.
Your project already has a small AI team running on one section; this step decides in advance where those loops must stop and hand the call back to you.
The hands-on half of this section lives in the chapter’s companion notebook: open it in Colab with the badge at the top, and work the steps there.
Commit your own diagnosis first, then delegate. Each prompt starts a loop, and with several loops in play the discipline is the same as with one: read the output, interrogate it, refine, run again, and never let a loop close a question that belongs to you. Each prompt is a checkable job, not a request for a verdict.
Which claims your draft can defend, where each claim must stop, whether an agreement among your roles was ever independent, and the moment to escalate: those stay yours. No role decides its own trustworthiness, and no head-count of roles decides your claim boundary or your ethics. When an AI output would settle one of those rather than inform it, you stop and you decide.
Write your escalation rules before the next round starts. List the specific decisions no loop may close: your claim boundary, your uncertainty, which sources are real, what counts as done, and anything with an ethical edge. Keep the list short enough to remember under pressure.
Locate the decisions that must escalate.
Act as a research-integrity assistant. Here is my draft's headline claim and the three never-delegate decisions I think it contains: [paste]. Using standard research-defense criteria, list which decisions in this draft an AI output should never be allowed to settle, and which of my three I mislabeled. Do not settle any of them for me.After running, verify: check its list against the escalation triggers you wrote yourself; a decision it omits is the one to guard hardest. Counters illusion of completeness (a tidy list that quietly drops the call that matters most).
Add the stopping conditions that are not about content. How many turns a loop may run before you read it, and what an agentic tool is allowed to change without asking you first.
Take one load-bearing pair of outputs your roles have returned, whether they agree or disagree, and diagnose it on the record: real disagreement, correlated error, genuinely independent agreement, or a consensus your own prompt manufactured. Say which and why.
Red-team the consensus.
Act as a hostile reviewer who believes my three roles only agreed because my prompt led them. Argue that their consensus is manufactured, name the leading words in my original ask, and give me a neutral rewrite. Do not reassure me.After running, verify: if it cannot find a single leading word, push back and demand the worst one, then rerun the review neutrally. Counters sycophantic agreement (praise that rehearses your ego, not your evidence).
Resolve it yourself with a check the loops cannot run: recompute the number, open the source, or compare against a figure that was never in any prompt. Write the resolution and the check together, because one without the other is just an opinion.
Take one place your roles agreed on something that mattered and test whether the agreement was independent. Rerun one of them with a different framing and different context, or verify the point yourself outside the tools.
List the agreements to verify.
Here are three reviewer outputs on the same draft: [paste all three]. For each pair, name one specific way they could be wrong the SAME way: a shared assumption, a shared anchor, or a shared prompt. Rank the pairs from most to least likely to share a blind spot.After running, verify: for the top pair, rerun one role with a different framing and different context and see whether the agreement survives. Counters correlated errors across tools or roles (agreement that is one flaw echoed back).
Log the pair you examined, your diagnosis, and the check that settled it in your AI Research Ledger, with the override on record when you made one, and verify at least one output with a named method from the Verification Guide. An AI reviewer may run the check with you; the decision to accept or reject stays yours.