40 Final Research and AI-Management Portfolio
This chapter is part of a book in active development and has not yet been through the author’s review. Content may change as the review advances.
The research decision. Out of everything your project produced, you decide which claims you will stand behind in public, and you assemble the record that shows how you ran your AI team, what you kept in your own hands, and how each surviving claim was checked.
40.1 Why this decision matters
The decision on the table: which of your claims you are willing to be wrong about in public, and what record you can show for how you checked them.
“By the time you stand up to defend, I do not count your tools. I ask which claims you are willing to be wrong about in public, and whether you can show me you checked them yourself.” — a defense examiner, opening the folder you handed over
You reach this chapter with a full project behind you and one thing left to do: gather it into something you can defend out loud. Months of AI-assisted work leave a trail of drafts, role outputs, and half-finished checks. Left in a pile, none of it is defensible. The examiner will not read the pile. She will ask which claims you own, where each stops, and how you know. This chapter is where you decide those answers on purpose, before the room decides them for you.
40.2 The concept
Your final submission has two halves, and this chapter builds both.
This chapter’s real subject is the release audit: the last check you run before work leaves your hands, asking whether every promise the work makes is one the evidence, the permissions, and the record can still support. What you assemble for that audit is your dossier, and how you package it depends on where the work is going — a repository, a supervisor, a journal, a course. The audit does not change with the destination.
A research portfolio is the assembled body of your finished work: your claims, the evidence behind them, and the verification that makes each one defensible (ALLEA – All European Academies 2023). Example: for a lab project it holds your headline result, the measurements behind it, and the check you ran to confirm the number is right.
Alongside it sits an AI-management portfolio, the honest record of how you ran your AI work from the first curiosity to the last check — one assistant or a directed team, whichever your project actually used: which tasks you delegated, where you took the decision back, and which calls you never handed to a tool. Example: it shows a reviewer role drafted your limitations, that you overrode it when it softened a claim you could not support, and that you alone chose the claim boundary.
The decision that closes both is your stopping rule, a written statement of what “done” means, phrased around your own verified confidence rather than the tools’ agreement (Nosek et al. 2018). Example: “I stop when I can defend each surviving claim to a hostile reviewer with a check I ran myself (National Academies of Sciences, Engineering, and Medicine 2019).” A stopping rule of “when the roles all agree” is a trap, because a team of look-alike tools can agree loudly and be wrong together.
That trap has a name you will guard against: false consensus, agreement that exists only because the tools shared a blind spot or you asked a leading question. Example: you ask three reviewers to “confirm my draft is ready,” and all three oblige, so the chorus tells you nothing. The cure is an independence check, a step that makes agreement worth more than a head count: a check you run yourself, outside the tools.
The whole thing ends in an evidence defense, a short public defense of your claims at their boundaries, followed by questions you cannot rehearse for. Example: you state what you found, name exactly what you do not claim, and answer each question from your own record.
40.3 A worked example
Picture a small plant-biology project. You tested how many seeds from one packet of lettuce seed actually sprout, using a germination test, a standard method where you lay a counted number of seeds on damp paper under steady warmth and light, then count how many have sprouted after a fixed number of days. Three trays of 50 seeds each gave an average of 88% germination. Now you choose the claim you will defend.
You run three AI reviewer roles on your draft. A reviewer role praises your headline sentence, “Lettuce seed germinates at 88%.” A diagnostician role flags the same sentence for reaching past your evidence: you tested one packet in three trays, not lettuce seed in general. That disagreement is a gift, because it points straight at the boundary decision. You take the pen back and rewrite the claim to what your data hold: “This packet, tested in three trays of 50, averaged 88% germination, with the trays spanning 84% to 92%.” The narrower sentence is the one you will defend.
Then you run an independence check the tools cannot fake. The packet’s own label states a germination rate of 85%, so you compare your 88% against that value by hand. The two numbers land close, which is real reassurance because the label was never part of your AI workflow. Your stopping rule is now satisfied, so you stop. Into the AI-management portfolio go four things: the roles you ran, the reviewer-versus-diagnostician conflict and your override, this stopping rule, and the label comparison that made your reviewers’ agreement worth more than a head count.
Reporting what your evidence holds, with the spread that produced it, is the reproducibility standard this whole book has been building toward (National Academies of Sciences, Engineering, and Medicine 2019).
The block below runs the germination test tray by tray. The spread it prints is what the narrower claim is built from.
import numpy as np, pandas as pd
SEED = 464
rng = np.random.default_rng(SEED)
# Three trays of 50 seeds from ONE packet. Trays differ; seeds within a tray
# share their tray's conditions.
trays = []
for tray, sprouts in enumerate([42, 44, 46], start=1): # 50 seeds per tray
seeds = rng.permutation(np.r_[np.ones(sprouts), np.zeros(50 - sprouts)])
trays.append({"tray": tray, "sprouted": int(seeds.sum()),
"rate %": round(seeds.mean() * 100)})
df = pd.DataFrame(trays)
print(df.to_string(index=False))
print(f"\naverage across the three trays : {df['rate %'].mean():.0f}%")
print(f"tray-to-tray spread : {df['rate %'].min():.0f}% to "
f"{df['rate %'].max():.0f}%")
print("\nthe defensible sentence carries all three numbers: this packet,")
print("three trays of 50, and the spread — not 'lettuce seed germinates at 88%'")40.4 An AI failure case
You send your finished draft to three reviewer roles and ask each to “confirm it is ready to submit.” All three return the same verdict: “Looks solid, no major problems.” It feels like three confirmations. It is one. The three roles run on the same base model, you gave them the same leading prompt, and they share a blind spot: none flags that your headline number carries no uncertainty, no range across your three trays. That is false consensus, and trusting it would send an overclaimed result into your defense.
You catch it with a reframe and a check the tools cannot see. First you drop the leading ask and run a neutral one: “List the three weakest claims in this draft and why.” Now a role names the missing range. Then you leave the tools entirely and recompute the spread of your trays yourself, which hands you the 84% to 92% boundary the reviewers all skipped. The lesson is blunt: agreement is worth only the independence behind it. Count the checks you ran yourself, never the roles that nodded.
40.5 It is your turn
You are working inside Studio 12: Special topic: agentic AI, release, and the next cycle. Keep what you write here; the studio’s milestone chapter is where it joins the other lessons’ pieces into one artifact you can defend.
Your project already has a note, a capsule, and a record of the AI work you directed — loop wiring and escalation rules if you ran a team, a verified ledger if you ran one assistant; this last step assembles all of it into the artifact you hand over and the defense you give.
The hands-on half of this section lives in the chapter’s companion notebook: open it in Colab with the badge at the top, and work the steps there.
Commit your own answer first, then delegate. Each prompt hands the tool reach, not the decision, and each one opens a loop rather than ending in a single answer: you read what comes back, push on the weakest part, and run it again. Closing the loop is your move, not the tool’s. The honest minimum mode is one ordinary assistant playing the review roles in turn; record the roles as you actually ran them.
The tools may draft, review, and surface hard questions. They may never decide which claims you put your name on, where a claim like “88% germination” must stop, whether your reviewers were ever independent, what your stopping rule is, or a single word of the public defense. When agreeing roles bless a claim, deciding whether that agreement is real or just correlated is yours. So is the answer you give when the room asks how you know.
Finish the research artifact itself: your paper, note, or poster, with each claim stated at its boundary and your uncertainty in the same sentence as your headline number. This is the document with your name on it.
Assemble your AI-management portfolio around the team you directed in this studio: the decomposition of loops with their wiring, one conflict or suspicious agreement settled by your own non-AI check, your stopping rule and never-delegate list, and one independence check that made an agreement worth more than a head count. If a single assistant played the roles in turn, document the roles as you ran them; record what actually happened, never invented activity.
Locate the real benchmark.
Act as a research assistant in plant biology. I measured the germination rate of one packet of lettuce seed. I want a published or label-stated reference value to compare against. Give the value, the exact source it comes from, and a link. Only state a figure you are confident is real; if you are unsure, say so and tell me where to look.After running, verify: open the source or read the packet yourself before you trust the number (primary-source reading). Counters confident fabrication, a reference value quoted with an invented source.
Write your stopping rule as a sentence you could read aloud, phrased around what you have verified rather than what your tools approved. “Every surviving claim has a check I ran myself” is a stopping rule. “The reviewers agreed” is not. Beside it, write your next-study agenda: the question this work opened, tied to the limitation you could not resolve.
Rehearse the defense in three movements: your claim and its boundary, the central choice you made and the road you did not take, and the verification you lead with. Then have someone ask you the question you least want to be asked.
Red-team the defense.
Act as a hostile but fair cross-examiner at a research defense. Here is my headline claim, its boundary, and my two weakest verification steps: [paste]. Ask me the five hardest questions you can, each tied to a specific claim or check. Do not soften them, and do not answer them for me.After running, verify: if every question is gentle or your claim is called “well-supported,” push back and demand the weakest link. Counters sycophantic agreement, a partner that rehearses your ego instead of your defense.
Read the whole artifact once more against your ledger and cut anything you cannot trace to a check. What is left is what you can defend.
List every claim to verify.
Here is my draft: [paste]. List every empirical claim in it as a table: the claim, the number or measurement it rests on, and whether that claim stays inside my single sample or reaches past it. Do not add any new claims.After running, verify: check the table against your own list of what you measured, and confirm nothing reaches past your data unmarked. Counters illusion of completeness, a tidy list that omits the one claim quietly overreaching.
Close the ledger with the last entry, and verify your headline claim one final time with a named method from the Verification Guide. An AI reviewer may run the check with you; the decision to accept or reject stays yours.
That is the whole book. You started with a curiosity you could not yet answer, and you now have a finished piece of research, a record of every tool that touched it, and a set of claims you can defend at their edges. The tools were fast. The judgment was yours the entire way, and that is the part that makes it your work.
Milestone next. This was the last lesson of Studio 12. Milestone 12: Your release and next cycle is where the lessons’ pieces become the studio’s versioned artifact. Produce it before you move on.