Milestone 8: Your robustness audit
This milestone chapter is part of a book in active development and has not yet been through the author’s review. Content may change as the review advances.
This chapter closes Studio 8: Stress-test and adjudicate. Its lessons each ended at It is your turn; what you wrote there arrives here as working material, and this is where the pieces become one artifact you can defend.
What this milestone produces. A pre-listed robustness grid, negative tests with their assumptions stated, diagnostics, an adversarial-review record, and your adjudication of what survived.
What you bring
You also carry forward the last milestone’s artifact: Milestone 7: Your first reproducible analysis. This one builds on it, and the chain assumes it is versioned and filed.
Check you are carrying each of these before you start. If one is missing, go back and work that lesson’s It is your turn first; the practice below assumes it.
- Lesson 24 — Robustness and Sensitivity: your pre-listed robustness grid with its full results, direction and size, plus the named null check.
- Lesson 25 — Diagnostics and Negative Tests: your negative test and diagnostic, each with the prediction you wrote before running it.
- Lesson 26 — AI as Adversarial Reviewer: your commissioned adversarial review with every flag adjudicated against the data, confirmed or refuted.
- Lesson 27 — Recognizing False Confidence: your confidence audit: the not-yet-recomputed list, where belief in the headline number comes from, and your boundary sentence.
The practice
- Pre-list your checks before you run any of them, and commit to reporting all of them.
- Run the robustness grid and report a span within each commensurable panel, never across panels.
- Run the null check that your design licenses, and name it correctly.
- Commission at least one adversarial review, and add a human reviewer when one is available. Neither review is independent verification: confirm or refute every flag with a data check.
- Write the adjudication: what survived, what did not, and what you still cannot rule out.
A version, not a pass
Your milestone artifact is a dated, numbered version with the reason for the version attached. When later evidence changes it, you write the next version rather than editing the last one, because the sequence of changes is itself part of your research record.
The four rails, here
Four concerns cross every studio in this book. At this milestone they take these specific forms.
- Ethics, permissions, and data exposure. Report the checks that hurt your finding as fully as the ones that helped.
- Evidence, provenance, and reproducibility. A flag is real when a check confirms it, not when the reviewer sounds certain.
- AI activity, verification, and human decisions. An AI reviewer is another critique, not independent verification.
- Uncertainty, claim boundary, and revision history. Specification spread measures your choices; it is not an uncertainty interval.
Where this milestone sits
You arrived carrying Milestone 7: Your first reproducible analysis. Milestone 9 writes down only the claims this audit licenses, at the strength it licenses them. The chain ends at Milestone 12: Your release and next cycle, where you decide whether your finished research artifact leaves your hands.
What sends you back
A finding that does not survive its own pre-listed checks returns to Studio 5 or 7, not to the writing studio.
How this milestone is assessed
Each row scores 0, 1, or 2. 10 points total.
| # | Criterion | 0 | 1 | 2 |
|---|---|---|---|---|
| 1 | The milestone artifact exists in full, specific to your own project | absent, or generic text that would fit any project | present but incomplete, or specific in some parts and generic in others | complete and unmistakably about your project |
| 2 | The milestone is written as a dated, numbered version with its reason | no version, or a version with no reason given | versioned, but the reason is decorative rather than an account of what changed | versioned with a reason a reader could use to reconstruct your thinking |
| 3 | Every judgment in the artifact is defended by you, not asserted by a tool | conclusions appear with no reasoning, or reproduce a tool’s wording | reasoning present but thin where the decision was hardest | each judgment carries a reason you could defend under questioning |
| 4 | The four rails are carried in the form this studio requires | one or more rails absent where this studio required them | all rails present but one is nominal | every rail addressed in the form this studio requires |
| 5 | The milestone’s own product is present and defensible: A pre-listed robustness grid, negative tests with their assumptions stated, diagnostics, an adversarial-review record, and your adjudication of what survived | the milestone’s own product is missing | present but would not survive a careful question | present and defensible as stated |
No work at or after Studio 4 proceeds past a not-authorized determination. This is not scored and cannot be averaged away.
Your workbook
Open the workbook with the badge above. It walks these steps with a cell for each one, ends by writing your milestone version with its reason, and adds the rows this studio contributes to your AI Research Ledger and your Research Dossier.