Restricted Content

This chapter is password-protected. Please enter the password to continue.



← Back to guide overview

6  The System in Action

With the underlying building blocks, configuration layers, and agent definitions covered in Chapter 5, here is how the system executes in dynamic action. This chapter walks through the core mechanisms that make the runtime workflow powerful: specialized multi-agent review, adversarial QA loops, the orchestrator pattern, and automatic quality scoring.

6.1 Why Specialized Agents Beat One-Size-Fits-All

Consider proofreading a 140-slide lecture deck. You could ask Gemini:

“Review these slides for grammar, layout, math correctness, code quality, and pedagogical flow.”

Gemini will skim everything and catch some issues. But it will miss:

  • The equation on slide 42 where a subscript changed from \(m_t^{d=0}\) to \(m_t^0\)
  • The TikZ diagram where two labels overlap at presentation resolution
  • The R script that uses k=10 covariates but the slide says k=5

Now compare with specialized agents:

Agent Focus What It Catches
proofreader Grammar only “principle” vs “principal”
slide-auditor Layout only Text overflow on slide 37
pedagogy-reviewer Flow only Missing framing sentence before Theorem 3.1
r-reviewer Code only Missing set.seed()
domain-reviewer Substance Slide says 10,000 MC reps, code runs 1,000

Each agent reads the same file but examines a different dimension with full attention. The slide-excellence skill runs them all in parallel.

6.2 The Adversarial Pattern: Critic + Fixer

The single most powerful pattern in this system is the adversarial QA loop:

+------------------+
|  quarto-critic   |  "I found 12 issues. 3 Critical."
|  (READ-ONLY)     |
+--------+---------+
         |
    +----v----+
    | Verdict |
    +----+----+
     /       \
APPROVED   NEEDS WORK
    |          |
  Done    +----v---------+
          | quarto-fixer |  "Fixed 12/12 issues."
          | (READ-WRITE) |
          +----+---------+
               |
          +----v----------+
          | quarto-critic |  "Re-audit: 2 remaining."
          | (Round 2)     |
          +----+----------+
               |
          ... (up to 5 rounds)

Why it works: The critic can’t fix files (read-only), so it has no incentive to downplay issues. The fixer can’t approve itself (the critic re-audits). This prevents the common failure of Gemini saying “looks good” about its own work.

NoteClosing the visual loop with Native Multimodal Vision

The critic-fixer loop above is text-only — it compares LaTeX source to Quarto source. It can’t see the rendered slides. Google’s Multimodal Vision capabilities let Gemini accept rendered slide images (such as PDF slides converted to PNGs) directly into the agent context to verify layouts visually. If you want closed-loop visual QA — “does the rendered slide actually look right?” — this multimodal capability closes the gap.

Optional, not required: qa-quarto and visual-audit already catch most issues without it. Treat visual verification as an extra rung when text-level audits aren’t enough (e.g., suspecting a TikZ render glitch that the source code looks fine for).

6.3 The Orchestrator: A Pattern, Not a Daemon

Individual agents are specialists. Skills like slide-excellence and qa-quarto coordinate a few agents for specific tasks. In day-to-day work, you don’t have to think about which agents to run — the right skill runs the right agents for you.

The orchestrator protocol (.agents/rules/orchestrator-protocol.md) defines the 6-step loop (IMPLEMENT → VERIFY → REVIEW → FIX → RE-VERIFY → SCORE) that skills implement internally. It is not a runtime daemon: plan approval does NOT auto-trigger this loop. You invoke a skill (e.g., create-lecture, qa-quarto, review-paper --adversarial) and the skill runs the pattern within its own scope.

What’s mechanically implemented today: commit (verifier + quality_score), qa-quarto (critic-fixer loop), review-paper --adversarial (critic-fixer loop), slide-excellence (multi-agent fanout), review-paper --peer (editor + 2 referees + cross-artifact). See Pattern 2 for the complete workflow.

6.4 Quality Review: The 80/90/95 System

The quality-gates rule (quality-gates.md) defines scoring thresholds that commit and review skills apply. Thresholds are advisory — enforced inside specific skills, not by a repo-wide git pre-commit hook. A direct git commit bypasses the review. Every file gets a quality score from 0 to 100:

Score Threshold Meaning Action
80+ Commit Safe to save progress git commit allowed
90+ PR Ready for deployment gh pr create encouraged
95+ Excellence Exceptional quality Aspirational target
< 80 Blocked Critical issues exist Must fix before committing

6.4.1 How Scores Are Calculated

Points are deducted for issues:

Issue Deduction Why Critical
Equation overflow -20 Math cut off = unusable
Broken citation -15 Academic integrity
Equation typo -10 Teaches wrong content
Text overflow -5 Content cut off
Label overlap -5 Diagram illegible
Notation inconsistency -3 Student confusion

6.4.2 Mandatory Verification

The verification protocol (.agents/rules/verification-protocol.md) requires that Gemini compile, render, or otherwise verify every output before reporting a task as complete. Skills that implement the orchestrator pattern enforce this as Step 2: VERIFY. This means Gemini cannot say “done” from within those skills without actually checking the output.

WarningDon’t Skip Verification

Verification catches errors that might silently deploy, like broken diagrams or missing intercepts in scripts.

NoteModel quality can regress — verification is the only durable defence

AI providers regularly update model checkpoints and release system changes. These updates can sometimes lead to model-quality drift or subtle regressions on highly specialized tasks. Implication for our workflow: do not treat any given model checkpoint as a stable baseline.

The defences in this template all assume model quality drifts:

  • verify-claims with the Chain-of-Verification forked-verifier (the verifier cannot self-confirm even if the orchestrator’s checkpoint regressed).
  • audit-reproducibility with passport.yaml (numeric claims are anchored to specific script output values, not to the model’s recall of the values).
  • The cross-artifact review rule (a paper’s claims are checked against the code that produced them, not against the model’s intuition about whether they’re “plausible”).
  • HIGH-WARN gate-refuse on commit (Pass 3A I) — a fabricated citation or numerical contradiction blocks the commit even if the model “feels confident.”

This is also a useful reminder that running review-paper --variance N (Pass 2C E) is the empirical answer to “should I trust this output?” — a single point estimate of model quality on a single task hides variance that the template’s adversarial-review patterns surface.

6.4.3 Submission-readiness: stacking the verification lenses

For a paper headed to a journal, three orthogonal lenses run together — each catches a different class of failure:

Lens Skill What it catches Block-commit on failure?
Grammar / overflow proofread typos, search-and-replace artifacts, overfull \hbox, citation-format inconsistency No (advisory)
AI-voice tells (v1.9.0) humanize boilerplate transitions (“Moreover”, “It is important to note”), AI-cliché lexicon (“delve”, “navigate the complexities”), hedging stacking, sycophancy No (advisory; author edits manually)
Factual claims verify-claims fabricated citations, numerical contradictions, directional contradictions Yes — HIGH-WARN gate-refuses (v1.9.0)
Numeric provenance audit-reproducibility manuscript value ≠ script output value within tolerance Yes via passport.yaml (v1.9.0)

These are complementary, not redundant. A paper can pass proofread (clean grammar) and fail humanize (the prose reads as AI-drafted). It can pass humanize (your own voice) and fail verify-claims (citation fabricated). It can pass verify-claims (citations real) and fail audit-reproducibility (Table 2 doesn’t match the code). Run all four before submission; reach for review-paper --variance N for the simulated peer-review verdict on top.

humanize design choice: detect-only, no --rewrite mode. Auto-rewriting AI tells degrades prose quality (cross-vendor research finding) and introduces new tells. The author reads the report and edits — that manual step is the price of preserving voice. If the report flags 8+ HIGH-severity tells per 1000 words, rewrite the affected paragraph from scratch rather than patching tell-by-tell.

6.5 Creating Your Own Domain Reviewer

The template includes domain-reviewer.md — a skeleton for building a substance reviewer specific to your field.

6.5.1 The 5-Lens Framework

Every domain can benefit from these five review lenses:

Lens What It Checks Example (Economics) Example (Political Science) Example (Physics)
Assumption Audit Are stated assumptions sufficient? Is overlap required for ATT? Is ignorability defensible given the observed covariates? Is the adiabatic approximation valid here?
Derivation Check Does the math check out? Do decomposition terms sum? Do conjoint AMCEs identify under Hainmueller–Hopkins–Yamamoto assumptions? Do the units balance?
Citation Fidelity Do slides match cited papers? Is the theorem from the right paper? Is the manipulation-check threshold cited from the original validation study? Is the experimental setup correctly described?
Code-Theory Alignment Does code implement the formula? R script matches the slide equation? cjoint/survey::svyglm weights match the design? Simulation parameters match theory?
Logic Chain Does the reasoning flow? Can a PhD student follow backwards? Does the causal claim survive the standard counterfactual challenge? Are prerequisites established?

The template ships two concrete domain-reviewer customizations in .agents/agents/domain-reviewer.md: an econometrics example (assumptions, identification, R/Stata code-theory alignment) and a political-science example (ignorability, conjoint AMCEs, cjoint/survey package defaults). Both follow the 5-lens structure; either is a viable starting point for your own field.

To customize, open .agents/agents/domain-reviewer.md and fill in:

  1. Your domain’s common assumption types
  2. Typical derivation patterns to verify
  3. Key papers and their correct attributions
  4. Code-theory alignment checks for your tools
  5. Logic chain requirements for your audience