Review-Gated AI: How to Ship Agents That Never Silently Fail
The failure mode nobody talks about is not the dramatic hallucination. It is the plausible-sounding output that ships quietly, gets cached, and becomes load-bearing before anyone notices it is wrong. Every AI-assisted project in this portfolio is built around a single rule to prevent that: agents write to review queues, humans promote, and nothing ever publishes itself.
Why "just ship the AI output" breaks in production
Large language models are fluent. That fluency is the risk. A model that hedges visibly is easy to catch. A model that produces a confident, well-structured paragraph about a 19th-century Hawaiian chieftain, citing a plausible-sounding source that does not exist, is much harder to catch, especially when it arrives inside a pipeline that runs at 2 a.m. and writes to a database that feeds a public page.
Three failure modes appear repeatedly across AI-assisted content systems. First, hallucination without grounding: the model produces a claim that has no traceable source, and the pipeline treats it identically to a claim that does. Second, provenance loss: by the time a fact appears on a webpage, the chain of evidence that justified it has been discarded, so there is no way to audit or retract it. Third, silent errors: a malformed API response, a rights conflict, or an image that depicts the wrong subject gets passed downstream without any human ever seeing it, because the pipeline was designed to keep moving.
Each of these is solvable. The solution in every case follows the same shape: insert a human gate between what the agent produces and what the system ships.
Review queues, evidence gates, human promotion
A review-gated pipeline has three components. First, a review queue: instead of writing to the production data store, agents write to a staging area. The schema is identical, but the row carries a status field that prevents rendering. Second, an evidence gate: before anything can enter the queue, it must pass a verifiability check. For narration, that means substring-grounding against the source text. For structured data, it means citation traces to a real record. For images, it means a ledger entry confirming the subject matches. Third, human promotion: a reviewer sees the item, its evidence, and its provenance chain, then either promotes it to production or rejects it with a reason that gets written back to the ledger.
The ledger is the part most systems skip, and it is the part that compounds in value over time. When a reviewer rejects an item, that rejection is not just a signal to discard the output. It is training data for the queue itself. The next time the same agent encounters a similar input, the ledger can surface the prior rejection and its reason before the agent wastes compute on an output that will fail the same check.
Four systems, four domains, one pattern
The pattern is not domain-specific. It applies equally to a travel content pipeline, a museum data aggregator, a historical research tool, and a multi-agent operations platform. Each implementation below illustrates how the same pattern adapts to a different domain.
Named agents with explicit roles
The Big Aloha Guide pipeline uses four Hawaiian-named agents, each with a bounded responsibility. Kiaʻi (guardian) validates. Kilo (scout) observes and reports. Nīnau (questioner) researches. Haku (composer) writes. No agent does anything outside its role, and no agent is allowed to promote its own output.
Haku produces narrations using Claude Haiku. Every narration, across 2,220 total, is substring-grounded against its source material before it enters the review queue. If the narration contains a claim that cannot be located as a substring in the source text, it is rejected at the evidence gate before a human ever sees it. The human reviewer only sees narrations that have already passed mechanical grounding. That means reviewer attention goes to judgment calls, not fact-checking.
The image ledger is the other structural piece. When a reviewer rejects an image because it depicts the wrong subject, that rejection is written to a selected/rejected ledger keyed on the image URL and the subject. The next time Kilo surfaces the same image for a different article, the ledger flags it before any work begins. Wrong-subject images are remembered, not rediscovered.
Evidence-gated publication across 13 APIs
MetaMuseum reconciles structured data from 13 museum APIs into a unified knowledge graph. The reconciliation agent, Janus, handles cross-source merging. Themis handles rights and provenance. Calliope drafts curatorial copy. The rule is the same across all three: agents suggest, humans review and approve before anything ships.
The structural guarantee here is SHACL validation in CI. Every piece of agent-produced data that claims to be CIDOC-CRM-compatible JSON-LD must expand to valid RDF and pass the SHACL shapes before it can enter the review queue. This is a second gate that runs before the human gate. A rights conflict or a malformed provenance chain fails SHACL before a curator ever sees the record. The human reviewer sees only records that are structurally sound, so their attention goes to intellectual judgment about whether the reconciliation is correct, not whether the data is well-formed.
Citation verification as a first-class constraint
MetaHistoryBook generates AI dossiers on historical figures, pulling from Wikidata, Wikipedia, and Wikisource. The "From Wikidata" badge on each dossier is not decorative. It is a constraint enforced in the pipeline: every claim in a dossier must trace to a real Wikidata fact or an attributed quote from Wikipedia or Wikisource.
The citation gate works at the claim level, not the document level. The agent is required to produce structured output that pairs each sentence with its source reference. The pipeline verifies that each reference resolves to a real record with the claimed content before the dossier is eligible for review. A dossier that contains a single unverifiable claim does not enter the queue at all. The human reviewer is never in a position of having to check whether a source exists. They are only deciding whether the AI's interpretation of a real source is accurate and appropriate.
Adversarial agent and hard merge boundaries
MetaGen Auto is a multi-agent operations platform where the agents are not content producers but code contributors. The review-gate pattern takes a different form here because the output is pull requests, not narrations or dossiers.
The structural rule is that the AI never merges its own code. Every deliberation the system runs ends at a human gate that files a real ticket. The agent can propose, plan, and draft. It cannot ship. Additionally, a dedicated agent named Glaucon has a single job: before any decision is finalized, Glaucon steelmans the rejected option. The goal is to force the pipeline to surface the strongest version of the alternative before a human approves the chosen path. That means the human reviewer is not approving an AI recommendation in isolation. They are approving it after seeing its best counterargument.
The auth layer enforces the same philosophy at the infrastructure level. A 3-tier RBAC model with deny-by-default boundaries means the agents operate in a least-privilege environment. An agent cannot escalate its own permissions. The boundary between what the system can do autonomously and what requires human action is enforced at the authorization layer, not just at the application layer.
Rejected decisions are not waste, they are memory
Most AI pipelines treat rejected outputs as discarded noise. The item fails a check or a reviewer rejects it, and the system moves on. That is a missed opportunity. A rejection carries structured information: what was attempted, why it failed, and what context surrounded the attempt. If that information is written to a persistent ledger and made available to the agents that produced the rejected output, the system stops making the same class of mistake twice.
In the Big Aloha Guide image pipeline, the ledger is keyed on image URL and subject. An image of Kīlauea that gets rejected because it depicts the wrong crater is remembered at the URL level. If the same image surfaces again in a different article about a different location, the ledger fires before Kilo invests any work in evaluating it. The rejection from six months ago prevents a redundant mistake today.
The ledger pattern generalizes. A rights conflict that Themis catches in the MetaMuseum pipeline can be written to a ledger keyed on the museum's object identifier. Future reconciliation passes for the same object will start with the knowledge that a prior rights claim was contested. A dossier claim that fails citation verification in MetaHistoryBook can be written to a ledger keyed on the Wikidata entity and the claim type. If the same entity is reprocessed after a Wikidata update, the pipeline knows which claim types have historically been problematic for that entity.
This is not fine-tuning. It is not retrieval-augmented generation. It is simpler than either: a structured record of prior failures that the pipeline can consult before repeating them.
What the pattern looks like in architecture
The review-gate pattern has five components that appear across every implementation in this portfolio, regardless of domain.
- Staging schema with status field. Agent output writes to a staging table or collection with a status column set to pending. The production rendering layer filters on status = approved. Nothing the agent writes is ever visible until a human changes that status.
- Mechanical gate before human gate. Automated checks run before a human ever sees an item. Substring grounding for narration, SHACL validation for structured data, citation resolution for research claims. Items that fail mechanical checks never consume reviewer attention.
- Evidence surface in the review UI. The reviewer sees the item alongside its evidence chain. Not just the output, but the source it was grounded against, the references it cites, the provenance it claims. The reviewer is making a judgment call, not a fact-checking pass.
- Rejection writes to ledger. When a reviewer rejects an item, the rejection is written to a persistent ledger with the reason, the item key, and the context. That ledger is available to agents in subsequent runs.
- Agents never promote their own output. This is the hard rule that every system above enforces. The agent that produces an output is not the agent that approves it. The human is the only entity with promotion authority. This is not a workflow preference. It is an architectural constraint enforced at the permission level.
The result is a system where the AI's contribution is real and substantial, but where the failure mode is always visible before it ships. The agent is not a black box that occasionally produces wrong answers. It is a collaborator that produces candidates, and every candidate has a traceable evidence chain that a human can inspect before making the call to publish.
The goal is not to distrust the AI. It is to build systems where trust is earned incrementally, and where the cost of being wrong is paid at review time rather than in production.