How I Build an AI Answer You Can Actually Trust

There's been a ton of talk about AI governance lately, and honestly? Good. It should be everywhere. But most of what I read lives at 30,000 feet — principles, policies, frameworks, "responsible AI" statements. All useful. None of it answers the question I actually care about, which is much smaller and much more annoying: at the exact moment an AI gives an answer or takes an action, how do you know it's right?

I posted a short version of this the other day and ended it by saying I'd spare everyone the dissertation. A few people asked for the dissertation. So here it is — the longer, nerdier walk through how I actually build this stuff. Fair warning up front: some of what follows is running today, some of it I'm still wiring up, and every one of these ideas has more depth and more hard edges than a blog post can hold. This is the shape of it, not the whole of it.

One more thing before we start, and it's the part I'm a little smug about: this post is about a process for producing verified, grounded output — so I ran the post itself through that exact process before publishing it. It was drafted by one model, fact-checked against live authoritative sources, adversarially reviewed by three other models from different families, reconciled, and then approved by a human. The full write-up of how is at the bottom. It felt dishonest to preach it and not practice it.

The framing: one source vs. peer review

Here's the mental model I keep coming back to.

A single call to one model is a research paper written from one source. It might be right — but nobody checked it, nothing corroborates it, and you'd never hand that in. What I build is closer to a paper written from many sources, and then peer-reviewed before it ships.

That's the whole thing in two sentences, and the rest of this post is just me unpacking what "many sources" and "peer review" mean when you build them out of software instead of grad students.

The reason the single-call approach fails isn't that models are dumb. They're remarkably good. The problem is that a lone model answers with total confidence whether it's right or wrong, it has no idea what it doesn't know, and it can't tell you when the thing it "knows" quietly went out of date six months ago. Confidence is not accuracy. So the entire design goal is to stop trusting confidence and start requiring corroboration.

The corpus: assembling the sources

Everything starts with the corpus — the verified ground truth the AI checks itself against. This is the part people skip, and it's the part that matters most. A verifier is only as good as the truth you measure against.

For subjects I actually know cold — let's use infrastructure as the running example, because it's where I have the most scar tissue — I don't start from a model's guess. I start from my own words and experience. That's the seed. But my experience isn't automatically correct or current either, so it doesn't get a free pass. It gets corroborated.

That draft runs through several different models — Sonar, GPT, GLM, others depending on the job — each looking from a different angle. Is this accurate? Does it line up with best practices from authoritative sources? And critically: is it still current? Sonar is particularly good at that last one because it's grounded in live sources, so it catches the moment something I was sure of stopped being true.

And here's the rule that makes the whole thing work: the model that checks is never the model that wrote. A model is blind to its own mistakes the same way you miss your own typos — the same blind spots that produced the error also fail to catch it. GPT does not get to grade GPT's homework. You want independent reviewers with different training data and different failure modes, because the whole point is to catch the errors one model can't see in itself. Correlated errors are the enemy; independence is the strongest defense we have against them — it reduces the shared blind spots, it doesn't abolish them.

Only what survives all of that — my experience, cross-checked across independent models, verified accurate against authoritative sources, and verified current — earns a place in the corpus. Then it gets vectorized (turned into embeddings so it can be searched by meaning, not just keywords) and chunked into small, self-contained pieces. The chunking matters more than it sounds: when the AI is later generating an answer or executing a task, it doesn't reload a whole document, it pulls the two or three exact chunks that are relevant and checks itself against those. Small, precise, verifiable units of truth.

Keeping it alive

A corpus is a liability the day it goes stale. Best practices move, versions deprecate, the "right" way to do something in 2024 quietly becomes the wrong way. So the corpus isn't a snapshot — it's maintained.

There's a small worker running on a schedule up in Cloudflare whose only job is to re-check the corpus against the current state of the world — Sonar plus a couple of others — and flag anything that's drifted. Living sources, not a frozen library. This is one of the pieces I consider genuinely load-bearing: without it, everything downstream is slowly rotting and nobody notices until it bites.

Using it: grounded generation

When it's time to actually produce something, the model generates with the corpus, not from memory alone. It retrieves the relevant verified chunks and grounds its output in them — the technique people call retrieval-augmented generation. The difference in practice is night and day: instead of "here's what I vaguely recall about this," you get "here's the answer, anchored to these specific verified facts, and here's where they came from."

That grounding is also what makes the next step — the review — possible. You can't meaningfully check an answer against nothing. You can check it against the exact chunks it claimed to be built on.

The verifier: peer review, in software

This is the part I find the most fun, because it's where "be accurate" stops being a hope and becomes a mechanism.

Every output and every action goes in front of a verifier. And the verifier runs on the same principle as the corpus: the author never reviews its own work. The model that proposed the answer is not the model that approves it. A different model — different family, different blind spots — actively tries to poke holes in it, grounded against the corpus. The established term for the core of this is LLM-as-a-judge; the wider pattern is what practitioners call adversarial or cross-model verification. In plain terms: a second, independent set of eyes whose job is to find the problem, not to rubber-stamp.

For an action — say the system wants to make an infrastructure change — the review happens in three beats:

• Verify before. Does this proposed change even make sense? Is it grounded in the corpus? Does it match current best practice? What's the blast radius if it's wrong?
• The gate. Based on that review, the change either proceeds on its own or stops and waits for a human (more on the gate in a second).
• Verify after. If it did proceed, confirm it actually did what it was supposed to — the change landed, the thing that should be true is now true, nothing else broke.

The gate: when it's automatic, when I'm in the loop

The gate is the honest heart of the whole system, because it's an explicit admission that not everything should be automated. Here's roughly how it decides.

It goes automatically when all of these are true: the change is well-understood, low blast-radius, the verifier is confident, and it's fully grounded in the corpus. Think routine, reversible, boring. The kind of change I'd approve without a second thought if I were doing it by hand.

It stops and pulls me in the moment any of that isn't true: the change is ambiguous, high-risk, novel, hard to reverse, or the verifier can't confidently ground it. Anything with real consequences that the system can't prove is safe waits for a human. That's not a limitation I'm apologizing for — it's the design. The machine earns autonomy on the things it can prove, and asks for help on the things it can't. Autonomy is something you earn per-decision, not a switch you flip once.

Some checks aren't left to a model's judgment at all. Certain things — the genuinely non-negotiable stuff, compliance boundaries, hard safety limits — are enforced by deterministic gates, not probabilistic ones. A model can be persuaded; a hard rule can't. You want the important "no" to be a wall, not an opinion.

The part that closes the loop: contracts

Here's my favorite piece, because it makes the system self-governing instead of just self-checking.

Every component — every worker, every process — gets written up as a source-of-truth article, and each one carries a contract: what it takes in, what it's required to put out, how it's supposed to behave, what it must never do. That's not documentation rotting in a folder somewhere. Those contracts get folded back into the corpus.

Which means the loop closes on itself. When the AI uses a component, it checks its own actions against that component's contract — the spec becomes part of the ground truth it's measured against. And when I go change something by hand, I'm following the exact same contract. One shared spec, holding both the machine and me to the same standard. The methodology of one paper becomes a cited source for the next. The system documents how it's supposed to work, and then that documentation becomes the thing it's held to.

That's the property I was really chasing. Not "the AI is smart," but "the AI — and the human — are both accountable to the same written, verifiable truth, and that truth keeps itself current."

The honest limits

None of this is magic, and I'd be doing exactly the thing I dislike if I pretended it were.

Independent verifiers reduce correlated errors; they don't eliminate them — two models can still be wrong the same way, especially on genuinely ambiguous questions. Grounding is only as good as the corpus, and a corpus is only as good as the maintenance behind it. The gate's risk judgments are themselves fallible, which is exactly why the important stuff falls back to hard rules and humans rather than trusting the judgment. And "verified current" has a shelf life measured in the cadence of the refresh, not in real time. This is a system for making good outputs far more likely and bad ones far more likely to get caught — not a guarantee. Anyone selling you a guarantee is selling you the single-source paper with a nicer cover.

The jargon map

If you speak the language and want the terms mapped to the plain-English above: this is grounding and retrieval-augmented generation for the corpus; harnesses and quality gates for the control flow; LLM-as-a-judge with adversarial, cross-model verification for the review; deterministic guardrails for the non-negotiables; and a human-in-the-loop on the calls that actually matter. Every one of those is its own rabbit hole with real depth under it — I'm happy to go deep with anyone who genuinely wants to.

Why I bother

The governance conversation is mostly about writing the rules — and we should. But rules on paper don't enforce themselves. This is the plumbing that actually enforces them at runtime: accuracy, currency, and "don't do something dumb," checked by something other than the thing being checked, with a person on the decisions that carry weight.

A paper with real sources, peer-reviewed before it ships. Way less glamorous than the policy debates. Way more fun to build.

How this post was made — and checked

I'd be a hypocrite to write two thousand words about grounded, peer-reviewed AI output and then publish it straight from a single model's first draft. So I didn't. This post went through the same pipeline it describes, and here's the honest accounting of it — models, methods, and what actually changed as a result.

1. Generation — Claude Opus 4.8. The draft was written by one model (Opus 4.8). Per the rule in the post, that means Opus is the author and therefore disqualified from being its own reviewer. Its job ended at the draft.
2. Grounding — Perplexity Sonar. I pulled out the seven load-bearing technical claims — things like "model confidence isn't accuracy," "a different model catching another's errors reduces correlated failures," "retrieval-augmented generation reduces hallucination," and "human oversight belongs on high-risk decisions" — and ran each one through Sonar, which answers grounded in live sources rather than memory. The point was to check the claims against current authoritative work, not vibes. They held up, with citations to peer-reviewed research (for example the EMNLP work showing retrieval cuts hallucination by roughly 60%, and the "Correlated Errors in Large Language Models" paper), and to the NIST AI Risk Management Framework, whose GOVERN and MEASURE functions explicitly call for defined human-oversight configurations on AI decisions — which is the formal backing for the whole "human-in-the-loop on the calls that matter" section.
3. Adversarial review — three other models, none of which wrote it. The draft then went in front of three independent judges from different model families — Meta's Llama 3.3 70B, Mistral Small 3.1, and Meta's Llama 4 Scout — each explicitly instructed to refute the post: find every factual error, misused term, and overclaim, and to leave properly-hedged claims alone. Different training data, different blind spots, and critically, not the author. This ran on Cloudflare Workers AI at the edge.
4. Reconciliation. I read every flag and kept the ones that survived cross-checking against the grounded sources, discarding the ones the sources contradicted. Two real corrections came out of it and are in the text you just read: I stopped calling "adversarial verification" and "cross-model verification" formal terms of the literature (only LLM-as-a-judge is — the rest is practitioner language), and I softened the line about independence "curing" correlated errors to say it reduces them, because the research is clear that two models can still fail the same way. The judges also flagged things I kept — e.g. one insisted "deterministic guardrails" wasn't a real term; Sonar's grounded sources said otherwise, so it stayed. That's the reconciliation step doing its job: the verifier proposes, the grounded truth disposes.
5. Compliance gate — automated, deterministic. The post also ran through my content pipeline's legal/compliance gate: a hard-coded denylist for the things you can never say — outcome guarantees, "risk-free," absolute-safety claims — plus a corpus-grounded advisory pass against the relevant rules. It came back clean: zero blockers, zero flags. Fittingly, that gate is one of the deterministic guardrails the post describes — a hard "no" that isn't a model's opinion.
6. Human in the loop — me. Nothing here published itself. I made the final call on every flag and every edit, and hit publish myself. The last gate is a person. As designed.

That's five models, a live-grounded fact-check, a deterministic compliance gate, and one human, to ship a blog post. Overkill? For a blog post, sure. But that's the point — the machinery is the same whether the output is a paragraph or an infrastructure change, and the only way I know it works is to run it on things I can check by hand. This was one of those things.