A Classifier That Shows Its Work

Here is a problem that sounds trivial and is not: take the name of a business — just the name, nothing else — and hand back the correct industry classification code (NAICS) for it. No address, no tax ID, no paperwork. A person types a name into a box; the system returns the right code, ranked, and shows why it believes what it believes.

The naive version of this is one model and a hopeful prompt. You type a name, the model free-associates an answer, and it does so with precisely the same confidence whether it is right or catastrophically wrong. I have written before about why that is not good enough — confidence is not accuracy — so I will spare you the sermon and show you the machine instead.

One design decision up front, because it drives the rest: it classifies on the name, deliberately, and never on the address. An address is a trap. A business may not have moved into a location yet, and a single address tells you nothing about which of several identically named companies you are actually looking at. The name is the question; the address is a bad shortcut that feels like an answer.

The wrong turn I took first

The first version got one thing wrong, and it is worth admitting, because the fix is the whole point. The tool serves a defined footprint — a two-state service area — so the obvious instinct was to narrow the search to that region and let everything else fall away. For local businesses, it worked flawlessly.

Then the big national names went in, and the thing fell over. A national company is, by definition, headquartered somewhere else. Ask "is this business in these two states?" of a filter built around a headquarters, and it reads a household-name chain as out-of-area and throws it out — even though that chain plainly operates on half the corners in the region. The small, local shops sailed through; the giants came back "not in the area," which is exactly backward.

So I pulled geography out of the gate entirely. The footprint is not a wall; it is a tie-breaker, and only when a name genuinely belongs to two different companies in two different places. Where a business keeps its headquarters is never allowed to disqualify it.

What it actually queries

Before any model is allowed to have an opinion, the system goes and gets evidence. It fans out, in parallel, across a stack of public data sources: securities filings, federal spending and contract records that carry self-reported industry codes, national registries for licensed professions, open geographic and facility data, and public care-facility datasets. Each source runs isolated from the others, so a slow or dead one returns empty rather than throwing an exception that takes the whole request down with it. A failed lookup is a shrug, not a crash.

On top of those structured sources sits a live web-reasoning layer — Perplexity's Sonar — which I run to answer a narrower question than "search the web": what does this specific business actually do, what industry is it in, and does this name, in fact, belong to more than one distinct company? Sonar reads the open web and comes back with a grounded, cited summary rather than a list of blue links, and it is the piece that most often catches a same-name collision before it has a chance to become a wrong answer.

Reading the company's own words

The last piece of evidence-gathering is the one I like most: it reads the company's own website. A search step locates the business and its site, and then the page itself is fetched and stripped — scripts, styles, and markup peeled off — down to a few thousand characters of plain text. What the business says it does, in its own words, rather than what a model guesses it does from the name. It is cheap, and it is frequently the single most useful thing in the pile.

How it rationalizes the data

Now, and only now, a model gets involved — and it is kept on a short leash. The candidate codes are constrained to the real taxonomy: the model is handed the allowed codes and told, in no uncertain terms, that it may not invent one. It reads the collected evidence and returns up to three ranked candidates, each with a confidence tier and a rationale that points at the specific piece of evidence supporting it. When a name genuinely maps to more than one distinct business, it says so and returns each of them, rather than flattening a real ambiguity into one confident, wrong answer.

That disambiguation is exactly where the next piece is going in: Exa's Company Search, which I am wiring in specifically to resolve a name to the right entity — to take a name shared by several companies and separate it into distinct, identifiable businesses instead of a blurred average of all of them. The division of labor is clean: Sonar tells me what a business does; Exa is there to pin down which business it actually is.

The corpus, in three layers

The backbone of the whole thing is the corpus, and it comes in three layers. The first is the full classification taxonomy — every valid code and its official title — loaded two ways: as a hard validity net, and, vectorized with an embedding model (bge-m3), as a semantic ground truth the system can search by meaning rather than by keyword. The second is a curated table of well-known entities, for instant high-confidence matches on the obvious cases. The third is the one that matters most over time: a verified cache that compounds. Every answer a human confirms is written back, so the next identical lookup is instant, cited, and does not spend a cent on inference. The corpus is the part of the system that gets smarter while you sleep.

Two verifiers, both live

This is the spine, and it is the part most people skip.

The first is a deterministic verifier. Every code a probabilistic step proposes is checked against the authoritative title table. If it is not a real code, it is dropped — no exceptions, no "close enough." The official title is stamped on from the table, not recalled from a model's memory. A language model never gets the last word on whether a code actually exists.

The second is a cross-model verifier, and the rule behind it is simple: the author never grades its own work. Synthesis runs on one model — Claude Sonnet. An independent reviewer, a different open-weight model from a different lab entirely — OpenAI's gpt-oss-120b, running on Cloudflare Workers AI at the edge — is handed the name, the evidence, and the proposed answer, and asked one narrow question: does the evidence actually support this? It returns a yes or a no, a confidence number, and any issue it sees. If the reviewer disputes the top pick, the answer is flagged rather than shipped. This is a documented failure mode rather than a hunch: a model asked to grade its own output tends to prefer it, so the reviewer has to be someone else.

Both verifiers are running in production, on every request, right now — not a box on a slide, not a "future roadmap item." I tested the cross-model check against the obvious cases and the hard ones, watched it agree where it should and hesitate where it should, and only then called it done.

This is not just my instinct

I did not invent any of this. I wired together what the people who study the failure mode keep telling us to do. MIT researchers published a method this year for catching overconfident models, and the framing is blunt: a model can be confidently wrong, asking it the same question repeatedly and getting the same answer does not make the answer correct, and the reliable tell is disagreement across different models — which is exactly what the cross-model check is for. MIT's own teaching guidance says the quieter version of the same thing: evaluate the output with human judgment rather than assuming accuracy, verify it against authoritative sources, and ground it in trusted material before generating. And the federal framework for this, NIST's AI Risk Management Framework, treats a system as "valid and reliable" only when that is shown through objective evidence, with a human available to intervene where the system cannot catch its own errors — not assumed because the answer sounded sure of itself. Grounding, an independent check, and a human on the hard calls. I am not ahead of the guidance here; I just shipped it.

Where it runs

The whole system lives at the edge — Cloudflare Workers for the logic, Vectorize for the corpus, Workers AI for the second-opinion model. There is no server to patch and nothing to wake up. A name goes in; a ranked, verified, cited answer comes back.

The last rule: it never just guesses

The system is not permitted to refuse, and it is not permitted to bluff. If the evidence is thin, or the reviewer disputes the answer, it does not throw up its hands and it does not manufacture certainty — it hands the call to a person, and that person's decision feeds the verified corpus, so the machine is a little sharper the next time around.

The pattern is the point

The interesting part is not the classification. It is the shape of the thing. Take a short, ambiguous input; gather evidence from wherever it already lives; ground the answer so it cannot be invented; let a second, independent model check the first one's work; keep a human on the calls that are thin or disputed; and write every confirmed answer back so the system compounds.

That shape is not specific to industry codes. It is reusable anywhere a messy input has to map to a governed, correct answer — contractor and professional licensing, provider credentialing, entity resolution, cleaning and de-duplicating records against a controlled vocabulary. The public data is almost always already out there. A second opinion is cheap. The verifier is the part most people skip — and it is the part that makes any of it safe to trust. The gap is usually just building the connector, plus the discipline to make the machine prove that it is right.

That is the entire trick, and it is barely a trick: query widely, ground everything, let a code table and a second model check the first one's homework, and keep a human on the one decision that actually carries weight. The confident single-model version would have been finished in an afternoon. This one shows its work — which is the only version to which I would attach my name.