If Your AI Can’t Show Its Work, Don’t Trust the Number

A number shows up in a dashboard, a report, a recommendation engine. It’s precise — 94.3% confidence, $2.1M projected savings, moderate risk. Precision reads as trustworthy. But precision isn’t the question that actually matters. The question is: if I asked where this number came from — not why the model thinks it’s right, but what data, what version, what transformation actually produced it — could anyone answer? In most AI systems I’ve reviewed, the honest answer is no. Nobody can walk it back. The number just arrived.

Provenance is not explainability

These get treated as the same problem. They aren’t. NIST’s AI Risk Management Framework lists them as separate trustworthiness characteristics for a reason — “accountable and transparent” sits apart from “explainable and interpretable.” Explainability asks the model, or a second model built to explain the first one, to narrate why it produced a given output. Provenance asks something the model can’t answer for you at all: what actually happened, mechanically, before this number existed. NIST’s own framing of traceability is blunt about the difference — someone owns the risk, the decision can be traced after the fact, and a human can understand and correct it when it’s wrong. Traced. Not explained. Those are different verbs.

The distinction matters because an explanation can be fluent, plausible, confident, and entirely disconnected from what actually happened — the exact failure mode I wrote about in Hallucination Is a Feature, Not a Bug. A model that hallucinates a number will also, if you ask it, hallucinate a perfectly reasonable-sounding explanation for that number. The explanation is generated by the same mechanism as the number. It doesn’t check the number. It rationalizes it. Provenance is the only question in the pair that doesn’t route back through the model grading its own homework.

A citation is not provenance either

The obvious fix looks like: make the model cite its sources. Retrieval-augmented generation already does this — ground the output, attach a footnote, done. The research on how well that actually works is not encouraging. A 2024 evaluation of how LLMs cite medical references found that out of 110 statement‑source pairs GPT-4 produced with retrieval, 105 weren’t actually supported by the source it cited. A separate 2026 study, evaluating source attribution in LLM research agents specifically, found GPT-4’s retrieval-augmented responses were only 54% supported at the full-response level — and drew a distinction I think is the whole point here: citation faithfulness is not citation correctness. A citation can point to a real, relevant, accurate source and still not be provenance, if the model didn’t actually use that source to construct the answer. It just decided, after the fact, that the citation looked right next to the claim.

A footnote answers “does a source exist that could support this.” Provenance answers “is this the source that actually produced it.” Most RAG citations only answer the first one.

What happens when nobody can trace the number fast enough

Zillow’s iBuying algorithm priced homes it was willing to buy directly from sellers, at scale, for years. When the pandemic broke the historical patterns the model was trained on, its offer prices stayed confidently wrong for long enough to matter. In the third quarter of 2021 alone, Zillow bought 9,680 houses and managed to sell only 3,032 of them, at an average loss of $80,000 per property. The company took a $304 million inventory write-down, total losses cleared $500 million, and Zillow shut the entire Offers division down, laying off a quarter of its workforce in the process. CEO Rich Barton was direct about the cause: the algorithm could no longer accurately predict prices, and the business had been built on trusting the number it produced.

I don’t think the core failure was that the model was wrong — models drift, that’s expected, it’s the same lesson from Guardrails Are Not Governance about accountability not ending at launch. The failure was speed of detection. A number that’s wrong for a week is a bad trade. A number that’s wrong for two quarters, at scale, while nobody can cheaply reconstruct what assumptions it was built on to catch the drift, is a $500 million write-down. Provenance is what makes the second timeline look like the first one.

What building it into the architecture actually looks like

This isn’t a new problem AI invented, and it isn’t a problem AI has to solve from scratch. Section 404 of Sarbanes–Oxley has required exactly this discipline for financial figures since 2002: a documented internal control, a named party accountable for each number, and an auditor's attestation to the effectiveness of the whole process — specifically so a number in a filing can be walked back to its source under audit. Nobody asks a company’s earnings report to explain itself in prose. They ask for the trail.

The technical version of that trail already has a name. The W3C’s PROV data model formalizes provenance as a graph of three things: entities (the data), activities (what was done to it), and agents (who or what did it) — connected by explicit relations like wasGeneratedBy and wasAttributedTo. It’s deliberately boring and deliberately structured, which is the point: a provenance record isn’t a paragraph explaining itself, it’s a queryable graph you can walk backward from any number to every input and transformation that produced it.

Two more pieces close the loop before a number even gets generated. Gebru et al.’s Datasheets for Datasets asks that every dataset document its own motivation, composition, collection process, and intended use — so provenance doesn’t start at inference, it starts at data collection. Mitchell et al.’s Model Cards do the same for the model itself: what it was evaluated on, across which groups, under what conditions, for what intended use. Neither of these is exotic. Both have real industrial adoption at Microsoft, Google, and IBM. Together with a PROV-style trail from input to output, they answer the provenance question end to end: what data, what model, what transformation, in that order, every time.

Content provenance has its own maturing standard outside of raw data pipelines, too. C2PA — whose steering committee includes Adobe, Google, Microsoft, OpenAI, and Sony — attaches a cryptographically signed manifest to generated content: what tool made it, what edits were applied, what source assets went into it. Tamper-evident, verifiable independent of whatever caption or explanation ships alongside the content. That’s the content-provenance version of the same idea PROV formalizes for data: don’t ask the artifact to vouch for itself, attach a record that does.

What I’d actually ask

Not “can the model explain this number.” It almost always can — fluently, confidently, and, per the citation research above, not necessarily accurately. The better questions:

• If I audited this number six months from now, could I reconstruct exactly what data and what model version produced it — without asking the system to narrate itself?
• Is this citation here because the model used it, or because it looks right sitting next to the claim?
• Who’s named as accountable for this number the way Sarbanes–Oxley names someone accountable for a line in a 10-K — and would that name survive an actual audit?

Explainability asks the model to tell you a story about itself. Provenance doesn’t ask — it checks. Build the system so it doesn’t need the model’s permission to be honest.

Part of a larger framework — The System Around the Algorithm