Your LLM Can Return Perfect JSON and Still Be Wrong
A real-world trap in Structured Outputs: enforcing schema validity does not guarantee data truthfulness. When a required field is missing from source text, the model invents a plausible value instead of returning null, producing type-correct but false data. The fix requires three layers: nullable fields to allow absence, evidence fields to show provenance, and post-parse validators to catch nonsense values. The essay walks through a payment-reconciliation pipeline where 2–3% of transactions had fabricated dates, caught only downstream.
Transcript
Pippa Okay, so there's this essay about Structured Outputs that just landed, and it's going to sound like a very boring infrastructure take until you realize it's actually about why your extraction pipeline is silently poisoning your data right now.
Tyler Mm-hm.
Pippa The author deployed a payment-extraction system using OpenAI's Structured Outputs — you know, the thing that guarantees the JSON is always valid. Three weeks in, their reconciliation starts flagging mismatches. Amounts match, sender matches, but the date is wrong. Like, consistently wrong. Two to three percent of transactions a week.
Tyler What do you mean wrong?
Pippa The date was always the day the extraction job ran. Off by less than an hour. And when they pulled the raw source messages, the pattern was obvious: every single mismatched transaction had no date in the source text at all. The schema said `transaction_date` was required, so the model just… filled it in. Hallucinated it.
Tyler Wait. So the schema enforcement worked. The JSON was structurally perfect.
Pippa Completely valid. Type-correct. Never threw an error. It was only when something downstream tried to reconcile against the actual transaction records that the mismatch showed up. So they'd been running this for weeks thinking the pipeline was working because, technically, it was. The JSON never failed validation.
Tyler Right, right. The schema can't distinguish between a value the model actually found and a value it invented to satisfy the schema itself.
Pippa Exactly. And the fix is kind of elegant. Three layers. First: make the field nullable. If the date isn't in the message, the model returns `null` instead of guessing.
Tyler That's the obvious one.
Pippa Right. But then there's the second layer, which is what I actually think is the real insight here. Add an evidence field. For every extracted value, require the model to also return the exact text from the source that backs it up.
Tyler Oh, interesting.
Pippa So if `transaction_date` has a value but `evidence` is empty, or the evidence doesn't actually appear in the source, you've caught a hallucination in the data itself. You don't need a downstream reconciliation to find it anymore. It shows up in the extract.
Tyler That costs tokens, though.
Pippa The author says it pushed output tokens up by roughly a third on a few-hundred-message batch. Latency matters at scale. So it's not free. But for a financial figure someone's actually going to act on, the author says it's worth it. For a zip code? Probably not.
Tyler Mm-hm.
Pippa And then the third layer is post-parse validation. You can't ask the model to enforce world-truth in a prompt — it's not a calculator. But a validator in your code absolutely can. Negative amounts, dates three days from now, stuff that makes no sense as a fact about the world. Pydantic catches that, every single time, for free.
Tyler So you're drawing a line between generation and validation. The model generates a structured response. The schema enforces structure. The validator enforces sense.
Pippa Three separate concerns. And most people treat Structured Outputs as if it solves all three, when it's really just the first one. The article has code for all of it — nullable fields, evidence wrappers, validators, even a retry loop that hands validation errors back to the model.
Tyler With a hard cap on retries?
Pippa Two attempts, max. Because by the third failure, you're not looking at a model problem. You're looking at bad source data, and the model's never going to fix that. So you hand it to a human instead.
Tyler That's the part that actually matters. Most teams would just let it keep retrying.
Pippa Yeah. And they'd burn API calls on something that was never the model's problem to begin with. The author's saying know when to give up and escalate — that's actually the useful call here.
Tyler This is going to be one of those essays that sounds boring until someone's pipeline starts silently corrupting data and they realize they skipped the evidence layer.
Pippa Exactly. Structured Outputs solved the JSON crisis, but they created a new one where the data looks perfect and is completely wrong. And nobody's talking about that part yet.
Tyler Until now.
Pippa Until now. It's the kind of infrastructure insight that feels obvious once you see it, but costs actual money and time until you do.