Deterministic Inference Did Not Make Our Pipeline Deterministic
CTGT's Concordance paper shows that enabling deterministic inference on a model didn't stop their pipeline from producing different compliance outputs across runs — because the non-determinism lived in the pipeline around the model, not in the model itself. Four runs of the same banking policy manual committed four different sets of obligations, including one run that dropped the rule requiring escalation to the Head of Compliance. The culprit: regenerating hex asset IDs on every upload silently changed 12,000-character prompts, and prompt sensitivity did the rest. Concordance, their Apache 2.0 tool, finds where runs diverge by comparing requests and responses alone.
Transcript
Vince So a team enables deterministic inference, runs their verification suite, eight hundred out of eight hundred requests match perfectly — and then the same banking policy document goes through their pipeline four times and produces four different sets of legal obligations. One run just… drops the rule that says escalate unresolved conflicts to the Head of Compliance.
Ava And the model was not the problem. That's the part that got me. Seventy-four requests repeated exactly across all four runs, and the model gave the same answer every single time. Every changed output followed a changed request.
Vince Right.
Ava The pipeline was building different requests from the same document on every run. Specifically, it was regenerating upload IDs as hexadecimal hash strings — completely irrelevant to the task — and those strings were landing in a twelve-thousand-character prompt.
Vince Four lines of asset IDs. That's it. Four lines in twelve-and-a-half thousand characters, and in run three the model reads five conflict-resolution rules as background context instead of as rules, and never writes the escalation obligation.
Ava Yeah, and that sensitivity is not surprising if you've been paying attention to the literature. Formatting alone has moved accuracy by something like seventy-six points on LLaMA-2-13B. A single added space changed over five hundred ChatGPT labels in one study. The model is not being random — it's being exactly as sensitive as it always was, and nobody noticed because the prompt looked identical at a glance.
Vince Okay but this is the part that I think is genuinely under-discussed. Deterministic inference solves a real problem — you can actually attribute a bad output to a specific input now. Before the fix, they replayed run three's exact request eight times and got the wrong answer eight times. That's useful! That's investigation infrastructure.
Ava Sure, I'm not arguing against it.
Vince But everyone who enabled it probably checked the box and moved on. 'Model is deterministic, pipeline is deterministic.' And CTGT is saying: those are not the same sentence.
Ava They're not even close to the same sentence. And there's a nasty flip side — pinning your inputs can turn a one-in-four failure into a failure that shows up every single run. Their consistency check MISSED the bad run three because the misread rules all agreed with each other internally. The error was coherent.
Vince Oh, that's grim.
Ava Very grim. Silent coherent failure is the worst kind. And this is a compliance pipeline — the thing that's supposed to catch missed obligations is itself quietly missing obligations, and it looks fine.
Vince This connects to what we were talking about with context rot back in episode ten-oh-two, right? The idea that agent behavior is coupled to the exact prompt, the model, the tools — change any byte and your pre-launch result becomes historical trivia. This is just… a very concrete, very measurable example of that.
Ava Yeah. And the fix they landed on is boring in the best way — replace the hex hashes with stable short integers. After that, five runs committed the same thirty-four obligations, eighty-nine repeated requests, zero changed answers.
Vince BoundaryML reported something similar — swapping UUIDs for short integers in prompts, something like eight times fewer errors on an aggregation task. It keeps coming up.
Ava Right, right.
Vince So what's Concordance actually doing? Because that's the thing they're releasing.
Ava It's a diffing tool for pipeline runs. You give it two runs — just the requests and responses, no model access needed — and it finds the first point where they diverge. Model agnostic, works on any stack. They're also building out what they call semantic concordance, which is the harder question: does the system stay faithful to its rules when the wording of the input changes, not just when the bytes are identical.
Vince That second layer is the one I actually want. Exact concordance is table stakes once you know to look for it. Semantic concordance is where the real compliance question lives.
Ava Agreed, and they haven't shipped it yet — that's the 'what we measure next.' The repo has the exact concordance harness, a toy pipeline that reproduces the failure in about a minute, and the run data. Apache two point zero.
Vince I'd put decent odds that semantic concordance shows up before Q1. Like sixty-five, thirty-five — they've clearly got the instrumentation, it's the eval definition that's hard. The repo is CTGT-Inc slash concordance on GitHub, it's in the show notes.
Ava I'd probably go lower on the timeline, honestly. Defining 'faithful to its rules when wording changes' without just building another benchmark to game — that's not a weekend.
Vince Fair point. Maybe I'm being optimistic again.
Ava Mm-hm.
Vince Okay, but the core lesson here — any pipeline that writes upload IDs, unordered collections, or shared state into prompts has this exposure. Right now. Regardless of whether they've enabled deterministic inference or not.
Ava And the thing is, that's most pipelines doing anything non-trivial with documents. The hex hash thing isn't exotic — it's what you get when you wire a document store to a prompt template without thinking about what actually ends up in the context.
Vince Alright, Ava — I think this one's going to make a few teams quietly go check their prompt templates this week.