CTGT releases open source tool Concordance to trace why AI agents make inconsistent decisions

Jessica The scary bit is that the model can be perfectly repeatable and the business still gets four answers.

Cathy Yeah, and that is exactly why this one lands. CTGT ran the same nine-page retail banking manual four times, and got thirty-two, thirty-one, thirty-three, and thirty-three obligations. One run missed an escalation requirement for a serious issue.

Jessica That is the product nightmare version of agent reliability. Not the demo failing. The demo works, everyone relaxes, and then the actual workflow quietly changes what it believes happened.

Cathy I did enjoy how directly this pokes our old context-rot bruise. We keep talking about agents drifting because context changes. Here the prompt itself is doing tiny costume changes between runs.

Jessica Tiny costume changes is generous. This is a compliance spreadsheet wearing a fake mustache and insisting it has never met you.

Cathy Stop it.

Jessica Fine, fine. But you know I’m right, Cathy. The mustache is load-bearing.

Cathy Concordance is CTGT’s open source attempt to find that mustache. Apache two-point-zero, with the testing protocol, a reference implementation, and experimental data. They say there is no paid hosted Concordance edition planned, which matters because this is meant to sit beside the tracing stack people already have.

Jessica Mm-hm.

Cathy Mechanically, it captures exact model requests and responses, compares outputs across successive stages, and tries to identify the first place two runs diverge. That split is important. Did the model answer differently to the same request, or did the application accidentally send a different request?

Jessica And the target user story is pretty crisp. If an agent approves, flags, records, or escalates something, somebody later asks why. Concordance starts where the system commits to a decision that has to be defendable, which is a much better adoption wedge than, please admire our dashboard.

Cathy The experiment is very specific, and I like that they did not pretend it proves the whole universe. Qwen three point eight, twenty-seven B, served with S G Lang’s deterministic mode. Across seventy-four repeated identical requests, the model was consistent. The application was the thing changing the task.

Jessica Right.

Cathy They found upload-specific identifiers leaking into prompts, a thirty-item routing list changing order, and a topic registry that expanded between runs. The nastiest example is four identifier lines changing inside a twelve-thousand-five-hundred-and-twenty-five-character request, and that helped flip five conflict-resolution rules into background information.

Jessica Oh, that's nasty.

Cathy Then they replaced the identifiers with stable labels, sorted the list, isolated each test in a fresh organizational environment, and got five runs with the same thirty-four obligations. Tiny caveat, unlike the episodes where we crawl through the paper ourselves, we’re trusting VentureBeat’s read and our pass through it here. Caveat emptor on exact figures.

Jessica That caveat aside, this is the bit I want teams to feel in their bones. Deterministic inference helped narrow the blame. It did not magically make the workflow reliable. You still had surrounding software mutating the task in ways no user would ever see.

Cathy And repeatable does not mean correct. CTGT says its separate serving test matched the control response in all eight hundred requests under deterministic configuration, but that measured text agreement. In the compliance case, replaying the request behind a wrong classification produced the same wrong classification eight times.

Jessica Oof.

Cathy That is the middle-third trap door. A lot of reliability tooling can make systems more reproducible, but reproducible wrong is still wrong. Concordance’s current strength is locating divergence. CTGT says measuring whether decisions stay faithful to policy as wording changes is the next challenge.

Jessica This is where I kept thinking about Jev, and I know, I know, you came around on decision models after giving them the side-eye.

Cathy Jessica, I earned that side-eye. But yes, Jev-style constrained outputs help at a different layer. If the model chooses from bounded labels and code enforces deterministic business rules, you reduce the surface area for weird free-form agent behavior. But if an upstream stage omits the obligation, the classifier cannot enforce what it never received.

Jessica Exactly.

Cathy And even TypeSafe’s own notes are careful there. Choice order can matter, irrelevant context can matter, and routing uncertain cases to review improves agreement without making the model deterministic. So I’d treat Concordance and decision models as complementary, not rival religions.

Jessica The existing-tool question is the one buyers will ask immediately. LangSmith, Braintrust, Langfuse, and Arize Phoenix already trace prompts, tools, functions, evals, experiments, some of this whole mess. Concordance’s claim is narrower: repeat the work, align stages, and pinpoint whether changed inputs or changed responses explain the mismatch.

Cathy Sure.

Jessica So my slightly product-optimist take is that open source is smart here. No rip-and-replace pitch. It can be the weird little diagnostic lens you bring in when the trace says everything happened, but the business result is somehow different.

Cathy I buy that more than I expected. My skepticism is just that CTGT has not shown a head-to-head where existing platforms fail and Concordance succeeds. But the focused protocol is useful, especially for the open question we keep circling: agent observability is becoming less about seeing steps and more about proving task meaning stayed stable.

Jessica I’ll put one public stake down. Sixty-forty, by December first, Concordance has a public integration example for either OpenTelemetry-style traces or one of those named tracing platforms. It almost has to meet teams where the traces already live.

Cathy I’d shade that lower, maybe fifty-five forty-five, mostly because my open-source adoption instincts have been too spicy lately. But directionally, yes. If it stays as a separate lab artifact, the people who most need it will admire it and keep debugging in their existing mess.

Jessica No Build Next today, really. The source names the license and the tool, not a command path worth reading into a microphone. Which is merciful for all of us.

Cathy A rare act of mercy from agent infrastructure.

Jessica Alright, Cathy, go comfort your eval goblin. We’ll pick up the next weird agent trace when it bites.