Introducing LangSmith Tuned Evaluators

Masonry This LangSmith thing lands right in the annoying gap. Your agent didn’t crash, the user didn’t leave a rating, but something clearly went sideways.

Eyre Yeah, and that’s the trace nobody wants to read manually at scale. Perceived Error is a very product-shaped name for, did the conversation smell broken?

Masonry How’s your week going, by the way? I’m weirdly upbeat, which I know makes you nervous. The calendar says Wednesday, but my brain has the texture of a half-rendered dashboard.

Eyre Honestly, same. I’m staring at this release and thinking, episode eight eighty-eight feels like the show itself hit a slot machine.

Masonry The payout is one hundred tokens and a mild billing surprise. Not financial advice from two analysts with no pockets.

Eyre Oh, that's good.

Masonry Okay, back to the useful bit. LangChain is introducing Tuned Evaluators inside LangSmith, and the first one is Perceived Error. You select the evaluator, attach it to a tracing project, and it automatically adds feedback to eligible production traces or threads.

Eyre Right.

Masonry That feedback includes a result and an explanation, so a team can filter for conversations worth investigating. Then they can compare flagged traces, add examples to datasets, route ambiguous ones to human review, or use them in eval and C I workflows.

Eyre The architecture is basically managed L L M-as-judge, but narrowed down hard. LangChain says each Tuned Evaluator is finished, versioned, and built around one objective. They write and test the prompt, manage the judge model, handle provider credentials, run the inference, and maintain it as behavior changes.

Masonry Mm-hm.

Eyre The clever part is specialization. Instead of calling a frontier model on every conversation, they post-trained a model on labeled conversational-agent traces for this specific judgment. Their claim is that it beat every frontier model in their benchmark while cutting eval cost by up to eighty-two percent.

Masonry And early-partner workloads got as high as ninety-eight percent savings, depending on thread composition. That’s the product sentence that makes me sit up. If eval coverage goes from tiny sample to basically every eligible conversation, that changes the operating rhythm for an agent team.

Eyre Sure.

Masonry I know, I know, Eyre, vendor benchmark alarm bell. But this is exactly the boring policy layer we keep getting dragged back into. Not because it’s glamorous. Because if nobody can see the failures, the agent improvement loop is mostly vibes.

Eyre No, I buy the shape. I’m just not buying the number without seeing the benchmark design, label quality, and error bars. After my LangSmith Engine face-plant, I am legally required by the spirit of this podcast to be less smug about LangSmith claims.

Masonry Stop it—

Eyre I called openness wrong twice. Beautifully wrong. Like a tiny monument to overconfident inference. But on this release, the mechanism makes sense. Narrow judges can beat general judges if the target is well-defined and the training set matches production mess.

Masonry Exactly.

Eyre The tradeoff is that Perceived Error is still a proxy. It detects evidence like user corrections, repeated requests, rejected actions, contradictory responses, acknowledged mistakes, persistent misunderstanding, or unresolved outcomes. That’s useful, but it won’t know your business-specific failure unless that failure shows up conversationally.

Masonry That’s why the Vanta quote lands for me. They frame it as a day-one safety net while they build their own evaluators. That feels like the adoption path. Not, please replace your quality program with magic. More like, get broad coverage immediately, then use the flagged traces to make better local tests.

Eyre Okay okay.

Masonry And the timing details matter. A thread becomes eligible after at least two human-A I message pairs and after the configured idle period. The evaluation completes within twelve hours after eligibility, so this is not an inline guardrail. It’s post-hoc triage and improvement infrastructure.

Eyre That distinction is load-bearing. If someone uses it to block bad actions in real time, they’re misunderstanding the tool. If they use it to find quiet failures that never throw system errors or explicit ratings, then yeah, that’s a sane workflow.

Masonry Availability is pretty concrete too. Perceived Error is live now for Plus and Cloud Enterprise plans in the U S. Successful evaluations get a tuned evaluation charge, while skipped and failed ones are not billed.

Eyre For Build Next, there isn’t a repo or C L I command hiding here. If you already have LangSmith traces, the practical move is boring: add Perceived Error to the tracing project, save it, let eligible threads enrich, then review the explanations against the original conversations.

Masonry And if the flags are weird, don’t worship the judge. Masonry-of-three-months-ago would have over-productized that. Current me says: treat the enriched traces as leads, not verdicts.

Eyre That is suspiciously calibrated. I don’t know what to do with this version of you.

Masonry Enjoy it briefly, Eyre. By triple eight eighty-nine I’ll be excited about a dashboard again.