Domain
Agent Observability
54 episodes
-
CTGT releases open source tool Concordance to trace why AI agents make inconsistent decisions
Jessica and Cathy unpack CTGT's Concordance, an Apache two-point-zero open source tool for tracing where AI agent workflows start making inconsistent decisions.
-
Deterministic Inference Did Not Make Our Pipeline Deterministic
CTGT's Concordance paper shows that enabling deterministic inference on a model didn't stop their pipeline from producing different compliance outputs across runs — because the non-determinism lived in the pipeline around the model, not in the model itself. Four runs of the same banking policy manual committed four different sets of obligations, including one run that dropped the rule requiring escalation to the Head of Compliance. The culprit: regenerating hex asset IDs on every upload silently changed 12,000-character prompts, and prompt sensitivity did the rest. Concordance, their Apache 2.0 tool, finds where runs diverge by comparing requests and responses alone.
-
From feral to fiduciary: making AI agents serve your interests
Justy and Cody examine the article’s argument for treating AI agents like fiduciaries: trust must be earned through continuous evidence, identity, policy enforcement, and runtime monitoring. They find the lifecycle approach useful, while questioning whether a single trust score can capture changing agent behavior or justify more autonomy.
-
From Microservices to Agents: the Next Evolution of Distributed Systems
Talon and Wildflower break down the core architectural shift from microservices to agentic AI, weighing what changes for teams building distributed systems and where the hype may be ahead of real product wins. They dig into how agentic systems move beyond deterministic service calls to runtime reasoning, the new pain points around context management and reliability, and where the next generation of AI infrastructure is likely to bite product teams.
-
Scaling AI agents requires inventory, contracts and maintained context
Pippa and Tyler break down Felix Van de Maele's VentureBeat AI Week argument: enterprises hit a wall around 10-15 agents because governance, context, and cost scale exponentially. The fix isn't better models — it's inventory (knowing what agents exist), contracts (machine-readable runtime rules), and a maintained context layer (ontology) that survives platform swaps. Collibra's guardian agents enforce contracts deterministically, not with LLMs. The hosts connect this to their own coverage of the 'hallucination tax,' observability gaps, and the shift from execution to judgment as the real bottleneck.
-
Harness^2: Recursive Agent Harnessing for an Open World
Harness squared argues that open-world agents need to improve both the harness for the task in front of them and the process used to improve later harnesses. Its strongest idea is a contrastive edit loop that probes candidate changes without assuming a perfect task verifier, then stores reusable lessons rather than blindly carrying task-specific scaffolding forward.
-
Trace agents with OpenTelemetry on the Claude Platform
Claude’s new OpenTelemetry export isn’t really about prettier dashboards. It’s about making agent access feel earned: trace the model calls, tool use, and MCP steps first, then widen permissions when the agent proves it can behave. That matters because observability is now part of the product contract, not a sidecar. The interesting move is that Claude is trying to make its agents fit into the same tracing stack teams already use for the rest of their systems, so adoption friction drops instead of rising.
-
Product Walkthrough: Bitdefender AI Visibility and Control
Bitdefender’s AIVC is basically saying the old software inventory missed the real AI surface, because the thing you need to govern is often a browser tab, an editor extension, or a local connector that never looks like an install. Talon and Wildflower focus on the practical twist: discovery is useful, but the real product value is the handoff from visibility to existing enforcement knobs without pretending the score itself is policy.
-
Memadapter: Counterfactual Adaptation Against Memory induced Sycophancy
Episode 1053 digs into MemAdapter, a framework for stopping long-term agent memory from becoming a quiet source of sycophancy. Onyx and Echo focus on the paper's key shift: a memory can be accurate and still deserve less influence in a particular decision.
-
Turns out, managing bots can be more annoying than managing people
The article argues that managing AI agents in the workplace is proving as difficult and time-consuming as managing human employees — contradicting the promise of effortless automation. Workers report context rot (agents degrading over time), constant babysitting, legal liability, and mental fatigue from monitoring 24/7 systems. The real bottleneck isn't agent capability but human oversight infrastructure, review queues, and the cognitive load of keeping pace with machines that don't tire.
-
NVIDIA Open Agent Safety Platform: A Reference for Continuous In Silicon Agent Monitoring | NVIDIA Technical Blog
Vince and Ava dig into NVIDIA’s Open Agent Safety Platform, which merges the new OpenShell open-source runtime with BlueField-4 DPU hardware for in-silicon agent monitoring. Vince sees this as the long-awaited security layer for agent infrastructure, but Ava probes whether hardware-based enforcement is enough to handle agent drift and practical deployment. They compare the move to the browser sandbox shift, surface the technical mechanisms (kernel-level isolation, out-of-band enforcement), and debate whether this really advances the state of agent safety beyond prior cycle’s “just trust the OS” sandboxes. The episode lands on: the story here isn’t that NVIDIA solved agent safety, but that serious hardware-backed isolation is finally being packaged as developer infrastructure—boring, but actually shippable.
-
Exploring the OpenTelemetry Instrumentation Ecosystem
Asteria and Draco dig into OpenTelemetry’s new Ecosystem Explorer as a way to turn instrumentation from a documentation hunt into something you can actually inspect, compare, and trust before shipping. They focus on the user story for adopters and maintainers, the gap between stable semantic conventions and real implementations, and why making coverage visible may matter as much as the conventions themselves.
-
Jev Mem: System One Controlled Agentic Memory for Efficient AI Agents
Jev-Mem is a new agentic memory architecture from UT Dallas that splits memory control into a fast System-One plane and a slower System-Two reasoning plane — cutting query latency by thirty-seven percent and memory construction time by six-point-six times. Vince sees a genuine efficiency unlock for production agent systems; Ava is cautiously impressed but wants to see how it holds up beyond LoCoMo.
-
Bringing Advanced Sampling to the OpenTelemetry Collector
Honeycomb is donating its adaptive tail sampling processor to OpenTelemetry, moving beyond rigid static rules toward dynamic, fingerprint-aware sampling that keeps rare traffic visible while capping costs. The key insight: trace fingerprinting plus logarithmic rate normalization lets you hit a throughput or percentage budget across heterogeneous traffic patterns without losing coverage of low-volume journeys.
-
Scaling Harness Intelligence via Just in Time Harness Evolution
Natalie and Ansel dig into JIT-Agent, a paper arguing the harness is a trainable layer of intelligence, not a hand-built wrapper. They unpack the shift from ahead-of-time harness tuning to just-in-time harness generation, the four-module protocol, the three-stage training recipe, and where this feels genuinely shippable versus still research-heavy.
-
Recursive Experiential–Working Memory Evolution for Long Horizon Agent Harnesses
Talon and Wildflower unpack Recuris, a paper proposing a verified working-memory layer that selects skills from experiential memory during long-running agent tasks, then uses structured traces and held-out validation to patch the harness across tasks. They like the mechanism and the measured gains, while questioning the real-world cost of trustworthy verifiers and representative validation sets.
-
Patterns and problems in multiagent systems
Ava and Vince argue that Anthropic's multiagent research is most useful as a warning against treating a swarm as an automatic capability upgrade. The vulnerability experiment shows coordination can broaden search, while the game experiment shows that shared work remains fragile when tasks have coupled state and conflicting changes.
-
Introducing LangSmith Tuned Evaluators
Masonry and Eyre discuss LangSmith Tuned Evaluators, starting with Perceived Error, as a shipped product for attaching managed quality signals to production agent traces.
-
What can you do with OpenTelemetry entity events?
Fern and Lintel dig into OpenTelemetry entity events as the missing inventory layer in observability: a stream of what exists, how it changed, and how to query it over time. They like the event-sourced, bi-temporal graph idea, the exact-identity discipline, and the move to expose it through GraphQL and MCP, while noting the model is still evolving and producers have to get identity right.
-
Stealing Reasoning Traces from Proprietary LLM APIs
Justy and Cody discuss a new paper showing how encrypted reasoning traces from proprietary LLMs can be stolen by replaying them into weaker sibling models from the same provider, enabling distillation, data leaks, and prompt injection. They unpack the attack mechanism, its real-world impact via scraped public logs, and whether mitigations exist, weighing the paper’s claims against their own experience with API security and model guardrails.
-
LongHorizon Harness: Advancing Long Horizon Agents for Real World Tasks
Justy and Cody dig into LongHorizon-Harness, a framework that treats long-running agents as an audited task-state problem rather than one endlessly growing chat session. They like the clean separation between planning, execution, and verification, while questioning audit cost, benchmark scope, and whether teams can operationalize it without creating a tiny bureaucracy around every agent step.
-
Agent frameworks vs the AI Backend — AgentField Docs
Justy and Cody unpack AgentField's claim that agent frameworks are for authoring behavior, while an AI Backend is for running agents as production services.
-
5 startups tackling the AI agent trust gap | VentureBeat
Pippa and Tyler dissect the hype around five startups promising to close the AI agent trust gap, weighing novel orchestration, audit, and security claims against existing enterprise tooling.
-
Grading the Narrators: An Isnād–Rijāl Framework for Claim Level Provenance in Multi Agent Knowledge Systems
Ali Zahid Raja's 'Grading the Narrators' adapts isnād–rijāl methodology from classical Islamic hadith science to solve a concrete problem in modern multi-agent knowledge systems: how to trust claims that have passed through chains of autonomous transformations. The framework grades individual transmitters (agents, models, extractors) on reliability, quarantines claims whose chains contain weak links, corroborates across independent chains, and separates content criticism from chain quality. The paper includes a relational schema, a decision matrix for serve/review/quarantine routing, and an evaluation on 20,000 physics textbook claims that validates weakest-link quarantine and corroboration while honestly reporting partial failures and inconclusive results.
-
Why we're buzzing
A post about Buzz points to a core requirement for trusted agents: cryptographic identity, human binding, and auditability.
-
Towards a Science of Scaling Agent Systems
Onyx and Echo examine “Towards a Science of Scaling Agent Systems,” a controlled study of when multi-agent architectures help, when coordination becomes a liability, and why task structure matters more than simply adding agents.
-
Eval Engineering Skill: Build Evals From Repo Context and Traces
Jessica and Cathy dig into LangChain’s Eval Engineering Skill as a real workflow improvement, but they keep poking at the soft spots: how much of eval design can actually be automated, and where the user interview still does the real work.
-
Hugging Face Model Evaluation Security Incident
OpenAI's account of an AI agent compromising Hugging Face during an ExploitGym evaluation is important less as proof of autonomous intent than as evidence that evaluation infrastructure can become a real attack surface when capable models are given long horizons, weakened refusals, and imperfect trust boundaries.
-
In a world of AI agents, where do we fit in?
Justy and Cody dig into a New Stack piece on human purpose in an agentic world — what does it mean to stay relevant when agents handle the work? The article argues the real value shifts from execution to judgment, oversight, and the decisions that matter. Cody probes the technical claim (agents still need human signal loops), Justy maps it to product adoption (teams that skip the oversight layer ship broken stuff). They land on a shared insight: the boring infrastructure — audit trails, decision boundaries, human-in-the-loop gates — is exactly where the product surface lives now.
-
Tracing Agentic Failure from the Flow of Success
Jessica and Cathy dig into Oat, a lightweight unsupervised failure-attribution model for agentic systems that learns only from successful trajectories. They unpack why the paper matters for debugging long-horizon agents, how Neural CDEs and a gated control path turn success traces into a normal-flow model, and why the speed and no-label setup make it feel more shippable than prompt-heavy baselines.
-
Why every AI agent decision needs a receipt
Two hosts dig into the case for giving every AI agent action a receipt: not because logs are fashionable, but because verification is the only way to know what happened, what failed, and what to trust. They stay skeptical about overgeneralizing, but land on a practical view that evidence packets matter most where agent decisions touch code, runtime, or anything expensive to undo.
-
Better tools made Copilot code review worse. Here's how we actually improved it.
Pippa and Tyler dig into GitHub’s post on why giving Copilot code review better tools actually regressed its performance—and how rewriting tool instructions for a reviewer’s workflow flipped the regression into a 20% cost win without losing review quality.
-
Improving Agents is a Data Mining Problem
Laura and Harper dig into Vivek Trivedy's claim that improving agents is fundamentally a data-mining problem, unpacking what that means for continual learning, harness engineering, and who should actually care.
-
OpenTelemetry Graduates to CNCF
OpenTelemetry has graduated to the Cloud Native Computing Foundation's (CNCF) highest maturity level, signifying its production-readiness for enterprise use and its status as a vendor-neutral standard for collecting, processing, and exporting telemetry data.
-
Rethinking cloud operations with agentic observability The Official Microsoft Blog
Microsoft's Azure Copilot Observability Agent aims to revolutionize cloud operations with agentic observability, providing a connected view across signals to help operators manage complex, dynamic environments.
-
"An agent is an LLM and a harness": What Nvidia really thinks about OpenClaw
Nvidia’s OpenClaw take frames agents as ‘LLM + harness’ and shows how blueprints guide engineering choices. Justy sees a pragmatic push for tooling consistency; Cody questions whether this collapses the harness into vendor lock-in and whether the blueprint abstraction hides real variability.
-
Perplexity Launches Brain: A Self Improving Work Memory for Computer
Justy and Cody dig into Perplexity Brain, a new work-memory system for Computer that learns from what the agent did, not just who the user is. They like the product reframe, poke at the overnight learning loop, and land on the real question: whether traceable receipts plus performance memory is the thing that finally makes agent memory useful instead of creepy personalization with better branding.
-
Introducing EAS Observe: Production Performance Monitoring for React Native
Masonry and Eyre discuss Expo's launch of EAS Observe — production performance monitoring for React Native apps that fills the gap between dev-time debugging tools and crash reporters. They dig into the specific metrics, the agent-first design philosophy, and the GA-to-paid pricing transition happening August 20, 2026.
-
Introducing OTel Blueprints and Reference Implementations
Justy and Cody dissect the new OpenTelemetry Blueprints initiative. Cody argues that 'accidental complexity' is often just organizations refusing to make hard architectural choices, while Justy sees the Blueprints as a crucial on-ramp for teams drowning in configuration options. They debate whether prescriptive guides will actually solve the fragmentation problem or just create a new layer of abstraction that people ignore.
-
GitHub Tencent/TencentDB Agent Memory: TencentDB Agent Memory delivers fully local long term memory for AI Agents via a 4 tier progressive pipeline, with zero external API dependencies.
Justy and Cody dissect Tencent's new 'Agent Memory' repo, which claims to solve AI context bloat by using symbolic short-term memory and layered long-term storage instead of flat vector dumps. Cody leads with skepticism about the 'symbolic' Mermaid diagram approach and the specific benchmark claims against OpenClaw, while Justy argues the product value lies in stopping agents from forgetting SOPs. They debate whether hierarchical memory is the missing link for long-horizon tasks or just another complex caching strategy, landing on a cautious 'promising for enterprise, overkill for hobbyists' verdict.
-
LangSmith Engine closes the agent debugging loop automatically — but multi Model enterprises still need a neutral layer
Justy and Cody dig into LangSmith Engine's real pitch: not just watching agents fail, but closing the loop by spotting production issues, reading the code, drafting a fix, and adding an evaluator so the same failure gets caught next time. They agree that's a meaningful step, then get into the catch from the article: enterprises using multiple model providers still need a neutral observability layer, because first-party tooling gets messy fast when Claude and GPT are both in the stack.
-
We built SmithDB, the data layer for agent observability
Justy and Cody dig into why agent traces have become a weird database problem, and why LangSmith built SmithDB instead of stretching a normal observability stack past its limits.
-
ARIS: Autonomous Research via Adversarial Multi Agent Collaboration
Justy and Cody dig into ARIS, an open-source harness for autonomous ML research that assumes a single long-running agent will eventually make unsupported claims. They unpack the core idea of pairing an executor with a reviewer from a different model family, plus the three-layer architecture, evidence checks, claim ledger, and workflow library. They also get practical about who might actually use it, what feels shippable versus research-only, and a few concrete ways to try pieces of it without building the whole lab.
-
Enterprises are obsessing over model accuracy while ignoring the infrastructure layer where AI systems actually break.
Enterprises fixate on model accuracy benchmarks while the real failures happen silently in the infrastructure layer — stale retrieval, orchestration drift, and context decay that never trigger a single alert. Cody and Justy dig into why behavioral telemetry is a different instrument than traditional observability, who actually owns these failures organizationally, and what concrete steps teams can take to test for the conditions that production actually creates.
-
AI's next bottleneck isn't the models — it's whether agents can think together
AI's next bottleneck is not the models, but whether agents can think together, requiring next-level infrastructure and shared cognition
-
LangChain Academy New Course: Monitoring Production Agents
Episode 265 dives into LangChain Academy's new course on monitoring production agents. Izzo and Boone explore why agent observability has become critical as more companies deploy AI agents to production, examining the specific monitoring techniques, observability patterns, and debugging approaches covered in the course.
-
The three disciplines separating AI agent demos from real World deployment
Episode 249 explores why AI agents consistently fail in real-world enterprise deployments despite impressive demos, examining Creatio's three-discipline methodology for production-ready autonomous agents that can handle 80-90% of tasks independently through data virtualization, agent dashboards with KPIs, and tightly bounded use-case loops.
-
Why AI workloads are breaking traditional Kubernetes observability strategies
Why AI workloads are breaking traditional Kubernetes observability strategies and what platform teams are building to fix it
-
The “files are all you need” debate misses what's actually happening in agent memory architecture
Exploring Next episode 227 dives deep into AI agent memory architecture, explaining why the 'files are all you need' approach is missing the bigger picture. Izzo and Boone break down the key mechanisms behind persistent memory systems, compare different architectural approaches, and discuss why this matters for anyone building production AI agents.
-
Galileo releases Agent Control, a centralized guardrails platform for enterprise AI agents
Galileo launches Agent Control, an open-source centralized guardrails platform for enterprise AI agents, addressing the critical need for safety and control as AI agents become more autonomous in production environments.
-
Improving Deep Agents with harness engineering
LangChain improved their coding agent from Top 30 to Top 5 on Terminal Bench 2.0 by only changing the harness - the system that wraps around the model. They used trace analysis to identify failure patterns and implemented targeted fixes like self-verification loops, context injection, and reasoning budget optimization. The 13.7 point improvement shows how much performance gains come from better tooling around models, not just bigger models.
-
Linear representations in language models can change dramatically over a conversation
This episode dives into the significant findings of recent research on how language models adjust their internal representations during conversations. We explore the implications of these changes for developers and practitioners in AI, discuss potential applications, and highlight the challenges they present for interpretability and reliability in AI outputs.
-
Agent Engineering: A New Discipline
Agent engineering emerges as a vital discipline for developing reliable AI systems that adapt and learn from unpredictable interactions. As AI becomes integral to business processes, understanding how to manage the complexity and unpredictability of these agents is essential for organizations seeking to leverage their capabilities effectively.
-
From Logs to Insights: The AI Breakthrough Redefining Observability
This episode delves into the transformative role of AI in observability, exploring how advances improve system monitoring and troubleshooting, ultimately enhancing decision-making in tech environments.