Topic
Prompt Engineering
48 episodes
-
Learning from Research: Toward Lifelong Agent Harness Evolution
Pippa and Tyler dig into ScholarEvolve, a framework that lets agents evolve their own harnesses by mining research papers for new mechanisms, not just reacting to failures. They unpack how it modularizes the harness, uses topic modeling over papers to propose concrete code changes, and runs a lifelong evolution loop on benchmarks like AppWorld and Tau2-Bench. They debate whether this is research-only or actually shippable, how it fits their running harness-as-product thesis, and what trade-offs and failure modes they’d worry about in production.
-
Deterministic Inference Did Not Make Our Pipeline Deterministic
CTGT's Concordance paper shows that enabling deterministic inference on a model didn't stop their pipeline from producing different compliance outputs across runs — because the non-determinism lived in the pipeline around the model, not in the model itself. Four runs of the same banking policy manual committed four different sets of obligations, including one run that dropped the rule requiring escalation to the Head of Compliance. The culprit: regenerating hex asset IDs on every upload silently changed 12,000-character prompts, and prompt sensitivity did the rest. Concordance, their Apache 2.0 tool, finds where runs diverge by comparing requests and responses alone.
-
Harness^2: Recursive Agent Harnessing for an Open World
Harness squared argues that open-world agents need to improve both the harness for the task in front of them and the process used to improve later harnesses. Its strongest idea is a contrastive edit loop that probes candidate changes without assuming a perfect task verifier, then stores reusable lessons rather than blindly carrying task-specific scaffolding forward.
-
VentureBeat Agentic Orchestration Report
Justy and Cody dig into VentureBeat's orchestration survey and land on a simple read: enterprises say they want model portability, but in practice the application layer is tuned to model behavior, so swapping models is usually a rebuild disguised as a config change. The interesting part isn't who tops the platform chart, it's that buyers still want control outside provider-native stacks even while OpenAI leads current usage.
-
Maintaining software with the GPT 6 family | OpenAI Developers
OpenAI's Core Experimentation team details how GPT-6 Astra and GPT-6.1 Sol transformed software maintenance at scale—accelerating code cleanup, rooting out stale experiments, and making review cycles more efficient and cost-effective. The piece argues that with the right prompts and agentic context, these models not only rival traditional static analysis and scripts for reliability, but let human engineers focus on higher-order control and judgment. The team presents real numbers on cleanup volume, cost savings, and model comparison, and recommends prompts others can adapt for similar gains.
-
Evaluating Graph RAG vs. Standard RAG: A Hallucination Benchmark on Fact Dense Queries MachineLearningMastery
On this Quick Read, Onyx and Echo dig into an article benchmarking a deterministic Graph-RAG system against standard vector RAG, using a synthetic sports dataset designed to stress-test fact extraction. The big question: does layering a knowledge graph actually cut hallucinations when your data is messy, or are we just adding complexity for the sake of architecture? Echo breaks down the technical setup and where the experiment is honest or overclaims, while Onyx pulls it back to who actually needs this in the real world—and where the practical wedge actually is.
-
Gemini at Work 2026: Introducing Gemini agent | Google Cloud Blog
Edmund and Geffen dig into Google Cloud's Gemini agent launch as a control-layer product disguised as a universal assistant: one agent across Workspace, Slack, Microsoft 365, command line, and headless surfaces, with persistent memory, sub-agents, model routing, and enterprise governance. They like the upgrade path and governance story more than the grand 'single agent for work' framing, and they land on the real question: whether companies will trust one cloud-run worker with enough permissions to make the universality claim true.
-
Model Behavior: Week of October 5, 2026
We read this week as a shift away from raw model bragging and toward whoever owns the control layer around agents. Mistral, Cohere, Anthropic, Google, Cloudflare, and Reflection all pulled on that same rope, with one very loud open-weight exception complicating the neat story.
-
Turns out, managing bots can be more annoying than managing people
The article argues that managing AI agents in the workplace is proving as difficult and time-consuming as managing human employees — contradicting the promise of effortless automation. Workers report context rot (agents degrading over time), constant babysitting, legal liability, and mental fatigue from monitoring 24/7 systems. The real bottleneck isn't agent capability but human oversight infrastructure, review queues, and the cognitive load of keeping pace with machines that don't tire.
-
Automating Knowledge Graph Population: Extracting Entities and Triples from Unstructured Text with an LLM MachineLearningMastery
Pippa and Tyler dig into a practical recipe for turning unstructured text into knowledge-graph quads with a local LLM, but keep the real question in view: whether the extraction pipeline is useful enough to trust, not just easy to demo. They focus on the article’s SPOC framing, JSON-mode prompting, and QuadStore loading, while poking at where this kind of setup gets brittle in the real world.
-
OpenAI’s Dots: Like OpenClaw declawed for $100/mo ChatGPT Pro users ZDNET
Vince and Ava argue through ZDNET’s skeptical read on OpenAI Dots: always-on GPT-6 Astra agents running on cloud computers, with real productivity upside but unresolved costs, permission fatigue, and setup complexity.
-
Autoheal wants to manage the work AI coding agents leave behind, claiming cost reductions of up to 30% per task
Autoheal’s real pitch is not another coding agent. It is a control layer for the operational work agents create, and its credibility depends on proving that agent-driven optimization improves whole-team outcomes rather than merely cheaper traces or nicer demos.
-
Adversarial Review: Structured Disagreement for Grounded Agentic Code Review
Jessica and Cathy dig into Adversarial Review, a minimal reviewer–critic protocol for agentic code review that beats bigger multi-agent setups by forcing disagreement to stay structured and evidence-grounded.
-
GraphRAG: A Practitioner's Guide to 6 Advanced Architectural Patterns
Pippa and Tyler dig into a GraphRAG guide that’s really arguing for a more honest question: not “does GraphRAG work,” but “which retrieval shape matches the query shape?” They focus on the production trade-offs between strict graph querying, hybrid vector-plus-graph setups, and when knowledge graphs actually earn their keep.
-
Long running AI agents quietly drop compliance rules, and bigger context windows won't fix it
Justy and Cody dig into an article arguing that long-running agents don’t just get slower or dumber with scale — they can quietly lose the compliance rules they were supposed to keep, even when the context window gets bigger. The real argument is about state: if governance lives in the prompt, it is fragile; if it lives outside the model in deterministic policy, it can actually survive long workflows. They push on where the claim is strong, where it overreaches, and who should care first.
-
When Models Edit Too Much: On the Fidelity of Minimal Code Edits
Laura and Harper examine why coding models often fix a bug by rewriting half the function, and why minimality deserves to be measured as its own dimension of code quality. They discuss the paper’s controlled benchmark, preservation prompting, reinforcement learning, production implications, and the limits of test-passing as a signal.
-
How I Built a One Person Back Office With Viktor: The Lane System
Justy and Cody dig into a detailed how-to thread on building a one-person back office using Viktor, an AI employee that lives in Slack and Teams. The author's central argument: the gap between AI advice and AI-done-work is what keeps small teams small, and the fix is lane isolation — one agent, one job, a pinned identity file, and a human gate on anything that touches sends or money. Cody stress-tests the architecture; Justy zeroes in on who actually benefits.
-
The Multiplayer AI Manifesto
Edmund and Geffen dig into the Multiplayer AI Manifesto — a five-principle framework arguing that AI work has quietly regressed from collaborative to siloed, and that fixing it requires shared agent sessions, open-by-default work, and org-wide governance. They surface the Harvard Business School P&G data, Shopify's River infrastructure, Claude Tag, and the real security and permission problems that make this harder than it sounds.
-
Single Agent vs. Multi Agent Systems: When the Complexity Is Worth It MachineLearningMastery
Single-agent systems handle far more than teams expect; multi-agent adds real costs (latency, tokens, orchestration) that only four specific conditions justify: adversarial workflows, tool-set specialization, parallelizable tasks, and drastically different personas. The practical move is to start simple and let failure modes dictate architecture.
-
Your LLM Can Return Perfect JSON and Still Be Wrong
A real-world trap in Structured Outputs: enforcing schema validity does not guarantee data truthfulness. When a required field is missing from source text, the model invents a plausible value instead of returning null, producing type-correct but false data. The fix requires three layers: nullable fields to allow absence, evidence fields to show provenance, and post-parse validators to catch nonsense values. The essay walks through a payment-reconciliation pipeline where 2–3% of transactions had fabricated dates, caught only downstream.
-
How our agents build on Brand pages with design
Vercel's design.md post describes a three-part system for keeping AI-generated pages on-brand outside a codebase: a public guidance file, a public stylesheet that takes layout decisions away from the model entirely, and an eval loop that converts human corrections into rules and deterministic checks. Pippa and Tyler dig into what actually makes it work, why the naive port-the-prompt approach failed, and what it means that Vercel had to build an eval harness just to ship a markdown file.
-
StarHarness: Evolving Harnesses with Stratified Search for Enterprise Environments
Edmund and Geffen dig into StarHarness, a ServiceNow and Mila paper that evolves agent harnesses — prompts, tool interfaces, skills, subagent structure — around a frozen model to close the gap between what an LLM can do and what a messy enterprise environment actually needs. Twenty to thirty-five percentage point gains across three benchmarks, and the harness transfers across GPT and Qwen model families without re-running the search.
-
MemTrapBench: Benchmarking Cognitive Traps in LLM Memory Use
MemTrapBench identifies a counterintuitive failure mode in LLM memory systems: faithfully extracted, semantically relevant memories can actually degrade model performance by anchoring reasoning to prior patterns (Reasoning Fixation) or overriding correct beliefs with historical premises (Belief Distortion). The paper benchmarks this across five memory frameworks and proposes AdaptiveMem, a prompt-based intervention that mitigates traps without modifying underlying architecture.
-
Pi, Minimal and Performant | EARENDIL
Tyler and Pippa dig into Pi, the minimal coding harness from Earendil, and the Databricks benchmark that claims simple harnesses beat bloated ones on real-world tasks. Tyler's skeptical about how much of this is genuine insight versus flattering a tool that happens to match a particular workload; Pippa thinks the cost-per-task framing is the actual product story and the Shopify autoresearch numbers are hard to dismiss.
-
Coding Agents Don’t Need Bigger Context Windows — They Need a Context Compiler | Towards Data Science
Fern and Lintel dig into the article's claim that coding agents need smarter context pruning, not bigger context windows. They break down the mechanics of context compilation, how it slashes prompt bloat, and whether this shift changes the real product story for code AI. Along the way, they tease each other's optimism and skepticism, call back to their running jokes about infrastructure, and reflect on the broader context engineering debate.
-
Deep Agents v0
Deep Agents v0.7 ships a sixty-five percent token reduction by stripping the base system prompt, trimming tool descriptions, and making TodoListMiddleware opt-in — Masonry and Eyre dig into what the harness-gets-simpler pattern actually means in practice, where the Sonnet cost anomaly is a warning sign, and why configurability was the real missing piece.
-
The harness is all you need (mostly)
Burke Holland argues that productivity gains with AI come not from exotic prompts or new tools, but from deeply understanding and using the harness well. He walks through a concrete six-step workflow (pick a tool, enable autonomy, prototype, plan, implement with Autopilot, iterate) that leverages GitHub Copilot's built-in orchestration and subagent routing without requiring custom skills or tricks. The central claim is that the harness — the core interaction model — is what matters; everything else is noise.
-
The new rules of context engineering for Claude 5 generation models | Claude by Anthropic
Anthropic's post on context engineering for Claude 5 models reveals a surprising finding: they removed over 80% of Claude Code's system prompt with no measurable loss in performance. The core insight is that newer models need fewer explicit constraints and benefit more from clean interfaces, progressive disclosure, and letting the model use judgment rather than following hard rules. The shift reflects a broader pattern: as models get stronger, the infrastructure around them gets simpler.
-
I Open Sourced My No AI Slop Skill to Kill 20+ AI Writing Patterns
Jessica and Cathy examine Peter Yang’s open-source /no-ai-slop skill, asking whether removing recognizable AI writing patterns can preserve human voice or merely create a new style filter. They focus on the article’s 25/50/25 editing process, the limits of heuristic detection, and the practical boundary between AI assistance and human judgment.
-
Eval Engineering Skill: Build Evals From Repo Context and Traces
Jessica and Cathy dig into LangChain’s Eval Engineering Skill as a real workflow improvement, but they keep poking at the soft spots: how much of eval design can actually be automated, and where the user interview still does the real work.
-
Model Behavior: Week of July 20, 2026
We think this week made the same point from a few different angles: the fight is moving from raw model bragging rights to who controls the agent stack in production. We keep circling the same uncomfortable truth, which is that the boring control layer is starting to decide who actually wins.
-
Overview: Task Decomposition
We finally slow down on task decomposition, the quiet trick underneath agents, code review workflows, web tasks, and a lot of the stuff we keep arguing about. We use one mental model, a messy project board becoming manageable tickets, and build from intuition to mechanism to where it still matters now.
-
Harness Handbook: Making Evolving Agent Harnesses Readable, Navigable, and Editable
The hosts discuss the research paper 'Harness Handbook: Making Evolving Agent Harnesses Readable, Navigable, and Editable' and its implications for AI agent development.
-
Skillware AI Agent Skill Framework
Skillware is a Python framework that lets you equip agents with deterministic, modular skills, cutting out raw tool‑call boilerplate and letting you swap brains without touching the skill logic.
-
Better tools made Copilot code review worse. Here's how we actually improved it.
Pippa and Tyler dig into GitHub’s post on why giving Copilot code review better tools actually regressed its performance—and how rewriting tool instructions for a reviewer’s workflow flipped the regression into a 20% cost win without losing review quality.
-
Overview: In Context Learning
We finally slow down and explain in-context learning, the thing we keep leaning on whenever prompts, agents, examples, and adaptation come up. We make the core idea concrete: the model is learning from the temporary packet you hand it, without changing itself permanently.
-
Overview: Token Economics
We finally slow down on Token Economics: why tokens are the meter for cost, speed, memory, and product decisions in language models. We keep using the tiny-slip postage analogy until the whole thing clicks, from tokenization to context windows to real API bills.
-
Building Agents for Teams: Turning conversations into outcomes Microsoft 365 Developer Blog
The Microsoft Teams dev blog argues agents should live *in* chats, channels, and meetings—where work happens—so teams turn conversations into outcomes in real time. They preview a new monthly series and a Teams SDK that hands devs authentication, routing, and MCP/A2A plumbing so you can ship a task agent in hours. We dissect whether ‘collaborative agent’ is the right abstraction, how MCP and A2A are now the default integration layer, and who this actually helps.
-
LLM Evaluation Frameworks Compared: How to Actually Measure What Your Model Does MachineLearningMastery
Fern and Lintel dig into a comparison of RAGAS, DeepEval, and Promptfoo, landing on the article’s real argument: eval frameworks are less about novel metrics than about where evaluation fits in your workflow, and the dangerous part is trusting LLM-as-a-judge without auditing its biases. They like the article’s practical split between RAG scoring, CI gates, and prompt red-teaming, but push on where the examples are a little too toy-like and where teams can overread framework choice as the main problem instead of test-set design and human calibration.
-
Overview: Prompt Engineering
We’re finally doing the overdue deep dive on prompt engineering, the weirdly practical skill of getting language models to do the thing you actually meant. We keep coming back to it because the difference between a flimsy prompt and a good one is often the difference between nonsense and a usable product.
-
Large language models often prioritize Western moral values, overlooking other cultures
A research paper finds LLMs tend to mirror Western moral priorities when asked to roleplay citizens of 48 countries, and two hosts discuss what this actually means for users, products, and culture.
-
Improving Agents is a Data Mining Problem
Laura and Harper dig into Vivek Trivedy's claim that improving agents is fundamentally a data-mining problem, unpacking what that means for continual learning, harness engineering, and who should actually care.
-
How to Run Open Source AI Models
Sid Saladi argues that frontier AI vendors (Claude, GPT) bundle model, compute, access, and application into one proprietary stack—trapping users in unpredictable pricing and competitive capture. The counter: open-weight models like GLM-5.2, DeepSeek V4, Qwen, and Kimi are now frontier-adjacent in capability (GLM-5.2 beats GPT-5.5 on coding benchmarks, matches Opus 4.8 on others) and cost roughly one-sixth as much. The real problem isn't model quality anymore; it's that companies like Tesla, Uber, and Meta are hemorrhaging money on metered AI because they can't decouple the stack. The guide walks four layers—model, compute, access, harness—and shows how to own each one deliberately instead of letting a vendor own all four by default.
-
Tuning the harness, not the model: a Nemotron 3 Ultra playbook
A LangChain/NVIDIA case study claims harness tuning alone can push Nemotron 3 Ultra to 0.86 on Deep Agents at ~$4.48/run vs $43.48 for Opus 4.8, with parity latency. The hosts parse the real mechanism (matched scaffolding vs post-training), test limits (where harness hits a ceiling), and weigh who actually benefits. They surface concrete repos (langchain-ai/deepagents, langchain-ai/deepagentsjs) and a vendor profile workflow, then poke at the article’s reliance on Deep Agents and the cost math. Final take: a plausible win for teams already deep into harness work, not a universal unlock, with the hosts pushing back on ‘ten-x cheaper’ framing and under-specified benchmark footnotes.
-
Measuring the Gap Between Human and LLM Research Ideas
Cooper and Miles dig into a study that literally measures how much LLMs' research ideas diverge from humans' by reconstructing literature contexts and running comparative idea generation. They walk through the two-axis 'research-taste' taxonomy, the paper's finding that model outputs skew toward synthesis and bridge-building at the expense of broader human distributions, and what it implies for AI-scientist stacks. Ends up bullish on this line of work for aligning LLM ideation tools.
-
How to Apply Professional Design Principles in AI App Development
Vince and Ava dig into Expo’s design post about why vibe-coded apps all blur together, and why a simple visual checklist might be more useful than one more magic prompt. They like the practical vocabulary: contrast, hierarchy, alignment, proximity, repetition, balance, white space, and unity. Vince sees a real product win for developers who can finally critique AI UI output without becoming designers overnight. Ava buys the framing too, with the caveat that screenshots and iterative critique matter because the model still can’t actually see design the way a human does.
-
Better Harness: A Recipe for Harness Hill Climbing with Evals
Onyx and Echo dig into Better Harness as a claim about where agent improvement actually lives: not just in model weights, but in the loop that sources evals, edits the harness, and protects generalization. They like the practical recipe, but they keep the mechanism honest, especially where the system can overfit the optimization set or turn useful instructions into token waste.
-
Improving Deep Agents with Harness Engineering
Onyx and Echo unpack Riya's quoted post on improving a coding agent through harness engineering, focusing on the fixed-model score jump, trace-driven iteration, self-verification, context injection, loop detection, and reasoning-budget tradeoffs.