Topic
Pass K Evaluation
3 episodes
-
Introducing GPT 6.1 Sol
GPT-6.1 Sol drops today — near-Astra performance on coding, computer use, and professional work at one-fifth the price, with cached input at ten cents per million tokens. Justy and Cody dig into whether the benchmark story holds up, what the cache pricing actually unlocks for agent builders, and where this lands in the broader cost-curve race.
-
When Models Edit Too Much: On the Fidelity of Minimal Code Edits
Laura and Harper examine why coding models often fix a bug by rewriting half the function, and why minimality deserves to be measured as its own dimension of code quality. They discuss the paper’s controlled benchmark, preservation prompting, reinforcement learning, production implications, and the limits of test-passing as a signal.
-
Evaluating performance and efficiency of the GitHub Copilot agentic harness across models and tasks
GitHub Copilot's agentic harness is a single cross-experience SDK component that orchestrates tools, context, and workflow across CLI, app, and code review. The team claims it delivers task-resolution parity with model-vendor harnesses while cutting token usage across several configurations, backed by public and internal benchmarks and real-world metrics. We debate technical validity, practical stakes for teams, and whether the harness should get most of the credit.