Episode 95 · Harness Design series
Planning, Verification and Building Your Own (Harness Design, Part 3)
Ines's agent now either gives up too early or keeps going after it's done. Fan, Zhang et al.'s planning ablation explains both. A persistent to-do list added 11.6 points for Nemotron-3 30B (median run 5 turns without it, 40 with it). It cut SWE-Bench cost by about 30% for Nemotron-3 550B and 32% for Mistral Medium 3.5 by trimming redundant verification, with a small, non-significant drop in accuracy. The episode covers verification and tests that give wrong answers, SWE-Bench Verified contamination and the SWE-Bench Pro retraction, the harness as a hidden variable in leaderboards, latency and cost, critiques of the paper, how to run your own ablation, a 13-point design checklist, and how it all maps onto multi-agent setups.
Listen to the show on your preferred podcast platform.
Inside this episode
- Cold open: what the agent does with more time
- What planning means in this paper
- The weak model: a list that keeps it alive
- The strong models: a list that tells them when to stop
- What a plan looks like in practice
- The middle model and the shape of a distribution
- Verification: the oracle and the over-checker
- When the oracle lies
- Evaluation pitfalls: the benchmark under the microscope
- The harness is a confound in every leaderboard
- Latency, turns and the shape of the bill
- What the critics say
- The big idea: harness components age
- Running your own ablation
- A checklist for building your own
- From one agent to many
- Closing: back at the whiteboard
Production credits & AI disclosure
Scripts generated with Claude Opus. Synthetic narration generated with Edge TTS. Reviewed, listened to, and approved by Sankalp Thakur.