When Models Edit Too Much: On the Fidelity of Minimal Code Edits

Laura The wild part is a model fixing a one-line off-by-one bug by deleting five lines and adding sixty. All five tests pass. So the patch is correct… and still kind of alarming.

Harper Yeah, because the tests answer whether the program survives the examples. They do not answer whether the model respected the existing implementation. This paper calls that gap over-editing, and it measures it instead of just complaining about ugly diffs.

Laura That matters right now because brownfield code is where these tools actually live. Nobody hands an agent a pristine empty file and says, make me software. They point it at the old function that has six invisible assumptions and a review queue already groaning.

Harper Right.

Laura And if the agent changes the local bug, the data flow, the input handling, and the nearby edge cases in one sweep, the human reviewer has to rediscover the whole function. That is not free automation. It is automation that moved the work upstairs.

Harper The evaluation setup is unusually clean. They take four hundred BigCodeBench problems, inject one or two localized corruptions at the abstract syntax tree level, and keep only cases where the corrupted version fails the original tests. The minimal repair is known by construction: reverse the injected corruption.

Laura Mm-hm.

Harper That lets them separate three things. Did the repair pass? How far did the output move from the minimal repair, using normalized token-level Levenshtein distance? And did it add structural burden, measured with cognitive complexity? It is not a perfect definition of good maintenance, but it is much better than pretending Pass at one captures all of it.

Laura The example makes the product reality painfully obvious. A one-line boundary correction passes five tests. GPT-5.4 adds validation, type coercion, NaN masking, and resampling. Those changes might be sensible in some other task, but here they are undocumented scope expansion wearing a green test result.

Harper And across twenty frontier models, among completions that passed all five tests, median inserted lines ranged from two to sixty. So this is not just one weird prompt or one eccentric model. The model can identify the defect and still choose the wrong granularity for the repair.

Laura That is the useful phrase for me: granularity mismatch. The system understands the problem locally, then expresses the solution globally. Which is exactly the kind of thing a developer experiences as, why did you touch that?

Harper The first intervention is almost embarrassingly small: add a preservation instruction. It reduces average excess Levenshtein distance from 0.195 to 0.131, cuts added cognitive complexity by 26.6 percent, and improves Pass at one by 2.3 points.

Laura Wait, it improves correctness too?

Harper A little, yes. The likely mechanism is that the instruction changes the search space. The model stops treating defensive additions and rewrites as free options, so it has fewer opportunities to disturb working behavior. But the paper is careful: more reasoning budget does not reliably produce more faithful edits, and larger models are not monotonically more conservative.

Laura Right.

Harper That is where I push back on the usual capability story. We often assume a stronger model will infer the developer's intent and exercise restraint automatically. This result says restraint is its own behavior. You may need to ask for it, evaluate it, and train for it directly.

Laura And the training comparison is the part that makes this more than a prompt trick. Supervised fine-tuning learns the corruption patterns it sees, then overfits. Reinforcement learning gets the best trade-off outside those patterns: 0.782 out-of-domain Pass at one with 0.050 excess Levenshtein distance, without degrading general coding ability.

Harper That is a real result.

Laura Yeah. It suggests the target is not memorizing what an off-by-one repair looks like. It is learning a preference for preserving unrelated code while still satisfying the tests. For a coding product, that could become a gate after generation: show the repair, show the tests, and show whether the diff is proportionate to the requested change.

Harper I would be careful with that last part. A small diff is not automatically a safe diff. The benchmark is deliberately local, and the authors exclude refactoring, feature extension, and architectural work. In production, a broader edit can be justified. The system needs a reason trail for why the scope expanded, not a fetish for one-line patches.

Laura Fair. The useful contract is probably: preserve everything you can, and make expansion explainable. If the request is fix the boundary condition, a sixty-line rewrite needs evidence. If the request is replace the parser, minimality means something else.

Harper And tests still have the same weakness we keep running into. A passing suite can fail to exercise the code the model changed. Edit fidelity adds a second signal, but it does not replace stronger oracles, repository rules, or human judgment when the behavior is ambiguous.

Laura This is also why I like the paper's framing. It does not claim over-editing is solved. It says fidelity is a distinct axis of repair quality, and that axis can be measured and learned. That is much more actionable than another leaderboard where every patch is just pass or fail.

Harper The code is available in the repository called over-editing. I would run the benchmark on a few internal repair tasks, especially ones with weak tests, then compare the diff shape and review time. I would not ship a score as a hard blocker before checking where minimality penalizes legitimate maintenance.

Laura And I would put the preservation instruction in the agent's repair mode now, with a visible warning when it changes unrelated functions. Harper, that is almost a premium-branded clipboard, but this time I mean it affectionately.

Harper I knew the clipboard was coming.

Laura Good. We can leave it beside the diff and pretend that counts as governance. See you next time, Harper. I am keeping the clipboard, though.