One test starts with a codebase. The other doesn’t.

DeepSWE gives an agent an original engineering task in an existing open-source repository. The agent has to figure out how the project fits together, make a change, and get the requested behavior working. The checks don’t demand the same patch a human would write. That’s close to the job you have when you need to add a feature to software you already ship.

ProgramBench gives the agent a compiled program and its documentation, but not the original source. The agent writes its own implementation, and hidden tests compare its behavior with the reference. That’s closer to replacing a utility you can still run but can’t edit. Passing those tests doesn’t tell you whether you’d want to maintain the replacement.

The test setup matters, too. The published DeepSWE leaderboard uses the same agent software across models for consistency. A score from another setup isn’t automatically a like-for-like comparison. Look for the benchmark version, model effort, and the tools the agent could use before reading a small lead as a reason to switch.

Try the job you’d actually hand over

Pick one real task and run it in a disposable copy. For maintenance, use a bug you can reproduce and check that the fix leaves neighboring behavior intact. For a replacement, give it the manual and a few examples. Hold back some awkward inputs and compare its output with the original.

If you’re comparing models, keep the agent software, tools and budget the same. Note what passed, what needed a correction, and how long it took. If the new tool works on the easy example but changes output a customer depends on, you haven’t found a replacement yet.