Same 9B Weights, 19% vs 45%: the Scaffold Gap

Does scaffolding matter more than model selection for agent performance? And if so, by how much?

On SWE-bench Pro (731 coding tasks, the harder, contamination-resistant variant), swapping between Claude, GPT, and Gemini moves scores by a few points. Swapping the scaffold around the same model moves them by 10.

Claude Sonnet beat Claude Opus, 52.7% vs 52.0%. Both had scaffolds, but the team wrapping Sonnet designed better error recovery and context management.

And this goes all the way down the capability ladder. A developer took Qwen3.5-9B, a quantized model you can run on a laptop, changed only the scaffold, and went from 19.1% to 45.6%, above GPT-4.5-preview on the same test.

The adaptations were specific to how small models fail: write guards that prevent file overwrites, a 2,048-token thinking cap that forces the model to commit instead of deliberating forever, 500-token skill injections per turn instead of a massive static preamble.

So should you just keep adding more scaffold?

Pioneer Agent found that naive scaffolding degraded performance by up to 43 points. Tsinghua tested adding a verification step that checks the agent's output before submission, and it actually dropped scores by 8.4 points. The model started optimizing for the verification structure instead of the task, producing outputs that passed internal checks but missed the actual goal.

What separates good scaffolding from over-scaffolding comes down to a few components.

Error recovery is the biggest lever across every study I've read, worth +5 to 15 points. Most agents treat errors as terminal and give up. Making them recoverable, with the ability to go back to a working state, changes the game entirely.

Context compaction is the second one: as conversations grow, old tool outputs pile up and eat the context window. Summarizing older results while keeping recent ones at full fidelity is worth +3 to 8 points, and none of the 13 open-source agents I've looked at fully solve the problem of context growing until it overflows.

Planning-execution separation, where the agent splits "what to do" from "do it" with different tool access for each phase, adds another +5 to 10 points.

The pattern across all of this is a scaling law: scaffold impact is inversely proportional to model capability. The smaller the model, the more the scaffold is the product. The larger the model, the faster you hit the point where additional structure starts hurting.

And that curve shifts with every model generation, so your optimal scaffold today could become overhead tomorrow.

If you're building agents, try this before chasing the next model release: profile your current scaffold against those three dimensions. How does it handle errors? How does it compress context? Does it separate planning from execution?