Memory Is Not Free

More memory does not mean better results. I added a memory module to my agent and its success rate went up by 2 points. Then I repeated the exact same test, changing only the seed, and the score moved by up to 25 points from one repeat to the next.

Same 1500-token budget for both agents, 56% with memory versus 54% without. That 2-point edge sits deep inside the natural noise of the test itself, and almost no memory benchmark reports that second number.

Memory is not free.

The stored workflow gets injected into the prompt on every step, and those tokens are no longer available for reasoning. So "agent vs agent+memory" is the wrong comparison. The fair one is "agent vs agent+memory, both under the same token budget". Most published results never impose that budget, which means the memory agent quietly gets to spend more to reach its score.

So I measured it.

A small, reproducible setup: a deterministic tool-world, 10 multi-hop tasks, a local 35B model, 5 repeated runs (different random seeds, the dice roll behind each run), and a hard cap of 1500 tokens per task applied to both agents. One agent plain (the "vanilla" baseline), one with an AWM-style memory that stores solved workflows and replays them.

Vanilla: 54% mean success rate. Memory: 56% (+2 pts), and about 2% MORE tokens spent. The same test repeated 5 times: the plain agent scored anywhere from 30% to 80%, the memory agent from 40% to 70%.

Where does that swing come from? The model makes a small dice roll on every word it generates. Over a multi-step task those tiny differences compound into different trajectories, and with 10 tasks per test, every task that flips moves the score by 10 points.

So the memory effect is roughly ten times smaller than the noise. Land on a lucky repeat and you would publish "memory adds 20 points". Land on an unlucky one and memory loses.

Before you trust a memory claim: matched against what budget? How many repeats? Is the spread bigger than the effect? Are tokens reported next to accuracy?