A Plain Hyperparameter Search Beat 15 Frontier Agents

Give an AI agent a GPU and an LLM. Can it make inference fast on its own?

Researchers tested it with 15 frontier agents. Each got 1 H100, a 7B model and 2 hours. A simple hyperparameter search beat them all.

What were the agents asked to do? Deploy an inference server for Mistral-7B, then make it fast on three bottlenecks: the wait before the first token, the speed of every following token, and the throughput (how many of 64 parallel requests get served per second). A fourth scenario mixed all three. The agents came from the Claude, GPT, Gemini and GLM families, each running its default coding harness. The hyperparameter search is a much simpler program. It tries one server setting after another and keeps the fastest.

How far did the agents get? 2 out of 3 runs shipped a valid working server. The best run came from Claude Sonnet 4.6 and reached 8.08x the speed of a raw PyTorch baseline. 7 of the 15 even beat vLLM's default configuration.

So where did the agents lose the race? The loop still finished ahead, at 11.53x. The agents stopped exploring early: 90% of runs tried at most 1 more configuration after their initial success. 96% brought up quantization, 97% brought up chunked prefill, but almost none ran a real experiment with either one. 94% stuck with vLLM even though the prompt offered TensorRT-LLM. Not one run ever shipped it.

So the agents never found the speed? They did, and this is the real lesson. Along the way, their runs touched configurations worth 12.34x on average, more than the search's final score. But those wins rarely survived to the finish line. The best setup got broken, overwritten, or never shipped as the final server. The agents lost less at searching than at keeping what they found.

Would more time fix it? No. Runs given 4 or 8 hours flatten or slightly regress. Discipline fixes it. When the prompt imposed a protocol (baseline first, 1 change per experiment, log everything), Claude Opus 4.7 jumped from 2.25x to 8.61x, and its valid runs went from 5 to 11 out of 12.

What do you do with this? Let an agent install and debug the server, then let a hyperparameter search find the fastest settings. Give the agent a protocol: measure a baseline, change 1 setting at a time, log every result. And ask it to track its best configuration and deploy that one at the end, not the last one it tried.

Source: InferenceBench (ELLIS Institute Tübingen / Max Planck Institute), inferencebench.ai