Φ-Bench: Can Large Language Models
Engineer the Infrastructure That Powers Them?

Insight Under a fixed budget with best-of-k scoring, all six models converge to nearly the same solution space; final rank is decided not by which optimization is discovered, but by whether each round reduces uncertainty. Strong models run a closed entropy-reducing loop — single change → isolated validation → bottleneck localization — so information compounds along the path, while weak models random-walk within the noise band. The same iteration budget compounds for the former and is mere noise for the latter.

Positive opus-5 (reward 0.683, dev 1.272; first on both). Three habits cut entropy each round: (i) every submission is a single-variable controlled experiment, with failures repurposed for diagnosis — a failed full-decay schedule → the round concludes effective training steps are the true bottleneck → a fast late-stage decay, landing the largest single gain (−0.06); (ii) noticing the online ¼-budget selection metric is noisy and misaligned with full-budget scoring, it builds a more stable local eval to pre-screen candidates; (iii) it alone tunes at full budget, catching knobs invisible under short budgets.

Negative qwen3.7-max (0.479, last) / dsv4pro (0.567) — both add entropy. qwen spends ~14 mid-run rounds on single-knob tweaks inside the noise band with near-zero information gain, reaching the decisive layer-reduction axis only in the penultimate round — budget exhausted while the curve is still descending. dsv4pro changes several params per round, so failures cannot be attributed and must be rolled back, ultimately resubmitting the same config to resample noise under best-of-k — luck substituting for information.