Second Place at GECCO 2026: The Benchmark Design Challenge for Continuous Optimisation

Second place — GECCO 2026 Benchmark Design Challenge for Continuous Optimisation. This note has been updated again, now that the results are official: GECCO issued certificates, and I received one for 2nd place. I participated as boris_almonacid, one of six entries. AC_TUWien won 1st place outright; the other five entries — Ananta Shahane, Andrew Gourley, Boris L. Almonacid (me), Jakub Kudela, and Rohit Salgeta — were jointly awarded 2nd place.

Most competitions in this corner of computer science ask you to solve an optimization problem. The BenchDesign competition, organized by Diederick Vermetten and Niki van Stein for GECCO 2026 (July 13–17, San José, Costa Rica, hybrid), flips that around: the task is to design one.

Entrants submit a set of 25 deterministic benchmark functions — bounded to [-5, 5], scalable across dimensions, written to a fixed function signature — and they’re scored not on how solvable the functions are, but on how well they separate five reference optimizers (CMA-ES, Differential Evolution, PSO, BFGS with restarts, and COBYLA), each working within a budget of 5,000 evaluations per problem. A good submission isn’t an easy problem or an impossible one — it’s one where the five algorithms actually disagree, revealing something real about what makes optimization landscapes hard, diverse, or distinctive for a given method. Full technical details live on the IOHprofiler BenchDesign track page, with the reference implementation in Vermetten’s BenchDesign repository.

The submission — Experiment: The Benchmark Design Challenge for Continuous Optimisation — is published on Figshare alongside the rest of the work on this site.

What the competition actually measures

The score is a rank-distribution metric: for each of the 25 submitted problems, the five reference solvers are ranked 1 (best) through 5 (worst), and a good suite is one where those ranks spread out — different algorithms winning and losing on different problems — rather than one solver dominating everything. Submissions are scored twice: once during training, against the exact solver configurations organizers published in advance, and once during a held-out test, where those same solver families reappear under different parameterizations the organizers kept hidden until after submissions closed — CMA-ES becomes a reconfigured ParametrizedCMA, DE becomes quasi-oppositional QODE, PSO becomes ConfiguredPSO, BFGS is swapped for the derivative-free Powell, and COBYLA becomes RCobyla with restarts. The gap between a suite’s train score and its test score is effectively a direct measurement of overfitting to the specific algorithm settings used during development.

Results

The organizers, Diederick Vermetten and Niki van Stein, presented the results as a poster at GECCO 2026. Diederick confirmed there’s no official version posted alongside the competition and gave permission to share a picture of it here:

The award itself is binary — 1st place to AC_TUWien, 2nd place shared jointly by the other five entries — but the poster also breaks down each entry’s actual score, which is worth including since it’s more informative than the placement alone:

Team	Train	Test	Change
AC_TUWien (1st place)	2.6	2.8	+0.2
boris_almonacid (2nd place, best score of the group)	2.2	1.8	−0.4
AndrewGourley (2nd place, best score of the group)	2.2	1.8	−0.4
rohit_salgata (2nd place)	0.6	1.4	+0.8
Ananta_Shahane (2nd place)	2.1	1.0	−1.1
jakub_kudela (2nd place)	2.8	1.0	−1.8

Within the shared 2nd place, my submission tied with AndrewGourley for the best test score of the group: a 0.4-point drop from train to test, a moderate rather than large generalization gap. AC_TUWien is the standout result regardless of the tie: their score improved from train to test (2.6 → 2.8), meaning their suite kept — and arguably sharpened — its ability to tell algorithms apart even once the specific solver configurations changed underneath it. A couple of other entries saw a larger gap between train and test scores than the field average, which is the kind of overfitting-to-configuration signal this competition’s train/test split is specifically designed to surface — a useful reminder for anyone iterating on a suite that generalization, not just training performance, is the thing being measured.

Takeaway

This submission generalized reasonably from train to test and matched the second-best score in the field, but the gap to first wasn’t really about difficulty — AC_TUWien’s suite apparently gave every reconfigured algorithm a more differentiated rank signature than this one did. I only have the results poster to go on, not the raw per-problem data, so I’ll leave it there rather than over-read the charts.