The competition

There is a lot of evidence that Neural Architecture Search can produce excellent models capable of ML tasks on well-known datasets, that is, datasets like CIFAR-10 and ImageNet where years of research have created a set of best practices to follow to achieve good results. This leads us to wonder, are we overfitting our NAS algorithms to well-known datasets?

We want to investigate the "real-world" use case of NAS, that is, searching for a state-of-the-art architecture on an entirely novel dataset. In such a case, there is no existing set of best practices to build from, nor extensive research into optimal architectural patterns, augmentation policies, or hyperparameter selection. In essence, we are asking how well NAS pipelines can work “out-of-the-box” with little-to-no time for tuning. To explore this, we have designed this competition to evaluate how well a NAS pipeline fares over unseen novel tasks and datasets.

In this competition, you will be asked to design a NAS pipeline, namely a data processor, a NAS algorithm, and a training policy. This pipeline will then be evaluated on our machines over a number of completely novel datasets: the challenge is to design a pipeline that is flexible enough to be competitive regardless of the datasets provided to it.

This year we are running the competition at AutoML 2026 (2026.automl.cc). AutoML is more alligned with NAS than previous venues, and we think it will be a more suitable home for the competition.

The competition will be run in three phases:

• Phase 1: You download our framework , and develop your method locally. Open Now!
• Phase 2: You send us your submission, and we do a shorter test run on our system.
• Phase 3: We rerun your last working Phase 2 submission for a longer final run.

Results are to be announced at AutoML 2026 in Ljubljana, September 28th - October 1st, with finalists being informed in August.