Design of the 2015 ChaLearn AutoML challenge
- Kristin Bennett ,
- Isabelle Guyon ,
- Gavin Cawley ,
- Hugo Jair Escalante ,
- Sergio Escalera ,
- Tin Kam Ho ,
- Núria Macià ,
- Bisakha Ray ,
- Mehreen Saeed ,
- Alexander Statnikov ,
- Evelyne Viegas
Proceedings of International Joint Conference on Neural Networks (IJCNN), 2015 |
The Automatic Machine Learning (AutoML) contest for IJCNN 2015 challenged participants to solve classification and regression problems without any human intervention. Participants’ code was automatically run on the contest servers to train and test learning machines. However, there was no obligation to submit code; half of the prizes were won by submitting prediction results only.
Datasets of progressively increasing difficulty were introduced throughout the six rounds of the challenge. Participants could enter the competition in any round. The rounds alternated phases in which learners were tested on datasets participants had not seen, and phases in which participants had limited time to tweak their algorithms on those datasets to improve performance.
This challenge pushed the state of the art in fully automatic machine learning on a wide range of real-world problems. AutoML was run on Codalab which is an open-source platform, powered by Microsoft Azure that provides an ecosystem for conducting computational research in an efficient, reproducible, and collaborative manner. The platform remains available beyond the termination of the challenge.