Why ML Benchmark Claims Can't be Compared

View on

Léo Dreyfus-Schmidt, former VP of Research at Dataiku, explains why rigorous benchmarking is essential to making meaningful progress in machine learning. When everyone claims state-of-the-art results without apples-to-apples comparisons, it becomes difficult to know what actually works. Léo shares why his team took a more principled approach to active learning, focusing on: • Why benchmarking is fundamental to scientific progress • The problem with incomparable state-of-the-art claims • How better metrics can help evaluate active learning over time • Why rigorous evaluation matters even when it isn’t the flashiest research