Implicit Regularization in Deep Learning: A View from Function Space

  • Aristide Baratin ,
  • Thomas George ,
  • César Laurent ,
  • Devon Hjelm ,
  • Guillaume Lajoie ,
  • Pascal Vincent ,
  • Simon Lacoste-Julien

ArXiv preprint

We approach the problem of implicit regularization in deep learning from a geometrical viewpoint. We highlight a possible regularization effect induced by a dynamical alignment of the neural tangent features introduced by Jacot et al, along a small number of task-relevant directions. By extrapolating a new analysis of Rademacher complexity bounds in linear models, we propose and study a new heuristic complexity measure for neural networks which captures this phenomenon, in terms of sequences of tangent kernel classes along in the learning trajectories.