How Neural Networks Learn Features: Low-Degree Correlations and Collective Dynamics

I will trace a line of work on how neural networks learn features, from hidden directions in two-layer models to intermediate representations that let deeper networks learn complex targets with far fewer samples. A unifying theme across these works is that neural networks learn features through low-degree correlations, which led us to Neural Low-Degree Filtering, a spectral algorithm that describes early layer-wise learning on real data, and its extension to joint training of layers. I will close by sketching promising approaches to understanding the full dynamics of hierarchical learning across layers, drawing on ideas from our work on a rigorous version of the dynamical cavity method.