Theoretical accounts attribute much of the success of deep neural networks to feature learning, in which training reshapes the representations within hidden layers. In this picture, a theory of deep learning must explain which features a nonlinear network learns, how its weights encode them, and how its neurons represent them. Yet, we still lack a general theory that answers these questions.

We study this question as part of PIRAMID’s Advancements in Learning Theory team , which uses statistical physics to build foundations for mechanistic interpretability and AI safety. Our progress and plans describe a theory agenda centred on hidden weight distributions and covariances that can record learned structure. The mean field sequence introduces this viewpoint by treating each neuron as one member of a population whose distribution must be self consistent. Statistical physics lets us connect the behavior of many interacting neurons with quantities describing the network as a whole, including its hidden weight distribution and prediction error.

We consider toy and solvable models to be important signposts on the road to a comprehensive scientific understanding of feature learning and generalization. When these models are built to the standard of statistical physics, they come with precise regimes of validity, error bounds, and universality classes. This makes it easier to say which conclusions carry over to real models and which should not. Rubin and collaborators give a compelling example of this in a nonlinear network, which we consider to be foundational for developing a more general theory of learned representations. In their solvable setting, they derive the hidden weight distribution and its covariance, with an explicit phase transition that signals a change in the hidden representation connected to improved generalization. We draw on their 2024 study of a feature learning transition and their 2025 theory of adaptive feature learning .

Working through these calculations by hand helped us develop a gears-level understanding of how the theory is built. We could see why the authors study particular quantities, which approximations make the calculation possible, and how the conclusions follow. Working through the calculations also shows where we can build on the model to develop a richer understanding. We wrote this post to share that understanding with readers who have no statistical physics background, and to provide context for recent work . Instead of the mathematically heavy adaptive field theory machinery of the original papers, we use a simpler cavity derivation that studies one neuron at a time, tracking the weight distribution after each major step to make the derivation easier to follow. We assume familiarity with neural networks, basic linear algebra, and probability.

Rubin et al.’s toy model describes feature learning in a network with two trainable layers, which learns a target that depends on one unknown direction in its input space. We keep the model narrow to make the qualitative picture and its derivation easier to follow:

The central prediction of the theory is a phase transition from a hidden weight distribution with no preferred direction to one containing populations aligned with the target direction. We can detect this learned direction in the covariance of the hidden weights, which acquires an extra rank-one variation along that direction. Once the hidden features align, the network can also represent the cubic component more strongly. The improvement in nonlinear prediction follows from a change in the representation, and the theory predicts a large reduction in the number of samples needed to learn it.

We begin with a network learning to predict a number from a high-dimensional input. We choose a target that depends on one direction and give the network examples without telling it which direction is useful. We can then study whether the network learns that direction and how learning it improves prediction.

We draw inputs from a Gaussian distribution with no preferred direction,

and choose an unknown unit vector . The target, which the papers call the teacher, depends only on the projection

To study how learning a direction helps with nonlinear prediction, we give the target both a linear and a cubic component,

Subtracting from removes its correlation with the linear component, since

The functions and are Hermite polynomials. Their orthogonality means that fitting the linear part alone leaves the cubic part unexplained, even though both depend on the same direction. The coefficient sets the cubic part’s strength.

The student network has two trainable layers and an odd nonlinearity,

Each hidden weight vector determines the input direction used by one neuron. We measure its alignment with the target direction by the overlap

Figure 1.1 shows the target and the hidden weight overlap we will track.

We expect the broad approach to extend beyond this setting. Learned structure appears as a change in the hidden weight statistics, which we read from the posterior.

The target uses one scalar projection. Its linear and cubic components share that coordinate. Feature learning appears when hidden weights acquire finite overlap with the target direction.

The linear part contains enough information to estimate the target direction. Consider the average

Only the component of along correlates with the target, so

The squared estimation error is , so examples identify to order one accuracy. The linear term supplies the direction; without it, this estimate has mean zero and does not provide the same route to learning the cubic target.

We still need to explain when the network puts the estimated direction into its hidden weights.

We now want to describe which weights the network favors after seeing the training examples. In the Bayesian model, we do so with a probability distribution over weights, called the posterior. We obtain it by combining a prior over weights with a likelihood for the observed targets.

We put independent Gaussian priors on the hidden and readout weights,

and use a Gaussian likelihood with variance on a training set of size ,

The hidden weight prior gives no direction a preference. The likelihood favors weights with smaller prediction errors. Multiplying the priors by the likelihood gives the posterior over all the weights,

We keep and explicit because their dependence on width is part of the model. Standard scaling takes and . The 2024 paper also uses a limit that reduces readout variance and likelihood variance together; the 2025 paper treats both standard and mean field scaling.

Equilibrated Langevin training, with the corresponding noise and weight decay, samples this posterior. We are studying that equilibrium distribution; the time needed to reach it requires a separate calculation.

The prediction error couples all the neurons through their sum, so we cannot yet calculate each neuron’s posterior on its own.

We first ask which feature would help one neuron fit the prediction error left by the others. Holding the other neurons fixed lets us study that question through a conditional posterior.

Remove one neuron from the network and call the prediction error left by the other neurons

For a candidate hidden weight , define its feature vector on the training set by

The selected neuron contributes , so the full residual is . In the network posterior, the other neurons’ priors are constant under this conditioning. The remaining terms are

To integrate over , we need the feature’s correlation with the residual and its squared size on the training set,

measures how well the feature matches what remains to be learned. measures the feature’s size, which we need to distinguish a useful correlation from a large feature. Write

A Gaussian integral over the readout weight now gives the hidden weight posterior,

Equation (3.1) is exact while the other neurons are fixed. The prior favors small weights with no preferred direction. The factor accounts for integrating over the possible readout coefficients, and the exponential rewards a feature that fits the residual. The denominator adjusts that reward for the feature’s size.

The fit reward depends on , so the signs and remain equivalent. The mean weight can stay at zero while an even moment such as records direction learning.

We still need the residual produced by the whole network. Self consistency will require that residual and the learned hidden weight distribution to agree.

The calculation uses the Gaussian integral after collecting the linear and quadratic terms in the readout weight. Expanding the squared error gives

then gives Equation (3.1) after removing factors that do not depend on .