This post will probably not be very informative to readers who don't know what an SAE (Sparse Autoencoder) is.

An interpretability researcher once told me, "SAEs can never fail, they can only be failed." But recently, SAEs have been suffering failure after failure, and as a result, the field has grown increasingly pessimistic about SAE work. Korznikov et al.'s Sanity Checks for Sparse Autoencoders: Do SAEs Beat Random Baselines? purports to be an example of such failures.

However, I have some disagreements with its methodology. The paper presents a synthetic setup with known ground truth features, and claims that SAEs fail to recover features in the "simplest toy setting with fully aligned hyperparameters." In this post, I argue that their setting is not forgiving to SAEs, as the authors imply, but is instead actively hostile in ways that real settings are not.

Sanity Checks for Sparse Autoencoders: Do SAEs Beat Random Baselines? asks two important questions:

This post will be focused on their answer to the first question.

The authors train an SAE on a synthetic toy model, with known ground-truth features, and according to their abstract, SAEs only recover 9% of true features. On the face of it, this seems pretty damning!

Now, earlier toy model experiments did show that SAEs could recover ground truth features, but the authors argue this is because they used a small expansion factor, where there were only twice as many "true" features as there were model dimensions, and that this is unrealistic. Their setup uses an expansion factor of 32, which matches real-world use cases much more closely.

They don't actually construct a synthetic neural network; instead, they create synthetic activations directly. First, they create their feature vectors, by sampling 3200 vectors from the unit sphere in 100-dimensional space. To create an activation, they just select about 20 feature vectors at random, scale each active feature by a magnitude drawn from LogNormal(0, 0.25), [1] and sum the scaled vectors.

They test two different regimes, one where each feature is equally likely to be active, and another where feature activation frequency has a long tail, so some features fire more often than others. (We'll focus on the constant regime for the most part)

They match their SAE to the setup. There are about 20 expected active features, so they target L0 = 20. There are 3200 true features, so they create 3200 SAE latents. Can the SAE, when given every possible advantage, recover the true features?

These results are, again, pretty damning! They claim that the SAE achieves high reconstruction accuracy while only recovering 3/3200 true features—personally, I disagree that 0.67 explained variance is "high reconstruction accuracy". [2] However, the SAE clearly only does slightly better than random guesswork at recovering the true features.

Now, one obvious objection is that L0/d_model here is 0.2, which is unusually dense. I'd expect there to be a lot of interference here, and in the real world L0/d_model is probably going to be closer to 0.05. Also, 100 dimensions is just not a lot, and maybe you can't really get blessing of dimensionality from it like you would in real models.

A less obvious objection is that if you scroll all the way down to Appendix B, they describe some other SAE architectures they used for this task, like TopK and Matryoshka-on-BatchTopK.

In contrast to BatchTopK, TopK recovers 99.9% of features . [3]

So, I did what anyone would do, and started replicating the paper. They didn't actually include all the hyperparameters they used, [4] so I made reasonable assumptions.

I was able to replicate this result! TopK recovered 99.6% of features, [5] and the other architectures failed miserably. [6]

This is kind of strange, if you think about it. TopK and BatchTopK are the same architecture, with the only difference being the selection rule. [7] Such a dramatic gap is really weird.

Peeking a bit closer at my training run, in every batch, most latents never fired. That means a small group of monopolist latents were dominating every time. By step 5,000, the same ~6% of latents took over, and they remained in charge for the remaining 295,000 steps. In comparison, with TopK, after the first 20k steps, >99% of latents activated on every batch.

This is the standard dead latents problem: There's no gradient on latents that aren't selected, so once they fall behind in performance it's hard for them to improve. A small group of generalist latents dominate before specialists have the chance to form. The standard fix, of course, is an auxiliary loss, AuxK. But even with AuxK, the dead latent fraction barely budges. I tried sweeping different hyperparameters, but that didn't help either. I even tried resampling , and that did nothing.

Looking closer at the difference between TopK and BatchTopK gives us a hint. In theory, TopK should also be suffering from this problem. The fact that it doesn't suggests that the problem lies in sample-level variance. Certain samples in BatchTopK could receive a significantly larger amount of latents, while other samples are starved. An obvious thought is that input vectors with greater magnitude get more latents assigned to them than those with less magnitude. Checking the data, this does appear to be the case. So what if we normalized the input?

Bussmann's original BatchTopK implementation actually normalized per-sample, folding back in the original magnitude and mean during reconstruction. [8] I copied this method, and lo and behold! BatchTopK achieved 99.3% recovery, comparable to (my) TopK's 99.6% recovery. This also fixes JumpReLU with 97.2% feature recovery.

But while this was the original design, per-sample normalization isn't common anymore! GemmaScope and SAELens don't use it (although they do exclude outliers like BOS tokens), but they still work, without such drastic dead latent rates.

Despite the paper's claims, this synthetic model doesn't demonstrate the failure of SAEs to capture real features. Real SAEs don't have such drastic dead latent rates. Real SAEs are much better at reconstructing in-distribution data than out-of-distribution data. The SAE trained on synthetic data seems to be a fully general vector encoder, reconstructing random vectors with matching norms just as well as it does in-distribution data.

So all this model demonstrates is that this particular synthetic toy model is hostile to many SAE architectures (but not TopK). This is a weakness of these SAE architectures, in that they are not very robust to these types of environments, but this particular failure mode has an obvious signature (the dead latent rates).

I hope that I have demonstrated sufficient evidence of the failure of Sanity Checks ' synthetic environment. The rest of this post will be dedicated to my personal explorations trying to understand precisely what it is about their environment that causes this divergence from real SAEs. [9]

Part of it is the density/dimensionality. If we raise d_model up to 800 while keeping the expansion factor of 32 (which means we're now using 25,600 features and latents), L0/d_model shrinks to 0.025, which is much more realistic. Under these conditions, TopK with auxk gets 99.9% recovered, and BatchTopK with auxk gets 53.5% with 46.3% dead latents. JumpReLU recovers 28.3% with 71.6% dead. So, it explains some of the difference, but not all of it.

While resampling didn't work at d_model = 100, when we try it at d_model = 800, it succeeds, raising BatchTopK's recovery to 85.6%. Anthropic resamples using 0.2x average active encoder weight norm, but when we try it with 1x instead, it actually raises recovery to 97.8%, close to TopK's.

So does this mean we've solved it? No! Real SAEs don't need resampling to avoid killing off half their latents.

I could continue to raise d_model, and I expect with More Dakka I could narrow the gap further. But GPT-2 only has 768 dimensions, and SAEs trained on it work just fine (in that they don't have abysmal dead latent rates). So, we should look for other divergences from real models that our synthetic setup has, while keeping d_model at 800.

An obvious guess is that our environment has unrealistic variance in per-sample magnitude, causing architectures like BatchTopK to perform poorly, and that real models have lower variance. While real models do have outlier tokens like BOS, SAEs typically exclude these, so I sampled GPT-2 activations on OpenWebText at various layers, excluding BOS. I subtracted out the dataset mean, since the decoder bias will usually be pretty close to it and is often initialized to the mean (or the geometric median).

As you can see, our toy model falls within the range of real GPT-2 activations, and in fact is on the low end. This means magnitude variance cannot explain the gap.

Why doesn't everybody normalize their SAE inputs? When you normalize, you're tossing away the magnitude, which could perhaps serve as an important signal. For example, maybe some features naturally activate with more magnitude than others, and total magnitude serves as a signal for that. So, maybe giving different features different natural "loudness" would improve performance. [10]

To simulate this, I draw a "loudness" variable for every feature from LogNormal(0, 0.5 sigma). Then, when creating activations, I scale each active feature not only by their activation strength (drawn from LogNormal(0, 0.25 sigma)), but also their loudness.

Running this with BatchTopK, we get 19.6% of features recovered, with 78.3% dead latents. Instead of improving performance, it actively degrades it. So feature loudness alone cannot explain the gap. (Also, my motivation for this was really dumb, see footnote 10 for why)

Real LLMs might have many more features than their SAEs have latents. Perhaps, this gives latents more opportunities to grab onto "free" features. Alternatively, this might be distracting, making each individual feature appear less often, or trapping latents in an unfortunate middle ground where they can never lock onto a feature closely enough to converge on it.

So, I ran BatchTopK SAEs on setups with 4x and 2x more features than latents. We're still at 800 dimensions, 25,600 latents, and on average 20 features are active at a time.

On 4x, we recover 0.1% of features (This, of course, means 0.4% of latents captured features, since there are 4x more features than latents), with 58% dead latents. Even per-sample normalization [11] performs similarly poorly, recovering only 0.2% of features. They both reconstruct data only slightly better than noise.

On 2x, we recover 16.3% of features (which means 32.6% of latents capture features), with 59.6% dead latents. Per-sample normalization actually works this time, though, recovering 49.1% of features (so, 98.2% useful latents), and only 1.7% dead latents.

So, having more features turns out to be distracting, in this case. [12]

Real LLMs might have higher average active feature counts than the L0s of their SAEs. What if we tried doubling the active feature count to 40, while keeping target L0 the same?