CODI is a model using latent reasoning. Previous work by Cywiński et al. (2025) has found that CODI's latent recurrent states (“thoughts”) do causally carry a computation's intermediate values, in a way that can be indirectly detected with LogitLens, though the thoughts themselves remained opaque. We here ask whether it is possible use activation verbalizers (AVs: NLAs and AOs) to still decode them.

To this end, we train a Qwen3-8B CODI to connect to current AVs. We find that LogitLens and J-lens read values only from late layers, but AVs recover the final answer and intermediate values well above chance, including from the thoughts themselves. This suggests that the values are in fact encoded, but requires more expressivity to decode. However, we find that editing the surfaced values, rarely steers the model output towards the corresponding counterfactual answer.

Conventional LLMs generate their chain-of-thought through a sequence of autoregressive text tokens ("verbal CoT"). In contrast, latent reasoning models use a sequence of vectors, which can amount to a reserved vocabulary, or consist in entirely opaque vectors (see Zhu et al. (2025) for a review).

Currently, verbal CoT is the major source of monitorability for a model [1] . Latent reasoning however would significantly lower the monitorability since the vector's content is not obvious anymore. Therefore, it is of interest to try to interpret latent reasoning in current model organisms. In this post, we study CODI, which is trained with comparatively weak supervision on its latent CoT, suggesting it might have considerable freedom to develop opaque reasoning patterns.

CODI ( Shen et al. (2025) ) finetunes an LLM to reason in a fixed budget (here, ) of "latent" (i.e., continuous valued) thoughts instead of a verbal CoT. At each step, the final hidden state is projected and fed back as the next input embedding, . Treating latent vectors as opaque, this increases the architectural upper bound on opaque serial depth (the computational depth between interpretable intermediate steps, Brown-Cohen et al. (2026) , Emmons et al. (2025) ).

Continuous feedback extends CODI's available opaque computation path. Left: verbal CoT does autoregression on legible text tokens . Right: latent CoT works similarly, but passes recurrent latent thoughts . is the residual stream at position and layer , and is the input embedding.

Training is done via self-distillation: the same model acts as teacher, reading a verbal CoT from the dataset, and as student. A single model undergoes three passes per batch: a teacher-forced pass over the supplied verbal CoT and answer, a student pass answering after the latent thoughts, and a direct pass answering with neither (noCoT). The verbal CoT is distilled into the latent rollout by aligning hidden states only at the last token before the answer at every layer, letting the representation develop at other states only through indirect supervision.

We here follow Cywiński et al. (2025) and train on 5 epochs on GSM8k-Aug-NL (the GSM8k train set augmented to 385k problems with GPT-4-generated chains, here in the variant whose CoT steps are written in natural language rather than equations, Deng et al. (2023) ) plus CommonsenseQA.

Interpretability of latent reasoning is a quickly developing field. The following works study CODI, and are particularly relevant to this post:

We find that on the trained checkpoint, latentCoT improves over the noCoT mode, and roughly matches the verbal mode (using greedy decoding). Skipping the latent thoughts altogether drops GSM8k test accuracy from 52% to 47%. This uplift is much smaller than in the original CODI paper, where latent thoughts more than doubled GPT-2's accuracy over noCoT training (from about 19% to 44%) and roughly matched verbalCoT [2] .

CODI's latent CoT is necessary on a fraction of problems. We here focus on the subset of "load-bearing" problems highlighted in red, where CODI needs its latent thoughts (it cannot solve the problem in a single pass).

Note that even within the base-needs-verbal-CoT set, CODI solves most of the tasks it gets right without any CoT (overall, about 85% of the problems solved with latent thoughts are also solved when skipping them). This is in line with the latent-unnecessity findings of Dilgren and Wiegreffe (2026) .

The advantage of latentCoT over noCoT in CODI is evidence that the latent CoT is doing something . To further analyze this, we do a series of ablations on the CoT, including removals, reorderings, foreign traces from other problems, always regenerating the later thoughts autoregressively after an intervention.

Even on load-bearing rows, ablations are not completely catastrophic. Fraction of load-bearing problems still answered correctly. Single-thought replacements are averaged over the six positions.

The fact that ablations retain some problem solving ability is surprising. This suggests that the model might not mainly do sequential autoregressive computation.

So far, we have analyzed where the CoT is load-bearing, and manipulated parts of it through ablations. Here, we now study what the content of the latent CoT is that underlies these results.

Cywiński et al. have used LogitLens to decode the residual stream at the thought positions. Here, we try this as well, applying both LogitLens ( nostalgebraist (2020) ) and J-Lens ( Gurnee et al. (2026) ) to both the residual stream as well as the latent thoughts at the thought positions .

Pooling the top-5 tokens over all six thought positions, both lenses read values well above chance, but only towards the output layers (see the appendix on the full LogitLens and J-lens top-5 tokens), consistent with Cywiński et al. However, the thoughts themselves stay opaque.

LogitLens (left) and J-Lens (right) fall short on the latent thoughts . On an example train problem, neither lens decodes anything from the recurrent thoughts , and the residual stream shows only the operation ("half"). Near the output, the computed intermediate 10 appears at as number words (" ten", "Ten", 十) and the answer 5 at , both at the first thought (full layers positions grid in the appendix).

We next use more expressive interpretability techniques to study whether we can still surface problem contents.

Natural language autoencoders (NLAs, Fraser-Taliente et al. (2026) ) autoencode activations via a natural language bottleneck. This incentivizes the bottleneck to verbalize content that helps reconstruction, in an unsupervised manner.

We here continue training of an existing Qwen3-8B NLA to verbalize the residual stream as well as the thoughts at CODI's thought positions . Alongside the reconstruction quality (FVE, fraction of variance explained) we score the pooled descriptions of all six thought positions on 60 held-out problems per checkpoint.

Verbalization performance increases with falling reconstruction loss. The readers verbalize a single vector per sample, drawn from all six thought positions in training; in the eval scans, all six positions are verbalized and scored jointly. Solid lines score containment of the gold final answer among the values mentioned in the verbalizations, dashed lines whether any gold intermediate other than the final answer is mentioned (values LLM-extracted without access to the gold). Faint horizontal lines: LogitLens and J-lens at each reader's own layer, on the same questions with the same criterion (for they are at zero).

Interestingly, we achieve much lower reconstruction error (<2% on ) than is typically reported for NLAs, yet verbalization performance stays modest for the final answer. This can be explained by a narrow task domain where the vectors are relatively low-dimensional and thus easy to reconstruct (see the appendix on thought dimensionality).

We then conducted an automated analysis of verbalizations over the course of training and give examples below.

Selected verbalizations by the reader (top) and the reader (bottom) of the second thought of the same problem, before (left) and after (right) reader training. Each cell is one sample at the training temperature ( ). In every sample, the verbalizations are genre prose, yet after training both end on the gold answer 12 as their text's "final token", while both samples write out the problem's equation after training, one with a wrong total.

So far, we have analyzed the thought content via decoding. But before in the Ablations section we had seen that certain interventions on thoughts leave the answer correct. Here, we try to reconcile these facts by targeted interventions on the thoughts, including using the NLA.

Schema of the intervention structure . We either intervene on the latent thoughts , a given residual stream layer , or an entire residual stream column , which includes the fed thought.

A way of doing an intervention is to consider pairs of problems whose tasks are identical, but differ in numerical values ("twin" problems, e.g. 3+6=? and 2+5=? are twins of an integer addition task). Transplanting thoughts from a donor problem should then change the computation/answer according to the donor values. We patch the original problem's thoughts into its twin's rollout and score how often the twin adopts the original's answer. We filter problems to the ones that CODI cannot solve without the latentCoT to select the subset where thoughts are load-bearing.

We patch either the thoughts alone or an entire "column" of residual stream , which transfers more of the donor CoT's state. The column includes the fed thought: is replaced together with all 36 residual streams at that position.

Transplanting thoughts between twin problems barely transplants the computation (left), while transplanting full residual stream columns considerably changes it, spread over the early thoughts (right). Unfilled bars are a skyline, i.e. the fraction of held-out problems where the NLA read of the respective fed position ( left, right) semantically matches any gold intermediate (the 1–6 tick shows the mean over the six single-position reads).

Interestingly, answer adoption is small for single positions, and considerably smaller for than for full columns, despite specifically considering load-bearing rows. Since a transplant also makes the twin lose its own answer in most cases (it keeps it in well under half of the pairs), CODI seems to require more than the latent CoT's "shape": the content matters, but cannot be transplanted via alone, presumably because the rest of the residual stream is recomputed from the twin's own question. Within , the effect comes almost entirely from the third thought: transplanting alone disrupts and transplants about as much as transplanting all six, while transplanting or has no effect at all. The third thought is also where the NLA reads intermediates best, in line with Cywiński et al. (2025) , who found the third latent vector to store an intermediate value.

The NLA, which we previously had used for decoding, can also be used as an intervention method. To do so, we take the NLA verbalization, edit its text by replacing a numerical intermediate value with the corresponding value of the twin problem, reconstruct the activation, and patch it back into the chain-of-thought (at all six positions jointly).

Example of an NLA intervention, replacing 20 with 40. The edited verbalization then gets reconstructed via the activation reconstructor (AR), yielding z'=AR(v'), v'=edit(v), v=AV(z). Chosen is a problem where the CoT is load-bearing.

We measure then whether CODI's answer after patching the reconstructed modified activation is changed, and whether it adopts the answer that should have resulted if we indeed edited the causal path of the computation.

Editing the value in the thought's verbalization rarely implements the counterfactual. Fraction of load-bearing twin pairs that keep their own answer (gray) or adopt the counterfactual answer (blue), for (left) and (right).

For , edits disrupt the computation more than a random perturbation of the same size, but they adopt the counterfactual answer in only a handful of pairs (7%). This is, however, not far below what any transplant achieves here: reconstructing from the donor's own description adopts the donor's answer in 12% of pairs, as often as patching the donor's raw (previous section). For , the edit is not distinguishable from a random perturbation and adopts the counterfactual answer in only 2% of pairs, against 19% for the donor's description.

In this post, we have trained and studied a CODI model on Qwen3-8B on GSM8k data. Like previous CODIs, simple methods like LogitLens fail to decode the model's latent recurrent thoughts . However, more expressive activation verbalizers, in particular NLAs, are able to surface intermediates and answer content from the thought positions, in particular from the residual stream, though the final answer only at a modest rate.

When doing interventions, we found that the residual stream taken as a whole over layers carries computation that will guide the model to a different answer, again more so than just from the thoughts . Replacing the thoughts , raw or edited via the NLA, however induced hardly any answer adoption, even though it did disrupt the answer.

Overall, this suggests that this CODI's thoughts are 1) somewhat more opaque than the residual stream, 2) do not form a clear computational substrate as an "idealized" CoT just doing plain arithmetic on the intermediates might.