Latent thinking in LLMs: reasoning without writing it down

Chain-of-thought made models smarter by letting them think out loud: the model writes its intermediate steps as words, then re-reads them. It works, but it's expensive — every reasoning step turns into tokens that have to be generated and held in the KV cache. So I started asking a different question: what if the intermediate thoughts stayed as continuous hidden states and never got decoded at all?

To dig into this properly rather than by intuition, I built latentGPT, a tiny (~0.5–1M parameter) latent-reasoning transformer that trains in minutes on a laptop. Scale isn't the point here. The point is reproducing the literature's central claim — accuracy rises when you spend more test-time compute by iterating a recurrent block in latent space — with the rigor the 2026 literature now expects.

#Two ideas, combined

latentGPT borrows from two places. COCONUT contributes the "continuous thought": the last hidden state feeds back in as the next input embedding and never becomes a word. Recurrent-Depth / Huginn contributes the factoring into prelude → recurrent core → coda, where the core loops an arbitrary number of times at inference but trains with a random iteration count and truncated BPTT so it generalizes past its training depth.

The prelude extracts context, a single recurrent core iteratively refines a latent state s (re-injecting context through an adapter at every step), and the coda decodes the answer. At inference I loop the core 1 to 16 times — that's the only "think longer" knob there is.

#Where the naive version breaks

A plain looped reasoner peaks and then collapses: add steps and accuracy climbs, then falls apart. The culprit is the recurrent operator's Jacobian blowing the state up. The fix is STARS — a penalty on the Jacobian's spectral radius that keeps the core contractive (ρ(J) < 1). In practice that means a double backward through attention (estimating the top singular value via power iteration plus JVP/VJP), which forces you onto the math SDPA kernel, since flash and efficient attention don't support double-grad. Training gets about 1–2% more expensive, but depth scaling stops being a coin flip.

Two more choices hold training together: random depth (1–8 steps per batch) so the model learns to refine gradually, and truncated BPTT so memory doesn't scale with the number of inference steps.

#How I check any of this is real

I don't trust a curve that just goes up. The benchmark is multi-task with a depth knob built in: pointer-chasing (follow a permutation k times), arithmetic chains, ListOps (nested MAX/MIN), and parity. Every task-by-depth cell gets exact-match accuracy with a Wilson 95% confidence interval. Comparison runs on two axes at once: against Qwen3.5-0.8B (about 800× larger) along the test-time-compute axis — latent steps versus CoT tokens at a matched budget — and against a parameter-matched baseline with no recurrence at all, so a win can't be chalked up to parameter count.

Plus diagnostics that go beyond the headline number: spectral radius (has to stay under 1), step-norm decay, and the cosine between consecutive steps (around 0.5–0.65 — a spiral trajectory, not an instant collapse to a fixed point), with logit-lens as a faithfulness probe.

#What it shows, and where it stops

With a stable core, accuracy really does rise with the number of latent steps, especially on deeper task instances, and the latent model beats its parameter-matched baseline — so the gain traces back to recurrence, not parameter count.

The limits are written directly into the repo, not buried in a footnote. There's a real trade-off between search and exact execution: latent reasoning wins on search-shaped tasks (pointer-chasing, ListOps) but loses to token-level CoT on arithmetic. Scaling is reliable but not monotone — an over-regularized core converges too early. And latent "thoughts" carry no faithfulness guarantee; it's easy to mistake them for reasoning in exactly the spot where the model is shortcutting.

That's the part of this topic I find compelling: latent reasoning isn't a universal win, it's a specific trade-off — most interesting under a tight budget, and on small models where memory is the bottleneck rather than FLOPs. That's what I want to study properly.