Gradient Dissent · Thu 22 Oct 2026 · NVIDIA's looped models

The loop gate sets a step size for every token and every channel

In a looped model the same middle block runs again and again. Each pass proposes a new state, and the gate decides how far to move toward it. It decides separately for every token and every channel. The figures respond to taps and drags.

0.35

every step size when training starts

\(\sigma(0)\cdot\operatorname{softplus}(0) = \tfrac{1}{2}\ln 2\)

≈0.2%

of the model's parameters

\(d^2 + 2d\): 9.4M at \(d = 3{,}072\)

−0.025

one-loop loss from the gate alone

1.660 against 1.685 on a 0.5B model

1 · Each loop moves every channel part-way toward a proposal

\[ s_i = s_{i-1} + \beta_i \odot \left(r_i - s_{i-1}\right) \]

\[ \beta_i = \sigma\!\left(W_\beta\, \tilde s_i + b_\beta\right) \odot \operatorname{softplus}(\gamma) \]

\(s_{i-1}\): the state before loop \(i\), a vector of \(d\) numbers for each token. It starts as \(h_{\text{in}}\), the input block's output.
\(\tilde s_i = s_{i-1} + h_{\text{in}}\): what the loop block reads. The input is added back on every loop.
\(r_i = \mathcal{R}_\theta(\tilde s_i)\): the loop block's proposal for the new state.
\(\beta_i\): the gate, one step size for each token and channel. \(\odot\) multiplies channel by channel.
\(W_\beta\) (\(d \times d\)), \(b_\beta\) and \(\gamma\): the gate's only parameters, shared by every loop.

Read the gate from right to left. \(\operatorname{softplus}(\gamma)\) is a learned cap for each channel. The sigmoid, between 0 and 1, says how much of that cap this token uses on this loop, and the gate matrix \(W_\beta\) computes it from the whole state.

2 · The same block and the same gate run k times

Loops

Koi-4B: 14 input layers, 24 in the loop, 14 output layers.

At one loop every layer runs exactly once, so the model costs the same as an ordinary 52-layer model. Each extra loop adds 24 layers of compute and no parameters. The arc over each block is the old state, which the gate blends with the block's proposal.

3 · One step: β = 0 keeps the state, β = 1 replaces it

Drag the blue and orange points.

β for channel 1 0.35 β for channel 2 0.35

Same β for both channels

0 keeps, 1 replaces, anything between blends. Plain looping is \(\beta = 1\): the proposal overwrites the state.
Each channel steps on its own. Untick the box and give the channels different βs: the new state leaves the straight line from \(s\) to \(r\).
Above 1 overshoots. That needs a cap \(\operatorname{softplus}(\gamma)\) above 1. The formula allows it, but the paper doesn't report the trained caps.

4 · The gate matrix reads the whole state to set each channel's step

Row \(j\) of \(W_\beta\) is a detector for channel \(j\). It takes a weighted sum over all \(d\) channels of \(\tilde s\): a large positive sum opens the gate, a large negative one closes it. The toy below has \(d = 8\); Koi-4B has \(d = 3{,}072\).

Token

\(\tilde s\) across the top, \(W_\beta\) below, one row per channel

This token's step \(\beta_j\), inside its cap

Channel

Change channel 3 of the state 0

Toy numbers with \(d = 8\), made up to show the arithmetic. They are not the paper's weights.

At initialization every step is 0.35. \(W_\beta = 0\) and \(b_\beta = 0\) put every sigmoid at 0.5, and \(\gamma = 0\) puts every cap at \(\ln 2 \approx 0.69\). Training starts from one uniform step and learns where to differ.
Every row reads every channel. Move one channel of the state and several gates move. So the step depends on what the token is, not only on which channel it is.
The cap belongs to the channel, the sigmoid to the token. A channel with a small cap never moves far, whatever the token says.

5 · Across loops the same matrix gives a different step

\(W_\beta\), \(b_\beta\) and \(\gamma\) are shared by every loop. The step still changes from loop to loop, because \(\tilde s_i\) changes as the state moves.

Loops

\(\beta\) for each loop (rows) and channel (columns)

How far each loop moved the state

Grey: the change the block proposed. Green: how far the gate let the state move. A toy loop block (a fixed random layer with tanh), with section 4's token and gate.

As the state settles, each loop proposes a smaller change. Dory1.0-4B, which has no gate, settles the same way: over 16 loops its state moved 2.6% at loop 2 and about 0.3% per loop by loop 10.

6 · At one loop the gate is a learned skip around the middle block

\[ s_1 = (1-\beta_1)\odot h_{\text{in}} + \beta_1 \odot \mathcal{R}_\theta(h_{\text{in}}) \]

This is one loop as the paper's 0.5B ablations run it, without adding the input back. Each channel keeps part of its input and takes part of the block's output. An ordinary model has no such choice: it always takes all of the block's output.

For section 4's token: how much of each channel comes from the block

Orange: the share taken from the block, the number at right. Grey: the share kept from the input.

The gate alone took 0.025 off the one-loop loss

GPT-0.5B trained on 300B tokens, validation loss; lower is better. Ladhak et al., section 6.2.

The model trained only at one loop runs every layer once, like the baseline. Without the input added back the two differ only in the gate, and the gate model is 0.025 lower.

7 · Only a learned, token-dependent step matches the baseline at one loop

Rule	Update	Result, GPT-0.5B
Replace	\(s_i = r_i\)	Highest one-loop loss; worse at 5 loops than at 4
Residual	\(s_i = s_{i-1} + r_i\)	Stops improving after 3 loops
Delta	\(s_{i-1} + \alpha \odot (r_i - s_{i-1})\)	Second at every depth, 0.006–0.007 behind
Adaptive delta, the gate	\(s_{i-1} + \beta_i \odot (r_i - s_{i-1})\)	Lowest loss at every loop count

Delta's \(\alpha\) is learned for each channel but is the same for every token; it starts at 0.05. Ladhak et al., section 6.1.

With the gate, one loop matches the baseline and five loops are 0.058 lower

GPT-0.5B trained with mixed loop counts (up to 5), validation loss. Ladhak et al., Table 14.

8 · Dory, NVIDIA's released 4B looped model, has no gate matrix

	Koi-4B, the paper's model	Dory1.0-4B, released weights
Between loops	A gated step, \(s + \beta \odot (r - s)\)	Replace, then normalize: \(\operatorname{norm}(\mathcal{R}(\operatorname{norm}(s + h_{\text{in}})))\)
Inside each layer	Ordinary residual layers	nGPT's step, \(h \leftarrow \operatorname{norm}(h + \|\alpha\| \odot (f(h) - h))\)
Who sets the step	Each token and channel, through \(W_\beta\)	Each channel, through \(\alpha\); the same for every token
Gate parameters	\(W_\beta\) is \(3{,}072 \times 3{,}072\), plus \(b_\beta\) and \(\gamma\)	None among its 507 tensors

Koi's step changes with the token; Dory's does not

Koi-style: \(\beta\) from \(W_\beta \tilde s\)

Dory-style: \(\alpha\) for each channel

Toy values; rows are tokens, columns are channels. Each panel has its own colour scale.

Dory's per-layer step is the paper's "delta" rule, moved inside every layer. In the paper's 0.5B ablation, delta came second, 0.006–0.007 behind the gate. The question for the room: does nGPT's per-layer step already do most of the gate's job?

More for Thursday: the 17 slides, the 10-minute audio overview and the companion write-up.

Sources and notes

Faisal Ladhak, Shantanu Acharya, Santiago Akle Serano, Simeng Sun, Boris Ginsburg, Krishna C Puvvada, "One Model to Loop Them All: Elastic Latent Recurrence for Language Models", arXiv 2610.12582, 8 Oct 2026. The gate and its initialization: Eq. 2 and section 3.1. Update rules: section 6.1 and Figure 4a. The one-loop result: section 6.2 and Figure 4b. Losses with the gate: Table 14. Parameter share and hidden sizes: Appendix A and Table 4.
Dory1.0-4B: the tensor list is from its safetensors header at revision b366b87f, read Tue 20 Oct 2026. The loop's form agrees with NVIDIA's vLLM implementation. The 16-loop settling figures come from an independent re-measurement of the released model.
nGPT: Ilya Loshchilov, Cheng-Ping Hsieh, Simeng Sun, Boris Ginsburg, "nGPT: Normalized Transformer with Representation Learning on the Hypersphere", 2024.
The interactive figures use toy numbers with \(d = 8\). They show how the arithmetic works, not a trained model's values.