What DPO changes inside a language model
A reader-driven tour of the features alignment amplifies and suppresses, and how much of it is visible in the preference data before training starts
| Topics | Language models · Sparse autoencoders · Post-training & RL · Safety & guardrails |
| Models | Llama-3.1-8B Tulu-SFT (subject) · standard DPO baseline |
| References | AI2 Dolci / Olmo-3 post-training · Direct Preference Optimization (DPO) |
Aligning a chat model with Direct Preference Optimization visibly changes how it answers, but the change is usually read only from the outputs. This recipe reads it one level down instead: it passes the model's activations through a sparse-autoencoder lens, turns the dense internal signal into a small set of nameable features, and measures which features alignment turns up and which it turns down. The unit of analysis is the feature cluster, a group of features that move together, plus the prompt clusters that drive each shift.
Read that way, the effect of DPO is narrow and partly anticipated. Of the 65,483 features the lens ever sees fire on the preference data, only 151 change their firing rate by a clear margin, about two in a thousand. And a forecast built only from the preference data, before any training, predicts the direction of the actual internal shift 79% of the time, far above a coin flip. Alignment is not a diffuse rewrite of the whole model. It is a small set of targeted nudges, much of which is legible in the data beforehand.
Key takeaways
- A sparse-autoencoder lens makes a training intervention auditable: instead of only noticing that outputs changed, the analyst can name which internal features a run amplified or suppressed.
- The internal footprint of a preference-tuning run can be sparse. When most features barely move, the few that do are worth reading individually rather than summarizing in aggregate.
- Part of what alignment will do is forecastable from the preference data alone. Measuring which themes separate winning from losing answers gives a usable prior on a training run's effect before committing the compute.
- Raw win-rate rewards length. When a method inflates response length, quality has to be read with a length-controlled comparison or the verdict is confounded.
How the dissected model was trained, and the length confound
The model opened up here is a Llama-3.1-8B, supervised-fine-tuned on AI2's Tulu data, then aligned with DPO on the Dolci-Instruct-DPO preference set. Getting an honest subject was its own short investigation. A learning-rate sweep of standard DPO found a setting that the model could absorb cleanly, preferred about 70% of the time over the SFT base with its knowledge, math, and instruction-following intact; pushed harder, the same recipe broke the model instead of aligning it. The win came partly with longer answers, so a length-normalized variant was tried to tame the verbosity. It made things worse at first, and the failure turned out to be an over-hot learning rate rather than the objective itself.
The lesson that survived is a measurement one. Raw win-rate quietly rewards longer answers, so the recipes only separate cleanly once quality is read with a length-controlled comparison, the metric this thread was missing at first. That is the model now placed under the lens.
Reading a model with a sparse-autoencoder lens, in a nutshell
Modern chat models are first taught to be helpful, then aligned: nudged toward the answers people prefer. DPO does this by showing the model thousands of pairs in which a human picked one answer over another and pushing it toward the preferred one. After alignment the outputs change, often for the better, but the machinery that produced the change stays opaque. The question this recipe inherits from the post-training literature is what alignment does inside the model, not just to its answers.
The instrument is a sparse autoencoder, a tool trained to rewrite a layer's dense activation vector as a much larger but mostly-inactive set of features, each of which tends to stand for something a person can name: a topic, a tone, a formatting habit. The handful of features active at any moment are an interpretable summary of what the model is doing at that point.
- Inputs & dependencies. The DPO-aligned Llama-3.1-8B and its SFT base, the Dolci preference pairs, and a sparse-autoencoder lens trained on one layer's activations.
- Method. Record which features fire on each answer; group co-moving features into named clusters; measure each cluster's change in firing rate after DPO; and, separately, predict that change from how strongly the cluster separates chosen from rejected answers in the preference data.
- Lens health. The lens captures about 82% of the variance in the original signal with no dead features, and recovers roughly 81–83% of the signal on the model's own freshly written answers. Good enough to read by.
Read what DPO changed, from the inside
The goal is to test whether DPO's effect on this model is legible and foreseeable: whether it lands on a small, nameable set of features rather than smearing across the whole model, and whether the preference data predicts which way each feature will move. The evidence, in order:
- The behavior moved, capabilities held. On its own answers the aligned model is preferred about 70% of the time, with knowledge, math, and instruction-following scores within a couple of points of the base. Median answer length roughly doubled, from 163 to 337 tokens.
- The internal change is sparse. Across more than a million prompt-theme combinations, only about 2% shift more than chance would predict; of 65,483 features ever active, just 151 move by a clear margin. The strongest and most consistent single change is numbered-list formatting.
- And partly foreseeable. A prediction built from the preference data alone agrees with the actual internal shift on direction 79% of the time. The magnitudes line up too, with a correlation around 0.45 across all themes and about 0.61 among the ones that move most. It is a partial forecast, not a precise one.
- What it does not show. This is one model, one preference dataset, one lens at one layer, trained once. The cluster names are the lens's hypotheses read off activating text, not proven causes, and where to draw cluster boundaries is itself a modeling choice.
Put together, the picture is encouraging for anyone who wants to monitor what alignment does. The internal effect of DPO is concentrated and nameable, and part of it is visible in the data a team already has before spending the training compute. That is a step toward checking which behaviors a run will amplify or suppress in advance, instead of only noticing after the outputs change.