An autoregressive speech model carries a moving, frame-by-frame emotion signal as it talks. Reading that signal shows arousal is the trustworthy axis, and a single direction added during generation steers the prosody.
| Topics | Multimodal · Audio & speech · Neural geometry · Steering · Linear probes |
| Models | Orpheus-3B (Llama-3B speech-LLM) · emotion2vec readout · random-direction control |
| References | Belief-trajectory steering (Stories in Space) · EmoSteer-TTS / CoCoEmo |
A model that turns text into speech has to decide not just what words to say but how they should sound. That prosody, the rise and fall of energy and pitch, is where emotion lives in a voice. The question this recipe takes up is whether an autoregressive speech model represents that emotion as something an analyst can read off its internal state and then deliberately change, the way recent interpretability work reads and steers belief states in a language model.
The move is to treat the speech model as a language model that happens to emit audio. Orpheus-3B generates one audio token at a time on a standard Llama residual stream, so the same probing and steering tools used on text apply directly. This recipe first reads the model's internal emotion signal frame by frame to see whether it forms a coherent path over an utterance, and then adds a single per-emotion direction during generation to test whether that representation is a causal lever on the produced speech.
Reading first pays off. The internal per-frame arousal signal tracks the arousal of the audio the model is producing, in sync and within each clip, at a median correlation of r = 0.68, and it beats a per-clip-constant baseline by +0.18 R² — so emotion genuinely moves across the utterance rather than being fixed once at the start.
Key takeaways
Recent interpretability work treats abstract variables not as static labels but as paths. Goodfire's Stories in Space shows that as a language model reads evidence in context, its belief about the answer moves along a low-dimensional trajectory in activation space, that the trajectory can be decoded with linear probes, and that steering along its geometry changes the model's behavior in a predictable way. A companion line of work on steering along manifolds for continuous concepts makes the same case: find a structured internal representation of a variable, then intervene on it.
This recipe asks whether the same approach carries from text to another modality, speech. An autoregressive speech model generates audio one token at a time on the same kind of residual stream, so the read-and-steer recipe should transfer directly. Existing emotional-speech work such as EmoSteer-TTS and CoCoEmo runs on flow and diffusion TTS models that do not expose such a stream, which makes Orpheus-3B the natural place to test the idea.
The goal is to establish, for an autoregressive speech model, whether emotion is a representation an analyst can read off the residual stream over time and then move, and to pin down which parts of that claim the evidence actually supports.
The core result is narrow but clean: arousal is a time-varying, output-tracking, steerable direction in an autoregressive speech model. The geometry around it is busier than emotion alone, valence remains out of reach with this readout, and overriding strongly emotional text needs more than a single direction at intelligible strength.
The techniques that read and steer large language models also reveal how multimodal generation models work. Linear probes, difference-of-means steering, sparse autoencoders, the logit lens, and activation patching all operate on a transformer's residual stream, and an autoregressive image or speech model runs the same transformer over a different token vocabulary. A probe trained on its activations, or a steering vector added to them, behaves the same way it would on text.
Multimodal generation models add something text models lack: an external output an analyst can measure directly. The emotion in generated audio, or the content of a generated image, is an independent readout to check what a probe claims against. This recipe is one instance, reading and steering emotion in a speech model with tools built for text, and the same approach extends to any generation model that emits tokens.