How LMs mix mechanisms to retrieve bound entities

Figure 2 of Mixing Mechanisms, rebuilt from scratch on causalab and verified against the raw interventions, then replicated across two model families — from a single Silico prompt

Topics	Language models · Circuits & causal tracing
Models	gemma-2-2b-it · Qwen3-8B-Base · index-match grader
References	Gur-Arieh, Geva & Geiger 2025 — Mixing Mechanisms (arXiv:2510.06182)

One intervention, three pointers

Each bar above aggregates many interventions like the one below. The model first reads a list of 20 bound groups of the form "{musician} performed {genre} music on the {instrument}," then is asked about one of them. Each group holds three entities, and t_entity names which of the three the question targets: in "Coltrane performed jazz music on the saxophone," t_entity = 1 targets the musician (Coltrane), t_entity = 2 the genre (jazz), and t_entity = 3 the instrument (saxophone). The question here uses t_entity = 3, giving the other two entities and asking for the instrument: "Coltrane performed jazz music on the ___" — the bound answer is saxophone.

Three other groups in that same list of twenty matter, because the intervention is built to make them compete:

• Group 1 — "Garcia performed folk music on the banjo." First in the list, and the only group naming Garcia.
• Group 2 — "Mingus performed soul music on the bass." Second in the list.
• Group 3 — "Hendrix performed blues music on the guitar." The only group whose instrument is a guitar.

The counterfactual is a re-bound copy of the list with a different question — one asked about Garcia. The group that matters there is the one bound to the query, now sitting in second place: "Garcia performed funk music on the guitar." So the counterfactual's query name is Garcia, its slot is second, and its answer is guitar. Patching that run's last-token residual into the original at layer 18 and re-reading Coltrane's answer, the three pointers each retrieve a different group, and the instrument that comes out names the pointer the patch carried:

• Positional (i_P = 2) → bass. The positional pointer copies the counterfactual question's slot. Garcia sat second there, so it reads whatever fills second place in the original list — Mingus's bass.
• Lexical (i_L = 1) → banjo. The lexical pointer follows the counterfactual's query name, Garcia, back to his group in the original list and returns the instrument bound to him there — banjo.
• Reflexive (i_R = 3) → guitar. The reflexive pointer is a direct copy of the counterfactual's answer, guitar, which appears in the original at Hendrix's group — guitar.

If the patched model still says saxophone the patch had no effect (excluded); any other instrument is mixed. Repeat across all twenty positions and the U-shape above falls out: when the queried group sits at an edge, the answer is almost always the positional one, and in the interior, where the positional pointer goes diffuse, it gives way to the reflexive pointer when an early slot is queried (t_entity = 1) and the lexical pointer when a late slot is queried (t_entity = 3).

The same U-shape on Qwen3-8B-Base

Re-running the identical interchange sweep on Qwen3-8B-Base reproduces Figure 2 at layer 23 (~64% relative depth, matching Gemma's ~65%).

What to try next

• When does each mechanism emerge over training? Re-run this interchange sweep across training checkpoints to time the appearance of each pointer. A natural hypothesis is that the positional mechanism arrives early because short lists are the most frequent case and so are learned first, and the lexical mechanism emerges later to handle the long tail of very long lists.
• How prevalent is the reflexive mechanism? Measure how often the reflexive pointer carries retrieval, across tasks and list lengths, relative to the other two. A direct pointer to a token is a smaller piece of information than the token itself and costs less of the residual stream, so it may be widely reused as a compact handle that is unpacked only when the bound token must be retrieved and un-embedded to predict the next token.
• Which templates does the pattern fail for? Sweep the binding-task templates and record where the U-shape and its t_entity modulation do not appear. Identifying those templates, and what they share, would show the limits of this account of in-context retrieval and point at why the mixture does not generalize there.

Related recipes

Replicating Interpretability in the WildRecovering the IOI circuit from Wang et al. (2022) with a native path-patching primitive. Replicating ROME's Causal TracingRecovering ROME's causal-tracing sites from Meng et al. (2022) with a native restoration primitive. Verbalizing the weekday manifoldA SelfIE read-out of a model's weekday geometry — a nameable ring plus a causal-subspace test.