Steer a VLM's gaze over images
Redirecting ten gaze heads overrides an explicit text prompt: a click-to-steer demo and a prompt-conflict sweep in Qwen3-VL-2B.
| Building | A ten-head gaze-steering handle that overrides a VLM's text prompt — a click-to-steer demo plus a prompt-conflict sweep |
| Models | Qwen3-VL-2B (subject) · Claude Sonnet (1-of-6 panel judge) · random-head control |
| Topics | Multimodal · Vision · Steering · Circuits & causal tracing |
When a vision-language model describes an image, a handful of attention heads point at the region it is talking about. Redirecting those ten "gaze heads" does more than nudge the caption: it can override an explicit text prompt that asks about a different region.
The obvious way to control what a model describes is to ask it in words, and on these comic strips a panel-specific prompt is a strong instruction the model follows on its own 82.2% of the time. The open question is which handle wins when the prompt and the steering point at different panels.
76.5% of the time, in a direct conflict, the caption follows the steered panel and only 4.7% follow the prompt. A ten-head attention edit beats the written instruction in roughly three of four cases.
This recipe builds a driveable, stress-tested handle on a VLM's gaze: a replication of gaze-head steering at 2B, an explorer over the real generations, and a prompt-conflict sweep that measures how dominant the handle is.
Key takeaways
- An attention-level control surface can outrank the text prompt: where a model looks can override what it is told to describe.
- A steering mechanism is only trustworthy alongside a specificity control. Redirecting random heads instead of gaze heads drops the effect to the floor.
- Pushing a replication into a condition it was never tested under, here prompt against steering, is what turns a reproduced number into a new finding.
Gaze Heads in a nutshell
When a VLM describes an image, its language backbone attends back to the image tokens. Gandikota & Bau (arXiv:2606.14703) find that this mechanism is localized: a small number of "gaze heads" concentrate on the specific region the model is currently describing. Because the signal lives in attention, it is also a control surface. Forcing those heads to attend to a chosen region steers the generated description toward it, without any fine-tuning.
The paper measures this on six-panel comic strips, where the model is asked which panel it is describing and the six panels give a clean 1-of-6 readout. The headline at the smallest model it quantifies, Qwen3-VL-2B, is that steering the top-10 gaze heads makes the model describe a chosen panel 68.6% of the time.
The control is what makes that number mean something. Steering ten random non-gaze heads instead leaves the chosen-panel rate at 1.3%, near the floor, so the effect is specific to the heads that track gaze rather than a general consequence of perturbing attention. Chance, picking one panel of six, is 16.7%.
The method, end to end:
- Inputs and dependencies. Qwen3-VL-2B (open weights), the authors' released gaze-heads code, and the released six-panel comic-strip dataset. A vision-capable judge (Claude Sonnet) reads each generated caption and forces a 1-of-6 choice of which panel it describes.
- Discover. On a held-out split of strips, rank every attention head by how much it attends to the panel currently being described. The top head sits at layer 19, head 5; the top ten fall in a narrow late-middle band, layers 16–21 of 28.
- Steer. At generation time, overwrite those ten heads' image attention to point at a target panel, and let the model caption as usual.
- Measure. Judge which panel the caption describes, and compare against the steering target. Applications follow directly: a training-free way to direct a VLM's description, and a probe for whether a model's "looking" is causal for its "saying."
Recipe goal: measure gaze-head steering robustness across prompts
The research objective of this thread was to take the paper's claim from a single number to something a reader can drive and stress, then to test how robust the mechanism is against the most natural competitor: the prompt itself. The thread built the evidence in three steps.
1. Replicate the mechanism (experiment #17). Steering the top-10 gaze heads on 250 held-out strips makes Qwen3-VL-2B describe the chosen panel 68.4% of the time (95% CI [66.1, 70.7]), against the paper's 68.6%. The random-head control sits at 1.4% and chance at 16.7%, with non-overlapping intervals. The discover and evaluation strips are disjoint, split by comic id, so head selection cannot leak into the score. A sweep over the number of steered heads peaks near ten, matching the paper's choice.
2. Make it tangible (experiment #26). The replication's stored generations became a self-contained explorer: pick a strip, see the base caption, click any panel and watch the caption the model produced when its gaze heads were redirected there. The explorer reuses experiment #17's generations verbatim, and a live GPU check confirmed the steering hook reproduces those stored outputs bit-for-bit, so the demo shows real model behavior rather than a mock-up.
3. Push past the paper (experiment #30). A real signal invites a harder question: the paper steers under a neutral prompt, but what if the prompt itself asks about a specific panel? The follow-up sweeps seven prompts (one panel-agnostic, six panel-specific by position) against every steering target, on 100 strips, and judges each caption.
- Steering still works under a panel-specific prompt: pooled steering accuracy is 75.0% (CI [73.6, 76.4]), even slightly above the panel-agnostic 67.8%.
- The prompt alone is a strong instruction: with no steering, the model follows a panel-specific prompt 82.2% of the time. So this is not a weak competitor.
- In a direct conflict, where the prompt names one panel and steering targets a different one, the steered panel wins 76.5% of the time (CI [75.0, 78.0]) and the prompted panel wins only 4.7% (CI [4.0, 5.5]); the rest describe neither.
Verdict: On these comic strips, the gaze-head attention handle is not just a steering knob but a dominant one: redirecting ten heads overrides an explicit panel-specific text prompt in roughly three of four direct conflicts.
Where to take it next
Does the override survive at larger scales and against stronger prompts? Re-run the conflict sweep at 8B and across other VLM families, widen the prompt axis to content-based and system-level instructions, and patch or ablate to locate where in the stack the gaze heads outcompete the instruction-following pathway.
Where do the gaze heads look when nobody is steering them? Track the gaze heads' attention across a broad range of natural images with no intervention, and check whether they reliably mark the region a VLM is describing in everyday scenes.
Does gaze-steering work on natural images, not just tidy comic panels? Rebuild discovery and steering on a region-annotated set such as COCO, steer toward an object's bounding box, and judge whether the description follows.