Probe the copy-number axis in Evo 2 7B
| Building | a linear probe + steering vector for tandem-repeat copy number |
| Models | Evo 2 7B (blocks.24) · ridge probe · generic-insertion baseline |
| Topics | Life sciences · Linear probes · Steering |
Tandem repeats — short DNA motifs copied head-to-tail — host some of the most consequential mutations in human genetics: a CAG run past ~36 copies in HTT causes Huntington's; expansions in DMPK and FMR1 drive myotonic dystrophy and Fragile X. Genomic foundation models like Evo 2 clearly know something about these regions — but they're black boxes. Does the model actually represent "how many copies are here," or is repeat length just a smear across thousands of dimensions?
In this cookbook you will build an internal copy-number readout: a linear probe that decodes repeat length from Evo 2 7B's activations, a steering vector that controls it generatively, and the controls that prove the axis is specific to repeat structure rather than generic insertion. The probe works: on held-out real genomic loci it recovers copy number at R² 0.93–0.98, the count sitting there as a single readable direction.
This methodology is valuable because the alternative is annotation: today you count repeats with specialized assays or hand-built bioinformatics pipelines, locus by locus. If the count is already sitting in a model's residual stream as a linear direction, you can read it for free, and, as it turns out, steer it. The catch is knowing the signal is real and not an artifact of "you inserted a long thing." Most of the work here is earning that confidence.
A probe like this is a general tool for prying open a black box: if a model has learned a quantity, a simple linear read of its activations can often expose it.
What you'll take away
- Train a probe to find internal activations that encode a specific quantity.
- Prove it's causal by steering along the direction and checking the output changes.
- Choose the right controls to rule out the obvious confound.
Find the axis
The question started loose — is there a "tandem-repeat manifold" in Evo 2? — and scoping turned it into something testable: take Evo 2 7B, read per-position activations at blocks.24, and ask whether a held-out linear probe can recover the copy number of a repeat run. The first pass used synthetic constructs (a clean, low-dimensional axis). But synthetic sequence is easy; the design that earns trust is the one that names the obvious objection up front and builds the control into the run.
Two objections got written into the plan before anything dispatched: (1) maybe the probe is reading flank geography, not the repeat — so probe on real genomic loci, held out by host gene, against a hard periodicity-removed control; and (2) maybe it just reads "something was inserted" — so later, a full insertion-control ladder. The plan is a working draft you can question and rerun: swap the layer, tighten the host-level splits, add a control.
Once you approve the plan, Silico builds the probe on its own — pulling Evo 2's per-position activations at blocks.24 for every locus, fitting the ridge probe under host-gene-held-out splits, and scoring it against the periodicity-removed control. The activations, the splits, and the baseline come out of one dispatched run, not a notebook you babysit.
The probe survives the move to real sequence. On held-out loci it decodes copy number well, beating the periodicity-removed control — and when a real repeat run is interrupted by a base substitution, the readout shifts by 2–4 copies, exactly as a genuine counter should. The synthetic-to-real gap from the first pass turned out to be flank geography, not a broken signal.
Steer it
A decodable direction could still be a passive correlate. That direction is just the probe's weight vector: the same line we read the count off of is the line we can push along to change it. The next experiments do exactly that and ask the causal question: if you push activations along the axis during generation, does the model's output copy number actually change? There is more than one way to push by the same amount, and the figure below lays them out.
The result is clean: pushing along the axis causally controls repeat length, and only length. Steer down and the repeat shuts off. Steer up and it grows, but as raw length rather than the target motif.
Steering down shuts off the repeat. A 40-copy CAG run collapses to 1–3 copies. The one thing to get right is how hard you push: push each motif by a different amount and the effect looks motif-specific, with CGG and CA seeming locked while CAG collapses. That is an artifact of the uneven push, not a property of the model. Push every motif the same distance along the axis and all three terminate, well beyond the random off-axis baseline, at every layer we tested. (The straight chord and the curved geodesic push work about equally well, so the axis is close enough to linear that following its curve buys little.)
Steering up adds length, not motif. From a neutral, non-repeat context, the model produces length that scales with the push, but as a homopolymer (a run of a single base), not the target motif. That is why this is a length axis: it controls how many copies, not which motif. Motif identity is carried by local flank context, not this direction.
Rule out the boring explanation
One objection can still sink everything: maybe the probe isn't reading repeats at all, just "a chunk of sequence was inserted here." If a generic length-of-insertion signal explained the result, the whole story would be mundane. So the final experiment is a control ladder built to break it — 1,800 constructs inserting four kinds of content into 50 real gene windows: repeat (periodic), shuffled (composition-matched but aperiodic), random (uniform), and gene content (a real downstream floor).
The generic-insertion explanation fails on every test:
- A probe trained on random inserts cannot read repeat length at all — transfer R² of −4.2 (worse than predicting the mean).
- The repeat and random length directions are orthogonal (cosine −0.06; confidence intervals exclude zero for
CA/CAG/CGG). - Projecting out the generic-insertion direction leaves the repeat readout unchanged (R² 0.97 → 0.97).
- Within-type decodability tracks structure, not size: repeat 0.97, shuffled 0.86, random 0.05, gene content −0.78. The repeat is the smallest mover per added copy yet by far the most decodable.
The verdict the run lands on: the copy-number axis is structure-specific. Composition carries a lot of it; periodicity sharpens it — but generic insertion is refuted.
Putting it all together
Six experiments, one axis. The probe reads copy number off real loci, the steering vector controls it generatively, and the control ladder shows it tracks repeat structure, not generic insertion. Silico synthesized the thread — goals, plans, the failed first steering protocol and its fix, the controls, the plots — into one narrative, and the interactive below travels with it so you can drive the count yourself instead of reading a static figure.
What this unlocks: an annotation-free copy-number readout that places HTT / DMPK / FMR1 alleles straight from their flanks, and a generative length knob — terminate or extend a short tandem repeat on demand.
When to reach for this
Internal probing pays off when you suspect a model encodes a specific quantity, you can build a clean label for it (here, known copy number), and you can design a control that isolates that quantity from its obvious confound (here, insertion size). It beats training a downstream predictor when you want a readout that is cheap, mechanistic, and steerable. It is the wrong tool when no held-out label or matched control exists to keep the probe honest, or when you need what this readout cannot give: exact counts deep inside a long expansion. The axis cleanly separates normal from expanded alleles, but its counting horizon saturates around k ≈ 8–21 copies, and a single model and layer (Evo 2 7B, blocks.24) are all this rests on so far.
The broader point: the expensive way to ask what a model knows about repeats is to train a battery of downstream predictors and infer the representation from behavior. The cheap way is to read the representation directly, with a linear probe and a steering vector, and let controls do the adjudicating. The whole result is a few decode R²s, a transfer matrix, and a matched-magnitude null. The same loop — scope the quantity, probe it on held-out real data, steer it to test causality, then break the boring explanation with a control ladder — extends to anything a bio-model might encode internally: GC content, splice proximity, conservation, regulatory state. Internal probing is how you turn these black boxes into instruments.