Driving novelty in protein generation models

A binder-design model keeps returning natural folds it memorized in training. A force applied while it generates makes the model forget those folds, recovering novelty — with no retraining.

Building	A generation-time nudge that recovers novelty from a copy-prone design model
Models	BoltzGen-1 · Chai-1 (independent check) · scrambled-fold control
Topics	Life sciences · Steering · Memorization

Try it yourself. Open the finished run in your own workspace and pick up where it leaves off.

Generative models are trained to reproduce their data, yet they are most useful when asked to produce something the data did not contain. Where novelty is the objective, memorization becomes a failure mode that is easy to miss, because a memorized output is internally consistent and scores about as well as a genuinely new one. Protein design offers a clear case. BoltzGen, a leading model for designing protein binders, will often answer a request for a novel binder with a natural protein it encountered during training, reproducing precisely the structure it was meant to invent. Recall is simply the path of least resistance, because a fold the model has seen many times can satisfy almost any request and nothing in its confidence score penalizes the shortcut, which makes the real question whether novelty can be made the default without retraining the model.

The lever turns out to be deliberate forgetting: a steering vector applied during generation. As BoltzGen refines a structure step by step out of noise, the steer adds a vector to the design's coordinates that pushes it away from the memorized fold — applied early, while the shape is still malleable, with no weights retrained. Applied at the right strength it converts a model that mostly recycles into one that mostly invents: across a sweep of ubiquitin binders, recall of the memorized fold falls from 95% → 16%, and on toward zero with a little more steer, while most designs remain foldable. This cookbook follows the chain of experiments that established the effect, calibrated it, and showed it generalizes.

Key takeaways

Memorization hides inside confident outputs. A recycled answer can score as well as a novel one, so catching it means comparing against what the model is prone to reuse, not trusting its own quality signal.
Novelty can be a generation-time intervention, not a retraining project. A force applied while a diffusion model builds its output can steer it away from memorized solutions with the weights left untouched.
A method from one paper can be replicated into another setting. The repulsion technique here is lifted from a published method for a different model and goal, then ported and tested in a new context, which is often faster than inventing an intervention from scratch.

The pull of memory

Memorization is hard to see precisely because it looks like success. A recycled fold is well-formed and confident, so it passes the model's own quality checks; counting copies requires comparing each design against the natural structures the model is prone to reuse. The clearest evidence that BoltzGen is recalling rather than designing appears when you vary nothing but the requested length of the binder. Copying is neither flat nor random. It spikes whenever the requested length matches the size of a fold the model already knows.

Copying tracks length, not target. The copying rate peaks where the requested binder length matches a natural fold the model knows: ubiquitin is 76 residues, and copying reaches 94% at exactly 76 residues, then a second, larger memorized fold takes over near 90–100 residues. A model that was genuinely designing would have no reason to prefer those particular lengths.

The tallest peak sits at 76 residues, the length of ubiquitin, a small protein the model has seen many times. Ask for a 76-residue binder and as many as 94% of designs are ubiquitin again; ask for something near 90–100 residues and a second memorized fold takes over. The pattern is not confined to one target. In a separate sweep of 1,200 designs, 20 of 24 conditions were fully collapsed onto a single fold, and 22 of 24 reached for the same small family of shapes regardless of which target was named.

The obvious correction does not help. Supplying the real, experimentally known binding site and instructing the model to dock there leaves the copying intact: across these targets the designs still converge on the default fold 100% of the time. Telling the model where to bind does not change what shape it reaches for. And the fallback folds are a fixed menu — four entirely new targets all collapsed onto the same small catalogue of natural shapes (fold similarity 0.94–0.99), each memorizing 100% of the time.

The binding site is not the lever. Where the real, experimentally known interface was supplied, copying stays at 100% with or without it. The copied shapes form a fixed menu: across unrelated targets they match a small catalogue of natural folds at 0.94–0.99 similarity.

Taken together, these observations point to one reading: the model reaches for memory whenever memory fits the request, and the usual control knobs — length, binding site, target identity — do not interrupt that reflex. Recall is the default and novelty the exception. If novelty is to become the norm, the intervention has to act on the generative process itself.

Engineering forgetting

BoltzGen builds a structure by diffusion: starting from random noise and refining it step by step into a clean fold. That step-by-step process is where the steering vector lives. At each step the design has a current best guess at its finished shape. When that guess drifts toward a memorized fold, a repulsive force pushes back. The steering vector is the gradient of that repulsion, added to the design's coordinates to displace it away from the fold. The model's weights are untouched. The effect is to make the model forget a specific structure for the duration of one generation, so that it has to settle on something else.

The mechanism is adapted from AF3-ReD, which added a repulsive bias to AlphaFold3's diffusion sampler to push each new prediction away from structures it had already produced, as a way to surface alternative conformations of the same protein. Here the same move is repurposed toward a different end: not exploring a protein's conformations, but preventing a binder design from reusing a memorized fold.

Two parameters govern the steer, and both came out of the scoping conversation rather than being fixed in advance:

How hard to steer. Too little and the memorized fold survives; too much and the structure is destroyed before it can settle.
When to steer. The vector is applied early, while the overall shape is still soft and can be redirected, and released for the later steps so the model's own denoising can relax the geometry into something foldable. It is also gated — active only when the design sits close to a memorized fold, fading as it moves away.

The folds to steer away from come from a fixed library of the shapes BoltzGen tends to copy, so the same steering vector transfers to a new target without per-target tuning. Scoping settled these choices collaboratively and turned them into an editable, rerunnable plan; two corrections from that exchange shaped the design materially — replacing a hand-written per-residue push with the true gradient of a Gaussian repulsive potential (so the steer is a principled vector, not an ad-hoc one), and adding a gate that validates the harder PD-L1 fold correspondence before the sweep runs.

Lab: stating the memorization goal and linking the AF3-ReD paper with a sketch of the method — **Scoping the intervention.** The goal is stated in plain language, with the AF3-ReD paper linked and its repulsion method sketched out as the approach to adapt.

Lab: the design reflected back, with the controls and the independent quality metric — **Scoping the intervention.** The goal is stated in plain language, with the AF3-ReD paper linked and its repulsion method sketched out as the approach to adapt.

Forgetting on, novelty up

With the plan approved, the run executed it end to end: generate de-novo ubiquitin binders across a sweep of push strengths, score each design for copying and foldability, and hold the controls and random seeds fixed so the conditions are comparable. The bookkeeping travelled with the run. Each design's push setting, copying score, and foldability check stayed attached to the same record, rather than scattered across separate notebooks.

The effect is large and orderly. With no push, 95% of designs are memorized copies. A modest early push cuts copying to 16% while keeping 67% of designs foldable, and a little more drives copying to zero. The forgetting does not simply trade one memorized fold for another. Across the working window the designs that escape spread out, their fold-to-fold similarity falling from 0.96 to 0.38, which is the novelty the recipe set out to recover. Past the window the structure stops surviving the push, and foldability falls to 41% as the output degrades into generic rubble.

A result this clean invites a simple objection: maybe the push is not removing the memorized fold so much as roughing up the structure until the copy detector no longer recognizes it. The plan included a control to settle this, and the run reported it alongside the headline. Replacing the real fold with a scrambled version of the same size and applying the identical force leaves copying at 97%. The intervention removes the specific fold it targets. Generic perturbation of the same magnitude does nothing.

A control separates forgetting from breakage. Pushing against the real memorized fold cuts copying from a 95% baseline to 16%. Pushing against a scrambled fold of the same size, at the same strength, leaves copying at 97%. The force has to know which fold to push away from, so it is removing memory rather than simply disrupting the geometry.

The same run also surfaced where the method fails, which is a question of timing rather than strength. The push works only when it is applied early, while the overall shape is still soft. The identical force applied late, after the fold has set, has nothing malleable to redirect and simply breaks the structure. On the harder PD-L1 target, an early push cuts copying from 51% to 27% while 89% of designs stay foldable, where the same push applied late collapses foldability to 12%. Strength sets how much memory is removed. Timing decides whether anything foldable is left behind.

Forgetting that generalizes

A correction that has to be retuned for every target is closer to a curiosity than a tool. The next experiment built directly on the previous one, inheriting its sweep, controls, and scoring, and asked whether a single fixed setup transfers. It assembled one library of the folds BoltzGen tends to copy and applied it, untouched, to four targets the method had never seen. The transfer held. Every new target memorized 100% of the time at baseline and de-memorized once the library push was on, with no per-target tuning.

Transfer is only useful if the surviving designs are worth keeping, so the follow-up scored each target by usable yield: out of 100 attempts, how many come back both novel and foldable. The baseline is zero everywhere, because the unmodified model copies every time. With the untuned library the yield runs from 0.21 to 0.45 across the four targets, strongest on TNF-α.

One library, four unseen targets. Usable yield, the share of attempts that come back both novel and foldable, with a single untuned library of memorized folds. Every target starts at zero, because the unmodified model copies 100% of the time.

Two further checks make the yield credible. First, what caps it is foldability, not copying. The push reliably knocks designs off the memorized fold, and the attempts that fail do so because the escaped design does not fold cleanly, which is a quality problem rather than a diversity one. An earlier scale-up made the same point from the other side. Pushing harder to drive copying lower eventually destroyed foldability, confirming that the limiter sits in structure quality and that the working window has an upper edge. Second, the designs that survive are not merely different, they engage the target better. Scored under a separately trained folding model that has no part in the de-memorization, the new designs match or beat the memorized copies, which often sat near the binding site without truly gripping it. On one target the fraction of intended contacts the design actually makes rose from 0.08 to 0.37 (p = 0.0004), and on another from 0.02 to 0.18.

How hard to forget. Drag the push strength to move along the measured curve. Copying (red) falls as the push grows, foldability (green) holds across the shaded working window, and beyond it the structure stops surviving the push. The schematic below the chart marks the three regimes: a memorized copy, a novel design, and over-pushed rubble.

What this does not show

The claim is deliberately narrow: at generation time, a gated repulsive force can move a copy-prone binder model toward novel, foldable, target-engaging designs, on the targets and metrics tested here. Several limits bound it.

Some targets resist. For PD-L1 and GFP the copying stays locked in across the usable range of push strengths, breaking down only once the displacement is large enough to destroy the design. When the fold the model wants is deep enough, there is no clean escape.
Yields are moderate and quality-limited. A usable yield of 0.16 to 0.45 means most attempts still fail, and they fail on foldability rather than on escaping the copy. Making the escaped designs reliably foldable is the open problem.

When to reach for this

The transferable idea is generation-time forgetting: while a model is building an output, push it away from the solutions it has memorized, then let it settle. The protein binders here are one instance. The technique pays off when three conditions hold together:

The generator builds its output gradually, through a diffusion or other iterative process with a malleable intermediate the force can act on while the answer is still forming.
The failure is collapse onto memorized solutions, not a general lack of quality, so there is a specific thing to push away from.
The solutions to avoid can be named, as a set of outputs or representations the steer can measure distance against.

A different approach wins when those do not hold. If retraining or fine-tuning is cheap and the copying is rare, fixing the weights is cleaner than steering every sample. If there is no soft intermediate to redirect, there is nothing for an early push to work on. The cost is real and worth stating plainly. The technique spends some output quality to buy novelty and works only inside a narrow strength window, so it suits exploration and proposal generation more than high-throughput production.

What it means

The behavior a model was trained into is not the only behavior available from it. BoltzGen's copying is baked into its weights, yet the fix never touches them. Reading off the model's own intermediate guesses and nudging them while the answer is still soft is enough to convert a model that mostly recycles into one that mostly invents. The same shape of intervention, a gated force applied early in an iterative generative process and released before the end, should transfer to other settings where a model collapses onto memorized solutions and novelty is the goal.

That this took one connected thread, rather than a pile of disconnected scripts, is the point worth carrying. Scoping turned a loose goal into a concrete, rerunnable plan. The run produced the headline together with the control that tests it. The follow-up inherited that setup and showed the effect transfers, and the write-up gathered the whole arc into something a reader can both follow and run.