RL against the fabrication probe: frozen-reward GRPO on Qwen3-4B

Summary

Turn the fabrication probe into a training signal: run GRPO on Qwen3-4B with a reward that scores its own generations for fabrication using the covariance probe from experiment #5. The policy's text is scored through a frozen Qwen3-4B, so the policy can't game the probe by drifting its internal representations off the distribution the probe was trained on. Reward = per-sentence probe factuality + KL leash to base (the minimal recipe). The experiment is decided not by the reward but by an independent held-out web-search judge: actual fabrication must fall while the model keeps making claims (no collapse to vagueness), and the gap between probe-measured and judge-measured improvement quantifies any residual reward hacking.

~14 h = torch_dist conversion (~0.3 h) + smoke (~0.5 h) + GRPO run (~10 h, dominated by the frozen-model forward pass inside the reward) + held-out generation & judge eval (~2 h). Wide-ish; the worker re-anchors off the first measured rollout-step time.

Hypothesis

Claim: Optimizing Qwen3-4B against the frozen-probe factuality reward reduces its real (web-judged) fabrication rate on held-out questions, without the probe reward being hacked and without collapsing informativeness.

• Supports: on held-out prompts (disjoint from RL-train and from #1's probe-training prompts), judged fabrication rate drops by ≥ 10 pp vs base, supported-claim count stays within ~15% of base (no abstention collapse), and the probe-vs-judge improvement gap is small (judged gain ≥ ~70% of probe-measured gain).
• Reward-hacked (the failure we're testing for): probe score improves but judged fabrication barely moves (large gap), or claim count collapses, or general helpfulness degrades. Any of these means the minimal reward is gameable even with a frozen host, an informative negative.
• Null: judged fabrication unchanged within CI → the reward signal is too weak/noisy to move behavior at this probe accuracy (0.816).

Why this matters

This is the payoff of the whole thread: does an interpretability artifact (a probe) actually make the model better, not just diagnose it? A win is a cheap, self-contained way to reduce hallucination without a web-search judge in the training loop. A hack is just as valuable: it tells you the probe is a measurement instrument, not a safe optimization target, before anyone ships it as one. The frozen-host design is the specific anti-hacking hypothesis under test.

Deliverable

Artifact + findings. The RL'd Qwen3-4B checkpoint (HF-exported), and the base-vs-trained comparison on judged fabrication, coverage, the probe-vs-judge gap, and a general-capability sanity eval.

Stack & existing code

• FireRL (~/.silico/libraries/FireRL, /rl:rl-train): GRPO from the stock scripts/run-qwen3-4B.sh, single-node colocated. Container image /mnt/data/artifacts/ai-neuroscientist/images/slime-nightly-dev-20260329a.sqsh. No fork.
• Custom reward via config seam: --custom-rm-path rl_custom.probe_reward:custom_rm, authored in the experiment artifacts dir (/data/<exp>/rl_custom/) and put on the container PYTHONPATH via RUNTIME_ENV_JSON. The library clone stays pristine.
• Reward model: experiment #5's calibrated layer-20 covariance probe + the frozen-host activation recipe (per-token residual at layer 20).
• Held-out judge: Claude SDK + web search, the same SAFE-style decomposition+verification used in #1/#5. Used only at eval time, never in the training loop.
• Capability sanity: lm-evaluation-harness (Goodfire fork) for a small general-ability check (e.g. MMLU subset) on base vs trained.

Input artifacts (prior experiments)

• #5: calibrated layer-20 covariance probe checkpoint + exact preprocessing (token-span pooling, layer, dtype). Resolve #5's artifact root (same-thread, preauthorized); verify load in preflight.
• #1: LongFact prompt sampling + the prompt IDs used for probe training, so the RL-train and held-out eval prompt sets can be made disjoint from the probe's training prompts.

Reward design

def custom_rm(args, sample) -> float, per rollout sample:

1. Take the policy's generated answer; segment into sentences (cheap regex/nltk, no web judge at train time).
2. Run the frozen Qwen3-4B over the rendered prompt+answer once; capture per-token layer-20 residual activations.
3. Apply #5's covariance probe to each sentence span → fabrication probability pᵢ.
4. Reward = factuality score = 1 − mean(pᵢ) ∈ [0,1] (mean, not sum, so length isn't rewarded).
5. No-claim case (zero sentences detected) → reward = 0.5 (neutral, not the maximum). This is the explicit abstention lever; with the minimal reward it's the most likely hack, so coverage is tracked as a primary metric (below).

KL to base is the framework's term (--use-kl-loss --kl-loss-coef), not part of custom_rm. The frozen model + probe load once as module-level singletons on a dedicated GPU and are reused across reward calls.

Known approximation: the probe was trained on atomic-claim spans; at train time we score whole sentences (cheaper). The smoke gate re-validates probe discrimination at sentence granularity, and falls back to lightweight per-claim decomposition if sentence-level AUROC degrades materially.

Data

• RL-train prompts: LongFact prompts, JSONL (--prompt-data, --input-key, --apply-chat-template, non-thinking), disjoint from #1's probe-training prompts.
• Held-out eval prompts: a separate LongFact slice, disjoint from both RL-train and probe-train. This is where the verdict is measured.

Training config (GRPO, to confirm at the approval gate)

• Advantage estimator GRPO; --use-kl-loss, --kl-loss-coef (start 0.05, tune if collapse appears); --eps-clip stock; lr stock for Qwen3-4B.
• Batching honors rollout-batch-size × n-samples-per-prompt ≥ global-batch-size (stock 32 × 8 = 256).
• Single node, colocated actor+rollout (Qwen3-4B fits one GPU); --qos=opportunistic for this experimental run; --account = team (worker confirms); no --partition, GPU defaults for CPU/mem.
• Checkpoints every --save-interval; export best to HF at the end.
• W&B (configured): log reward, KL, mean probe score, and the periodic held-out judged-fabrication + coverage eval. register_wandb_url(), register_key_metrics(["eval/judged_fabrication_rate","eval/supported_claim_count"]).

Anti-hacking evaluation (the actual result)

• Generate from base and trained Qwen3-4B on the held-out prompts (fixed decode).
• Judge both with the web-search SAFE pipeline → fabrication rate (fraction unsupported), supported-claim count (coverage/recall), precision and F1@K.
• Probe both (frozen host) → probe-measured fabrication.
• Hacking gap: Δjudged-fabrication vs Δprobe-fabrication. A large gap = the policy moved the probe without moving reality.
• Collapse check: Δ supported-claim count and answer length, base vs trained.
• Capability check: small MMLU-subset (lm-eval) base vs trained, to rule out broad degradation from RL.
• Report all deltas with bootstrap CIs over prompts.

Steps

1. Preflight: FireRL/MegaFire on expected branches, image present, #5 probe + #1 prompts load, Qwen3-4B loads (ungated), W&B verified.
2. Convert weights HF → torch_dist (one SLURM job).
   Precondition. The converted checkpoint loads as --ref-load. Protects: the whole run can't start otherwise.
3. Build & validate the reward harness (frozen host + covariance probe).
   Precondition. On a labeled held-out batch from #5, the reward harness reproduces probe discrimination (AUROC ≈ 0.816 at claim granularity; report sentence-granularity AUROC too). Protects: the entire run, a miswired probe (wrong layer/pooling/tokenization) would optimize noise. Fall back to per-claim scoring if sentence-level AUROC degrades materially.
4. Smoke test (2–3 rollout steps, 1 node), job name slco-smoke-.
   Precondition. Rollout completes, custom_rm returns finite rewards, and reward varies across samples (variance > 0). Protects: a constant/degenerate reward gives no learning signal and is a common reward-wiring bug.
5. GRPO run with periodic held-out judged-fabrication + coverage eval logged to W&B.
6. Final eval (anti-hacking section): base vs trained on held-out prompts; export best checkpoint to HF.
7. Publish the comparison results page + the trained checkpoint.

Resource plan

• 1 node × 8 GPU (Qwen3-4B colocated), --qos=opportunistic, --time padded to ~16 h. Worker checks queue depth and confirms node count at the approval gate (smaller-now beats larger-pending).
• Conversion + eval: short SLURM jobs.
• Outputs (torch_dist, slime checkpoints, HF export) under $SILICO_EXPERIMENT_ARTIFACTS_DIR (must sit under the mounted /data for the container).

Visualizations

• Training curves: reward, KL, mean probe score, and the periodic held-out judged-fabrication + coverage on shared x (rollout step). The probe line dropping while the judged line doesn't = hacking, visible live.
• Base vs trained bar pair: judged fabrication rate, supported-claim count, MMLU, with CIs.
• Probe-vs-judge gap figure: probe-measured Δ next to judge-measured Δ, the headline anti-hacking result.
• Before/after answer pairs: same prompt, base vs trained answer, claims color-coded by the judge verdict, the qualitative specimen (and a check that the trained answers didn't just get vaguer).

Preflight

• FireRL/MegaFire present on expected branches; container image exists.
• #5 probe checkpoint + #1 prompt IDs resolve and load; Qwen3-4B loads without a token (ungated).
• Claude SDK web-search judge path works on a sample claim.
• W&B verified (wandb login --verify).

Note: the FireRL approval gate (full config + compute plan sign-off) still applies, the worker will confirm node count, account/QoS, and the GRPO hyperparameters against live cluster availability before submitting.