| Building | A hallucination probe, then RL that uses it as a reward to cut the hallucination rate |
| Models | Qwen3-4B (subject + policy) · web-search SAFE judge · token-logprob baseline |
| Topics | Language models · Linear probes · Post-training & RL · Safety & guardrails |
When a language model hallucinates a fact, it often does so with full confidence, and asking it whether it is sure rarely helps. Out of the box, Qwen3-4B hallucinates on roughly one in five of the checkable facts it asserts. This recipe takes a different route: it reads the model's internal activations with a small linear probe to decide whether a generated claim is a hallucination, then turns that probe into a reward and trains the model against it. That cut web-judged hallucination on held-out prompts from 19.8% → 11.7%, a 41% relative drop, with no web judge anywhere in the training loop and general ability (MMLU) unchanged.
That drop matters because the usual ways to hold hallucination down are expensive. Verifying every claim with a web search is accurate but far too slow to sit inside a training loop, and a second model asked to grade the first tends to share its blind spots. The gain here comes from a cheaper place: a signal already latent in the model's own activations, read out for the cost of a single forward pass and never sent to an external checker during training.
The result rests on a four-step arc: confirm the signal exists, build a probe good enough to trust, show the probe beats what the model says about itself, then use it as a reward. Each step is a separate experiment in one Silico thread, and each fed the next.
Key takeaways
The thread opened with a narrow question: is hallucination even represented inside Qwen3-4B, or does the model simply not know when it is wrong? A linear probe trained on layer-16 activations, labeled by a web-search judge and tested on unseen prompts, reached 0.723 AUROC. That is modest in absolute terms but clearly above every control: it beat a token-logprob baseline by +0.090 and a length-and-position baseline by +0.190, while shuffled-label and shuffled-activation controls stayed at chance (peak 0.529). Hallucination is partly written into the activations, and it is not the same thing as the model being unsure.
A 0.72 monitor is interesting but not yet a tool, so the next experiment asked why it capped: a flawed readout, or subjects with no checkable facts dragging the average down. Each implied a different fix, so the run was scoped to separate the two.
The answer was pooling, not domains. A covariance probe that reads each claim's individual tokens reached 0.816 AUROC at layer 20 (ECE 0.039), up from the 0.723 mean-pool ceiling. Token-aware probes beat mean-pooling at every layer (best held-out gain +0.088), so averaging across the answer, as the first experiment did, was washing out part of the signal. The mean-pool probe reproduced the earlier 0.723 exactly, which confirmed the comparison was like-for-like rather than a pipeline difference. Domains were not the cap either: all 38 subjects cleared the verifiability bar, so there was nothing to filter out.
The same internal read also beats what the model says about itself. Asked to rate its own confidence on each claim, Qwen3-4B loses to the probe:
The symptom is stark in the raw numbers: Qwen3-4B rated its own unsupported claims 93.7 out of 100 for truth, against 97.0 for supported ones, a gap of barely three points. That signal, accurate and calibrated, was good enough to use as a training reward, which is where the thread turned next.
The capstone experiment turned the probe into a reward. It ran GRPO on Qwen3-4B with the layer-20 covariance probe scoring each generated sentence: the reward is one minus the mean probability of hallucination, with a KL leash to the base model so the policy cannot drift far.
The design choice that makes this safe to even attempt is the host. The policy's text is scored through a frozen copy of Qwen3-4B, never its own live weights, so the policy cannot raise its reward by quietly reshaping the representations the probe reads. The plan also fixed the decision rule in advance: the experiment is judged not by the reward it optimizes but by an independent web-search judge applied only at evaluation, on prompts disjoint from both the RL training set and the probe's training set.
The reward optimized smoothly over 200 rollout steps, with held-out probe factuality rising from 0.631 to 0.796 and no instability.
The verdict came from the instrument the optimizer never saw. On the 150 held-out prompts, the web-search judge measured hallucination falling from 19.8% to 11.7%, a drop of 8.1 percentage points (95% CI [−10.7, −5.6]) or 41% relative. General ability held: MMLU moved 0.680 → 0.678. The model did not buy factuality by refusing to answer either, with supported claims per answer falling only 22.7 → 19.8 (−12.8%), still about twenty verified facts per response. An interpretability artifact, used as a reward, produced a measurably more factual model.
Optimizing against a learned reward invites Goodhart's law: the model may improve the probe's number without becoming more truthful. This is the failure the experiment was built to detect, and the frozen host is the defense. The test is whether an independent judge corroborates the probe's reported gain.
It does. Measured on each instrument's own base rate, the probe-measured reduction (34.6% → 20.6%, −40%) matches the web-judged reduction (−41%) almost exactly.
The honest exception is behavioral, not representational. Trained answers came out about 30% shorter (853 → 593 tokens) and hit the generation length cap far less often (70 of 150 down to 23 of 150). Part of the factuality gain therefore comes from the model making fewer, more cautious claims, which raises the fraction supported partly by reducing exposure rather than only by being more truthful. The freeze blocked the representational hack the experiment most feared; what it did not prevent is the policy learning to say less.
The interactive below is built from the real per-claim judge verdicts on the held-out set, the same data behind the headline number. Each prompt shows the base and trained answers side by side, with every checkable claim colored by the web-search judge's verdict. The aggregate strip reports the full 150-prompt result; the controls step through individual prompts and sort them by how much hallucination fell, or rose.
Sorting by "where it got worse" is the point of including it: on a handful of prompts the trained model hallucinated more, and the brevity effect is visible directly, claim counts and answer lengths dropping alongside hallucination. This is the same artifact a write-up carries, so the trade-off can be driven rather than described.
Probe-guided RL is worth using when a few conditions hold:
A simpler approach wins when a reliable, cheap verifier already exists, in which case training directly against that verifier avoids the probe entirely. And in every case the brevity result argues for pairing the probe reward with a length or coverage term, so the policy cannot raise factuality by saying less.
The status quo for reducing hallucination puts an expensive checker in the loop: a human, a web-search judge, or a larger grader model, run on every sample. The internal signal is far cheaper, a single linear read over activations the model already computes, and here it carried enough of the load to improve the model without any external judge during training. The expensive judge was needed only once, at the end, to confirm the cheap one had not been gamed.
The broader point is that interpretability artifacts can be optimization targets, not just diagnostics. A probe that reads a concept off activations can become a reward that shapes that concept, provided the reward model is frozen so the policy cannot rewrite what the probe reads. That single design choice, plus a held-out verdict the optimizer never touches, is what separates a genuine gain from a gamed metric. The Silico loop is what kept the chain honest end to end: confirm the signal against controls, isolate why it capped, show it beats the model's own words, then optimize against it and check the result on an independent instrument.