Build a tool-call safety guardrail from a model's activations

A live monitor that flags an unsafe tool call before it runs, powered by a linear probe that reads the model's own activations rather than its words

Topics	Language models · Linear probes · Safety & guardrails
Models	Qwen3-8B (subject) · Qwen3-4B · lexical baseline
References	ShoppingBench (arXiv 2508.04266)

A single tool call can spend real money, leak a saved card number, or pay out an attacker's refund. A tool call is the small structured command an agent emits to act in the world, and software executes it with no chance to take it back. The dangerous ones rarely look dangerous. A budget-blowing add_to_cart or an injected refund emits almost the same command as the safe version, so a keyword scanner cannot tell them apart.

This recipe builds a guardrail for that moment: a monitor that scores each tool call the instant the model commits to it and raises an alarm before the action is handed off. The move it makes is to read the model's internal state instead of its text. A linear probe over the model's activations on the tool-call span catches what the surface command hides.

The guardrail is built on Qwen3-8B acting as a shopping assistant. The probe is the cheap part: a single linear rule over the residual-stream activations at one layer, trained on labeled safe and unsafe calls, that outputs one number between 0 and 1. It adds almost nothing to the cost of running the model, and it is what the live monitor loads and fires on.

On a held-out set of shopping tool calls the probe never saw in training, the guardrail separates safe from unsafe at 0.92 AUROC on Qwen3-8B, up from the 0.5 of a coin flip, and a smaller Qwen3-4B carries a weaker but real signal at 0.85. Wired into a live monitor that scores each call at completion, it runs at about 0.95 on matched agentic traffic.

Key takeaways

• A cheap linear read of a model's activations can catch an unsafe action that the surface text hides, because the activations at the call already encode the situation the call is wrong in.
• Read the narrow tool-call span, not the whole transcript. Pooling over the full prompt-plus-response dilutes the signal rather than adding to it.
• A detector can transfer to a new domain while its firing threshold does not. Recalibrate the threshold on ordinary traffic per deployment, and treat a fixed alarm point as domain-specific.
• A tool-call probe only means anything when the model is genuinely emitting a tool call. Feeding it prose produces confident nonsense, so the guardrail must gate on the model actually being in tool-call mode.

Linear probes for safety in a nutshell

When a language model produces a tool call, the call is the visible tip of a great deal of hidden computation. Every token the model processes leaves a trail of internal activations, vectors that encode what the model has taken in: what the user asked for, what the rules were, what the call is about to do. A linear probe is the simplest possible reader of that trail. It is a single linear rule, fit on labeled examples, that maps the activations over a chosen span to one score. Training it is cheap and running it alongside the model is nearly free, which is what makes it practical as an always-on guardrail rather than a second model in the loop.

The guardrail here is packaged as a probe pack: the trained probe weights plus a small metadata file naming the model, the layer, the hook site, the pooling, and the span the probe reads. A live monitor loads the pack, taps the model's residual stream at that layer while it generates, and scores the tool-call span at completion. Building the guardrail is therefore four concrete pieces fitting together.

• Inputs & dependencies. Qwen3-8B as the subject model, a corpus of labeled safe and unsafe shopping tool calls (matched synthetic pairs plus a held-out control), and the model's residual-stream activations harvested over the tool-call span.
• The probe. A linear probe fit on the call-span activations at a single mid-stack layer, chosen by validation performance, then frozen into a probe pack with its metadata contract.
• The monitor. A server that streams generation from the model, taps the same layer, scores the call at completion, and fires when the score clears a calibrated threshold.
• The output. A per-call safety score and a fire or quiet verdict, delivered before the action executes, with the decision on what to do on a flag left to the application around it.

Building a guardrail that holds up

The goal is a guardrail worth deploying: one that catches unsafe calls the surface text would hide, survives a move to a domain it was not tuned on, and runs as a live monitor without crying wolf. The thread behind this recipe pushed on each of those in turn, and the honest result is that the probe is the strong part while the firing threshold is the part that needs care.

• It works in domain. On held-out shopping calls the probe separates safe from unsafe at 0.92 AUROC on Qwen3-8B (0.85 on the 4B), against a 0.5 chance baseline. High enough to act as a reliable second pair of eyes, not a perfect gate.
• It reads the situation, not the command. Restricting the test to calls that are unsafe only because of context, calls that look fine on their own, the probe reading the call still reaches 0.947. Pooling over the whole prompt-plus-response drops to 0.867, and gluing prompt and call together does no better at 0.943: the compact safety signal sits right at the moment of action, because the call's activations have already absorbed the conversation before it.
• It travels to a new domain, the threshold does not. Moved unchanged to ShoppingBench tasks, the probe still beats a word-matching baseline, reaching 0.73 on near-constant add-to-cart surfaces where the lexical baseline is at chance (0.45) and 1.0 on autonomous over-budget purchases where the baseline lands at 0.29. But the home threshold of 0.97 catches only 39% of unsafe actions out of domain, because the score distribution shifts. Recalibrating the threshold on a separate sample of ordinary traffic recovers recall to 76% while holding false alarms near 10%.
• It only fires when the model is really calling a tool. An early deployment scored a benign English refusal at 0.9998, a near-certain false alarm, because the model had not been given tool definitions and wrote prose instead of a call. Passing the real tool schemas so the model emits a genuine call drops that same case to about 3e-13. The probe is a reader of tool-call activations, so the guardrail has to gate on the model actually being in that mode.
• It fires correctly once the monitor matches what the probe was trained on. Even with schemas passed, the live monitor stayed quiet on unsafe calls until its default tool surface was matched to the probe's training distribution. With the tools aligned the monitor scores each call at completion and reaches 0.952 live, unsafe calls firing at 1.00 against benign calls near 9.6e-7.
• It generalizes past this one setup. The working monitor was consolidated so the tools, the system prompt, and the tool-call gate are deployment configuration rather than code hardwired to Qwen3-8B shopping, which is what makes the same guardrail re-targetable to another model or task.
• What it does not show. The restricted-item violation axis sits at chance out of domain, but that test rests on only 39 unsafe examples, too few to conclude anything either way. The recalibration caveat is standing: any new deployment domain needs its threshold reset on local benign traffic, not inherited.

The out-of-domain threshold behavior is the part worth feeling rather than reading. The explorer below recomputes recall and false alarms over the evaluation set as the threshold moves, with a toggle between the home and new domains.

Interpretability as a runtime guardrail

The tool-call guardrail is one instance of a more general pattern: read a model's activations while it runs and gate on what they carry. Nothing about the method is specific to unsafe shopping calls. The same residual-stream tap could feed a panel of linear probes trained on different failure modes, deception, data-exfiltration intent, or jailbreak compliance, so a single cheap read covers several risks at once rather than one. Because the signal lives in the model's internal state rather than its output text, this kind of monitor watches behaviors that never surface cleanly in what the model says, which is exactly where output-level filters are weakest.

The harder and more interesting direction is moving from watching to acting. The probe defines a direction in activation space that separates safe from unsafe, and that direction can in principle be used to intervene during generation, suppressing the unsafe action instead of alarming after the model has committed to it. That turns a read-only guardrail into active control, and it raises the question that decides whether any of this holds up under pressure: whether reading internal state is genuinely harder to fool than reading the output, or whether an adversary who knows a probe is watching can optimize the signal away. Settling that is the difference between a useful monitor and a real safety layer.

Related recipes

Train hallucinations out of a model with a probeA hallucination probe, then RL that uses it as a reward to cut the hallucination rate. Reward shaping: trading off safety and complianceReward shaping as a tunable safety dose for DPO — an α-sweep of the safety/compliance trade-off. What DPO changes inside a language modelA reader-driven tour of the features alignment amplifies and suppresses inside a model.