Train an agent with reinforcement learning in Silico

Building	A GRPO run that trains a 4B agent to GPT-4.1 level
Models	Qwen3-4B-Instruct-2507 · benchmark ASR reward · GPT-4.1 / paper SFT+RL
Topics	Language models · Post-training & RL · Evaluation & benchmarking

Try it yourself. Open the finished run in your own workspace and pick up where it leaves off.

In this cookbook we run a reinforcement learning project end to end in Silico, training Qwen3-4B-Instruct-2507 with GRPO on ShoppingBench Products-Finder, an agentic benchmark where the model searches a catalog over many turns and must commit to a product that satisfies a set of constraints. The reward is the benchmark's own success check (ASR), and the focus is the workflow each stage runs through: scope the task, launch the training, read the curve, check the numbers, and extend the run.

What you'll take away:

How to scope an RL run. Settle the environment, the training data, and the reward with the agent before anything runs. The output is a plan you can read, edit, and rerun.
Build on Silico's RL stack. The platform ships RL skills and a training codebase for multi-turn agentic rollouts, with ready seams for a custom rollout loop and reward, so you point existing machinery at your task instead of assembling a trainer, an environment, and a reward yourself.

Scope the RL problem

Scoping fixes the three things an RL run depends on: the environment, the training data, and the reward. ShoppingBench is a multi-turn agentic task over a large product sandbox. Silico's RL skills and training codebase already support that shape, with multi-turn rollouts and seams for a custom rollout loop and reward, so the setup is configuration rather than building a stack from scratch. The binding constraint is data. The benchmark releases test instructions but no training split, so we synthesized roughly 3,000 Products-Finder instructions, with splits audited disjoint from the 250 held-out test instances on the exact line, the product id, and the query.

The reward is the benchmark's own success scorer, run in a sandbox shared between training and evaluation, plus a small format bonus. One confound is fixed in advance. The benchmark scores agents in a 30-turn flattened loop, whereas the first run trains in a trimmed 12-turn loop chosen for clean loss-masking. Whether this train/eval format gap affects the result is the question the follow-up run isolates. Scoping produces an editable, rerunnable plan, here revised mid-conversation to train across four nodes.

Lab: a scoping decision card with base model, reward, and task scope locked in — **Scoping the run.** The scoping conversation settles the three things the run depends on: the base model, the reward, and the task scope.

Lab: a diagram of the rollout loop — shopping query, agent loop over the sandbox, recommend, ASR reward, GRPO update — **Scoping the run.** The scoping conversation settles the three things the run depends on: the base model, the reward, and the task scope.

Launch the training

Once the plan is approved, the run executes it end to end: convert the model, generate multi-turn rollouts against the sandbox, score them with the reward, and update the policy. Because it runs on the platform's RL stack rather than bespoke glue, the data, splits, reward sandbox, checkpoints, and eval numbers stay attached to one experiment instead of scattering across one-off notebooks. And because the run streams its metrics to Weights & Biases, you can watch the curve form live instead of waiting for the job to finish.

Each saved checkpoint, written every twenty rollout iterations, is evaluated at full retrieval depth on the held-out set with 95% bootstrap confidence intervals, giving a curve rather than a single after-the-fact number. ASR rises monotonically across checkpoints and then flattens. By rollout 99 the policy has essentially converged, and the last checkpoint sits within noise of it.

Official-harness ASR for the untrained base, the trained policy, and the 12-turn policy ported to the official harness, with 95% bootstrap CIs and GPT-4.1 / paper reference lines — **Benchmark-comparable ASR.** The trained policy (58.4%) sits far above the untrained base (26.8%, non-overlapping CIs) and inside the confidence interval of both GPT-4.1 (59.6%) and the paper's SFT+RL (60.8%). Full retrieval depth, 250 held-out instances.

The gain comes mostly from one behavioral change: the trained policy commits. The base model often searches, re-searches, and terminates without recommending a product, which the reward scores as zero. After training, no-recommendation failures fall from 18.8% to 2.4% of instances, exact-product hits rise from 6.0% to 34.8%, and the mean number of turns drops from 7.9 to 6.4.

Verify before you trust the number

RL is evaluated against a live sandbox, which can fail in ways that look like the model failing. A handful of rollouts here showed up as zero-turn losses, which the per-instance records traced to the sandbox dropping connections rather than the policy misbehaving. After hardening the harness and re-scoring the affected instances, an early within-harness number moved from 50.0% to 61.6%, and every uncorrected ASR figure stayed provisional until its eval was re-run clean. The lesson generalizes: when a result depends on a live environment, check the environment before you trust the score.

Failure-mode breakdown of held-out instances: four genuine model-failure categories versus 63 infrastructure errors from dropped sandbox connections that were counted as failures before the harness fix — **Genuine failures versus infrastructure.** Of the losses charged to the policy, 63 were dropped sandbox connections rather than model errors, a count comparable to the largest genuine failure mode. Re-scoring after the harness fix takes them to zero.

Does matching the harness help?

A first result is the start of the loop, not the end. The named confound from scoping was the train/eval format gap: the first policy trained in a trimmed 12-turn loop, but the benchmark scores in a 30-turn flattened loop. The obvious next move is to close that gap by training directly in the official harness, so train and eval formats match. The follow-up run does exactly that, reusing the synthesized data, the reward sandbox, and the fixes the first run paid for.

It reaches reference level. The matched-harness policy scores 58.4% ASR [52.4, 64.4], inside the confidence interval of both references and far above the base (26.8% [21.6, 32.4]). But it ties the cheaper approach. The first run's 12-turn policy, simply ported into the official harness, scores 55.6% [49.6, 61.6], and the two intervals overlap almost entirely. The ported policy also scores about the same in its own harness (54.8%) as in the official one. The format gap the second run was built to close was costing almost nothing to begin with.

Full-depth ASR over training for the matched-harness run, with a 95% CI band and reference lines for base, the ported 12-turn policy, and the paper — **ASR over training, matched harness.** Each saved checkpoint evaluated at full depth on the held-out set. The curve crosses the ported-policy line around rollout 79 and tracks it thereafter without pulling clear, consistent with the overlapping intervals above. Training stopped on a NaN gradient at rollout 122, after the plateau.

The honest reading is that matched-harness training is a fine way to reach reference level, but not a free win over the cheaper policy. The point for an RL project is the shape of the test: name the confound during scoping, then run the experiment that settles it rather than assuming the expensive option is better.

Running RL in Silico

ShoppingBench was just the example here. What carries across projects is the loop, and in Silico it is the same one each time:

Scope the environment, the data, and the reward in conversation.
Launch the run on the platform's RL codebase rather than on bespoke glue.
Watch the curve form live in Weights & Biases.
Verify the evaluation before trusting it.
Build the next run on the last.

Point it at a different sandbox, reward, or base model and only the inputs change.

That loop is what made this result hold up. Because verification was part of the run, the dropped-connection errors surfaced and were re-scored before they reached the headline. Because the follow-up inherited the first run's data, reward, and fixes, it tested one clean change instead of starting over. And the configuration that did best was the simplest: plain GRPO with the benchmark's own reward reached GPT-4.1's band, while the more elaborate harness-matching follow-up only tied the cheaper policy. Fork the run below to take the same loop to your own task.