Building an AI Cancer Case Reviewer
Reproducing a molecular-tumor-board agent, then swapping out one subsystem at a time to find which design choices actually move the score
| Topics | Language Models · Evaluation & benchmarking · Ablation · LLM Guardrails |
| Models | GPT-5.1 agent · Claude judge · bare GPT-5.1 (no-tools baseline) |
| References | Ferber et al., Nature Cancer 2025 · ferber20 benchmark |
When a cancer is rare or advanced, treatment decisions are often referred to a molecular tumor board: a group of oncologists, pathologists, and geneticists who jointly review the patient's scans, biopsy, and tumor DNA before recommending a plan. It is slow, expensive, expert work, and several groups have now built AI agents that attempt the same job. These agents are not a single model. They are reasoning systems wired to tools, and many engineering decisions go into them: how the loop is orchestrated, how treatment guidelines are retrieved, how scans are read.
This recipe rebuilds one such agent faithfully, then takes it apart. Rather than tuning the whole system at once, it isolates three subsystems—the agent harness, guideline retrieval, and imaging—and swaps one variant at a time while holding everything else fixed. Each variant is scored on the same benchmark, shifting the question from whether the agent is good to which design choices drive its performance.
Silico implemented a complete tumor-board agent, ran 22 configurations of it across three subsystems, graded every run against a clinician-written rubric on real cases, and folded the results back into a single recommended build. Each design choice in the assembled agent is supported by an ablation rather than intuition, making the final system more evidence-based and inspectable. The payoff is concrete too: the assembled agent reaches 79% completeness, up from 64% for the bare model, and the three subsystems turn out to behave very differently from one another. One is a large lever, one barely matters, and one buys accuracy at a safety cost.
Every number here is inspectable. Silico turned the runs into an interactive case viewer, where any of the 20 cases can be opened against any of the 22 agent versions to see what was fed in, the agent's evidence-gathering loop, its full recommendation, and every rubric score with the judge's reasoning.
Key takeaways
- A whole agent can be implemented, ablated, and reassembled in one place, so every part of the final build is a measured decision rather than a guess.
- The scaffold around a model can matter more than the model. Measure the harness before reaching for a bigger model.
- More autonomy is not free. When the model's own runtime controlled the loop instead of an explicit script, the agent skipped evidence-gathering steps and lost points.
- Image access improves completeness, but direct pixel reading can introduce safety concerns. A lightweight "confirm before you commit" step mitigates them.
How it's built and graded
The agent, based on the system described by Ferber et al., runs in two stages. First it gathers evidence: it reads the scans, looks up the tumor's mutations in a cancer-genetics database, searches the medical literature, and runs any needed calculations. Then it retrieves guidelines and writes: it pulls relevant passages from a library of oncology treatment guidelines and drafts a recommendation that cites them, with a self-check pass at the end.
Everything is measured on ferber20: 20 real molecular-tumor-board cases published with a clinical study. Each case gives the agent what the board would have: a patient history, a molecular report listing the tumor's mutations, and radiology scans. It ends with an open question such as, "What are the next steps in her treatment?"
Grading a free-text treatment plan is the hard part. For each case, the original clinicians wrote down the specific points a good answer must contain, and across the 20 cases there are 109 such criteria. The agent's answer is scored on how many of a case's criteria it covers, a measure this recipe calls completeness. Because reading a long clinical answer against a checklist is itself a judgment task, the grader is a large language model.
- Inputs. 20 ferber20 cases, each with history, molecular report, and scans; a corpus of 413 oncology guideline documents; the OncoKB genetics database.
- Method. A two-stage tool-using agent (GPT-5.1) driven by an explicit harness, graded by an LLM judge against 109 clinician-written criteria.
- Calibration. For criteria evaluated by four oncologists in the original study, the judge matches the clinicians' consensus 92.6% of the time. Individual oncologists agreed with their own consensus 94.0% of the time and with each other 88.7% of the time. Completeness is the only measure validated against oncologists; four secondary measures (correctness, safety, citation grounding, helpfulness) are reported as approximate AI-graded proxies.
Approach 1 — The harness
The harness is the scaffolding around the model: the tools it can reach and the script that decides when to call them. Here it is a plain server-side loop, not the model's own agent runtime. It builds the first message from the patient case, the question, and a list of the imaging files, then hands the model a fixed menu of tools. Each turn the model either emits a tool call, which the harness executes and feeds back, or writes its final plan, which ends the loop.
Several ways to wire this up were compared: a bare model with no tools; a vanilla model given only web search; the full faithful agent with domain tools driven by the explicit two-stage script; and a model-driven variant that exposes the same tools but lets the model's built-in runtime decide the order and when to stop. Three findings follow.
- The domain tools are the lever. The full agent scores 15 percentage points higher than the same base model without tools (79% versus 64%). What helps is not access to the open web but access to the right domain tools.
- General web search adds nothing. Adding general web search did not improve completeness; if anything, the score dipped slightly, although the model searched in about 93% of runs.
- More autonomy was worse. Giving the model's own runtime control of the loop lowered completeness by about 9 percentage points. It ended the loop early and skipped evidence steps the script would have forced. The model API used to drive the tools made no measurable difference.
Approach 2 — Guideline retrieval
The second stage depends on pulling the right passages from a library of treatment guidelines, the classic retrieval problem. The corpus is 413 guideline documents across four public sources. When the retrieval tool runs, the model proposes several subqueries. Each is run against the corpus, and the combined results become the passages the model may cite. A final citation check rereads each cited passage and revises the answer as needed.
Because retrieval design inspires strong opinions, the study tested several approaches: vector search versus BM25; with and without reranking; home-built versus hosted indexes; and snippets versus full chunks or surrounding sections.
- The nine variants were statistically tied. They all land in the same narrow 78–82% band. Swapping the search engine did not measurably change completeness, keyword search tied with meaning-based search on completeness, and how much of each passage the model saw made no difference either.
- A caveat on the null. This insensitivity may partly reflect benchmark saturation, perhaps because the rubric is coarse or the cases do not require fine-grained retrieval. The honest reading is that retrieval design was not a lever here, not that retrieval never matters.
- What it implies for a build. Since the choices tie, the practical move is to pick the simplest and cheapest engine rather than the most elaborate one.
Approach 3 — Imaging
The third subsystem lets the agent use the patient's scans. There are three separate code paths for seeing a scan: a radiology read (a vision model returns a written report and the model works from that text), a measurement tool (MedSAM segments a lesion the model boxes and returns its size), and native multimodal input (the scan is attached to the message and the model reads the pixels directly). The imaging arms are on/off combinations of these plus a no-imaging baseline, and a safety variant that adds a confirmation pass.
- The value is the qualitative read. The benefit is the scan actually being read, not a side effect on the rest of the answer.
- The measurement tool barely helps. Adding lesion measurement to a written radiology read raised completeness by about 2 percentage points, an effect that was not statistically distinguishable from zero. The qualitative read already captures most of what the scan offers.
- Direct pixel reading ties on quality but encourages overtrust. On answer quality, reading pixels directly and going through a written report are statistically tied, and reading pixels is simpler and cheaper. But when the model read pixels directly it sometimes stated an image-derived conclusion, such as a cancer's stage, as established fact and let it drive a major recommendation. The evidence comes from a small number of examples identified by an LLM judge, not a calibrated safety evaluation, so it should be treated as a pattern worth investigating. After the confirmation step was added, no flagged cases remained in this evaluation.
Explore every case yourself
Every number here comes from running these agents on the 20 cases, five times each. To make the runs inspectable rather than a table of averages, Silico built an interactive case viewer: pick a patient case and a version of the agent, and it shows exactly what was provided to the model—the case, prompt, and images—along with the agent's evidence-gathering loop, its full written recommendation, and every rubric score with the judge's reasoning. It is the fastest way to see why a variant scored the way it did, including the flagged imaging over-trust passages in full.
Open the case viewer → 20 cases × 22 agent versions × 5 runs: inputs, trace, answer, and every score.