Topology · Themes in depth05 themes

Topology

In-depth examinations of the themes that run through Dr. Khozin’s work and research. Each follows one argument from the evidence to what should change, and links to the others where the ideas meet.

  1. 01 AI · Drug discovery · Human biology The Simulator’s Dilemma Why better molecules are not better medicines.
  2. 02 Classification · Pathology · AI From Organs to Algorithms Classifying cancer by mechanism, using tumor, microenvironment and host data.
  3. 03 Measurement · Endpoints · Imaging The Broken Instrument How measurement error in tumor response criteria biases trial results.
  4. 04 Trials · External validity · Equity · Tool There Is No Average Patient Why trial results differ from real-world outcomes, and a tool to reweight them for a target population.
  5. 05 Benefit & risk · Statistics · Tool The Patient-Centered Estimand From relative hazard to absolute benefit, and a tool to express any trial as patients helped and harmed.

The Simulator’s Dilemma

Better molecules are not better medicines.

Artificial intelligence can now predict how molecules fit together, read the genome’s regulatory code and design proteins not found in nature. The productivity of drug development has not improved. This theme follows one molecule through the tools of its day and through today’s, and argues that AI’s biggest opportunity is not designing the molecule. It is discovering the human biology that decides whether the molecule works.

  1. 01

    The tools are real

    Structure prediction, genome models and protein language models are a genuine step change in how molecules are engineered.

  2. 02

    The outcomes have not changed

    AI-discovered molecules pass Phase I at 80–90% but Phase II at about 40%, in line with history. Drugs per R&D dollar remain far below their 1950 level.

  3. 03

    The bottleneck is biology

    Fragmented data, labels drawn from human perception, products shipped without clinical proof and how little of human biology is mapped limit what any model can learn.

Part I

A bet on simulation

“Every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it.”

McCarthy, Minsky, Rochester & Shannon · Dartmouth proposal · August 31, 1955

The 1955 Dartmouth proposal stated a conjecture: anything described precisely enough can be simulated by a machine. The same conjecture now underlies computational biology: if the protein, the cell or the patient can be described precisely enough, drug discovery becomes an engineering problem.

The history of AI includes two “winters,” periods of reduced funding and interest, each following a phase in which expectations exceeded what the methods could deliver.

Figure 1Eighty years, two winters. Select a period.

Part II

The molecule that could not

RG7112 was designed to free the tumor suppressor p53 by blocking MDM2, the protein that holds p53 in check. Many tumors keep p53 silent this way, so releasing it is a direct strategy: restore the cell’s own brake.

The molecule was developed with the best tools available at the time and cleared each preclinical test. Step through its development to see where it failed.

Figure 2RG7112, stage by stage
  1. Binding assays identified the nutlins, the first selective small-molecule antagonists of MDM2.

    Vassilev et al., Science 2004

  2. X-ray crystallography showed how the molecules sit in MDM2’s binding pocket (PDB 1RV1; RG7112 bound, 4IPF). In a binding assay, RG7112 reached an IC50 of 0.018 µM.

    Vu et al., ACS Med Chem Lett 2013

  3. Structure-guided chemistry tuned potency and metabolism.

    Vu et al., ACS Med Chem Lett 2013

  4. Mouse pharmacokinetic modeling showed about four times the exposure at half the dose, with a half-life of 8.8 hours.

    Vu et al., ACS Med Chem Lett 2013

  5. In SJSA-1 osteosarcoma xenografts, RG7112 inhibited tumor growth by 74% at 50 mg/kg and caused regression at 100 mg/kg.

    Vu et al., ACS Med Chem Lett 2013

  6. Toxicology studies supported a favorable safety profile. Every preclinical gate was cleared.

    Vu et al., ACS Med Chem Lett 2013

  7. In patients with MDM2-amplified liposarcoma, the drug engaged its target and activated p53 as designed. Efficacy was minimal and hematologic toxicity was dose-limiting.

    Benefit

    p53-pathway activation confirmed in tumor biopsies. Only one confirmed partial response.

    Harm

    Serious adverse events, including neutropenia and thrombocytopenia.

    Ray-Coquard et al., Lancet Oncol 2012

It hit its target and missed the patient.

Part III

New tools of the trade

“For the very first time in human history, biology has the opportunity to be engineering, not science.”

Jensen Huang, 2023

A decade later, four families of models cover most of the steps RG7112’s chemists performed by hand.

Predicting how molecules fit together

AlphaFold 3

Predicts the joint structures of complexes: proteins, nucleic acids, small molecules and more.

Abramson et al., Nature 2024

RFdiffusion

A diffusion model for de novo design of protein structure and function, such as new binders.

Watson et al., Nature 2023

Reading the genome’s regulatory code

AlphaGenome

Takes 1 Mb of DNA in and predicts thousands of functional tracks at up to single-base resolution. Matches or beats the best external models in 25 of 26 variant-effect evaluations.

Avsec et al., Nature 2026

Evo 2

40 billion parameters, a 1 Mb context, trained on more than 9 trillion nucleotides. Predicts and designs across DNA, RNA and proteins.

Brixi et al., Nature 2026

Designing proteins not found in nature

ESM3

A multimodal generative language model that reasons over protein sequence, structure and function. It produced a bright new fluorescent protein only 58% identical to known ones: a distance equivalent to about 500 million years of evolution.

Hayes et al., Science 2025

Language models join the lab

TxGemma

Language models (2B, 9B and 27B) that predict therapeutic properties with natural-language reasoning. They match or beat the best generalist model on 64 of 66 tasks. Wet-lab validation is still a future step.

Google DeepMind, 2025

AI co-scientist

A multi-agent system built on Gemini 2.0 that generates novel hypotheses and research proposals.

Google, 2025

Part IV

Would today’s tools catch it?

“It takes ten years and billions of dollars to design just one drug. We can maybe reduce that down from years to maybe months or maybe even weeks.”

Demis Hassabis, 2025

The main claim is speed. Early clinical results are favorable through Phase I; the advantage is not evident in Phase II, where efficacy is first tested.

80–90%Phase I success for AI-discovered molecules
~40%Phase II success: in line with historical industry rates

Analysis of AI-native biotech pipelines · Jayatunga et al., Drug Discov Today 2024

Phase I mostly tests whether a molecule is safe enough to dose. Phase II is where it first has to help patients, and that is where the advantage disappears. To see why, run RG7112 through today’s toolkit.

Figure 3RG7112 through today’s toolkit

Within the models’ view

  • Binding pose in MDM2 from AlphaFold-class structure prediction
  • Potency and drug-like properties from property models

Outside it

  • On-target p53 activation in bone-marrow progenitors, which led to neutropenia in 6 patients and thrombocytopenia in 3
  • The therapeutic window in humans

The toxicity was on-target: RG7112 activated p53 in bone-marrow progenitors, a human tissue context the design models did not represent. Ray-Coquard et al., Lancet Oncol 2012

Part V

Why Eroom’s law won’t break

Eroom’s law, Moore’s law spelled backward: since 1950, the number of new drugs approved per billion US dollars of R&D has halved roughly every nine years: about an 80-fold decline, adjusted for inflation, despite large advances in science, technology and management. Since about 2010 the decline has leveled off, with a modest recovery that Ringel and colleagues attribute largely to better information for choosing targets and a shift toward rare and genetically defined diseases, not to AI. AI-designed molecules have not yet changed the trend.

Figure 4Drugs approved per billion R&D dollars, relative to 1950. Drag through the years.
1950The baseline: drugs approved per billion R&D dollars

Schematic of the published trend, not individual data points: an 80-fold decline from 1950 to 2010, inflation-adjusted, halving roughly every nine years. The trend has flattened since 2010. Scannell et al., Nat Rev Drug Discov 2012 · Ringel et al., Nat Rev Drug Discov 2020

What we know often surprises us

Single driver mutations are the exception: most patients with metastatic non-small cell lung cancer and other advanced cancers may not have tumors with a single driver. And even a true driver depends on context.

Figure 5Vemurafenib, same mutation (BRAF V600E), objective response

Feedback signaling through EGFR in colorectal tumors explains part of the gap. Chapman et al., NEJM 2011 · Kopetz et al., J Clin Oncol 2015 · Khozin, Kim & Pazdur, Nat Rev Drug Discov 2017

Our gold standards can be blunt tools

Animal models are the standard gate between the bench and the clinic. In oncology, their predictive value for human toxicity is close to chance.

108Oncology drugs studied
0.65Median positive predictive value
0.50Median negative predictive value
90%+Of drugs safe and effective in animals that are not approved in humans

Animal-to-human toxicity in oncology · Atkins et al., Br J Cancer 2020 · FDA, Roadmap to Reducing Animal Testing, 2025

Part VI

What’s working today

Engineering at full speed

Where the biology is already understood, AI-assisted design is paying off. The clearest signal is in licensing: China-origin assets now account for a large share of big-ticket deals, built by applying fast, AI-assisted design at full scale to validated targets.

42%China-origin share of large-cap licensing deals ($50M+ upfront), H1 2026
68%Of upfront dollars in those deals

J.P. Morgan, Q2 2026 Biopharma Licensing and Venture Report

These molecules are mostly not first in class. They are better versions of known answers, and no new biological understanding is required to make them.

The quiet wins are in the back office

Ambient scribes

Capturing clinical documentation.

Documents

Document management and drafting of regulatory submissions.

Workflow

Automation across business operations.

These tools improve efficiency; they do not add biological insight.

Part VII

Why biology isn’t moving

Four obstacles keep AI’s progress in engineering from turning into progress in biology.

  1. 1Fragmentation
  2. 2Arbitrary gold standards
  3. 3Medicine is not SaaS
  4. 4Limited knowledge of human biology

1Fragmentation

Of data. Trials, health records, imaging, omics and outcomes sit in separate silos. A model learns only the slice it can see, and no single slice contains the patient.

Of work. AI itself was the by-product of one conversation across disciplines. In the 1940s and 1950s, neuroscience (McCulloch and Pitts’s neuron as a logic unit, 1943), mathematics (Turing’s work on logic and computability, 1936 and 1950) and computer science (stored programs and the Dartmouth workshop, 1956) were in the same room. Today these fields are highly specialized. Medicine and AI both require multidisciplinary work and multidisciplinary data.

Suppose someone invented a biosensor and wanted to use it to make life-and-death diagnoses.

The reasonable answer today: that’s premature. Convene a workshop. Talk to the FDA.

2Arbitrary gold standards

Human visual assessment is a variable instrument. Much of the ground truth used to train AI comes from human reading of images. In an FDA meta-analysis of 13,677 patients in registration trials, the investigator and an independent radiologist reading the same scans disagreed on tumor response about 30% of the time, and up to 45% in ovarian and pancreatic cancer. Assessments by eye vary within and between observers, in radiology and histopathology alike.

The labels themselves are historical accidents. Consider what “lung cancer” means.

Surgeon’s eye

“Lung cancer”: an early surgical observation of where a tumor sits.

Pathologist’s eye

“Small cell” vs “non-small cell”: what a pathologist sees through a microscope. “Non-small cell” defines a cancer by what it is not.

Underneath

Distinct molecular drivers, grouped together as one disease.

Train AI on these labels and, at best, it learns to reproduce them faster: with no gain in biological insight.

Khozin et al., FDA meta-analysis, 2017 · Khozin, Clin Transl Sci 2024

3Medicine is not SaaS

Software playbook

Ship it

A regulatory clearance feels like a launch. Technology companies and their investors shy away from clinical trials.

Medical product playbook

Prove it

Medical products need clinical validation. A 510(k) clearance alone does not win clinicians or payers.

The two models carry different evidentiary requirements. Pear Therapeutics, a pioneer of FDA-authorized prescription digital therapeutics, filed for bankruptcy in 2023. Regulatory acceptance of simulation-based evidence has meanwhile advanced faster than industry adoption.

  1. Oct 2024

    FDA Science Board report on new methods

  2. Apr 2025

    Plan to phase out animal testing; AI toxicity data welcomed in IND applications

  3. Apr 2025

    Roadmap: organ-on-chip, computational models, in vitro assays

  4. Mar 2026

    Draft guidance: a validation framework

  5. Next

    No validated simulation submission yet

Regulators now accept simulation-based evidence in defined settings; industry uptake has lagged.

4Limited knowledge of human biology

1–5%?

An illustrative estimate of the share of human biology that has been mapped: a small fraction. It counts only gaps that are already recognized; actual coverage may be lower.

The human genome encodes about 20,000 proteins, many still uncharacterized, and research keeps returning to the ones already well studied. By one systematic ranking of how much is known about each protein family, nearly a quarter of human families (23%) still sit at the very bottom. A simulator cannot simulate what no one has measured.

Rocha et al., PLOS Biol 2023

Part VIII

The way forward

Labels should come from mechanism, not from human sensory input

From organs to algorithms: a bottom-up classification of disease that reduces reliance on subjective human interpretation.

Current cancer classification compared with AI-driven classification
Dimension Current classification AI-driven classification
01Primary basisAnatomical location and histologyMolecular, imaging and histopathological data, integrated
02PrecisionLimited by inter- and intra-observer variabilityPotentially more objective and reproducible
03Novel featuresLimited ability to find new subtypesCan identify de novo features and patterns
04Biological relevanceMay group heterogeneous cancers togetherAims to reflect underlying disease mechanisms
05Treatment matchingOften suboptimalPotential for more precise selection

Adapted from Table 1, Khozin, Clin Transl Sci 2024

From big data to smart data: study the whole patient

Moving from a reductionist center, a single structured table, to a holistic edge that spans the volume, variety, velocity and veracity of data: records, trials, sensors, imaging and omics, joined around one patient.

Three moves to consider

  1. 01
    Fund mechanism-based labels

    Replace labels from the human eye with molecular and multimodal ones.

  2. 02
    Build whole-patient data

    Link trials, records, imaging and omics.

  3. 03
    Validate where decisions are made

    Measure AI against human outcomes, not benchmarks.

Better molecules are not better medicines.

AI’s biggest opportunity is not designing the molecule. It is discovering the human biology that decides whether it works. RG7112 engaged its target as designed. It failed on human biology the models did not capture.

“We can only see a short distance ahead, but we can see plenty there that needs to be done.”

Alan Turing, 1950

Connections

Where this theme meets the rest of the work.

References

Dartmouth proposal, 1955A Proposal for the Dartmouth Summer Research Project on Artificial IntelligenceMcCarthy J, Minsky ML, Rochester N, Shannon CE. Bull Math Biophys, 1943A logical calculus of the ideas immanent in nervous activityMcCulloch WS, Pitts W. Mind, 1950Computing machinery and intelligenceTuring AM. Science, 2004In vivo activation of the p53 pathway by small-molecule antagonists of MDM2Vassilev LT, et al. ACS Med Chem Lett, 2013Discovery of RG7112: a small-molecule MDM2 inhibitor in clinical developmentVu B, et al. Lancet Oncol, 2012Effect of the MDM2 antagonist RG7112 on the P53 pathway in patients with MDM2-amplified, well-differentiated or dedifferentiated liposarcomaRay-Coquard I, et al. Nature, 2024Accurate structure prediction of biomolecular interactions with AlphaFold 3Abramson J, et al. Nature, 2023De novo design of protein structure and function with RFdiffusionWatson JL, et al. Nature, 2026Advancing regulatory variant effect prediction with AlphaGenomeAvsec Ž, et al. Nature, 2026Genome modelling and design across all domains of life with Evo 2Brixi G, et al. Science, 2025Simulating 500 million years of evolution with a language modelHayes T, et al. arXiv, 2025TxGemma: efficient and agentic LLMs for therapeuticsGoogle DeepMind and Google Research. arXiv, 2025Towards an AI co-scientistGoogle. Drug Discov Today, 2024How successful are AI-discovered drugs in clinical trials? A first analysis and emerging lessonsJayatunga MKP, et al. Nat Rev Drug Discov, 2012Diagnosing the decline in pharmaceutical R&D efficiencyScannell JW, et al. Nat Rev Drug Discov, 2020Breaking Eroom’s LawRingel MS, Scannell JW, Baedeker M, Schulze U. N Engl J Med, 2011Improved survival with vemurafenib in melanoma with BRAF V600E mutationChapman PB, et al. J Clin Oncol, 2015Phase II pilot study of vemurafenib in patients with metastatic BRAF-mutated colorectal cancerKopetz S, et al. Nat Rev Drug Discov, 2017From big data to smart data: FDA’s INFORMED initiativeKhozin S, Kim G, Pazdur R. Br J Cancer, 2020Pre-clinical animal models are poor predictors of human toxicities in phase 1 oncology clinical trialsAtkins JT, et al., Khozin S. U.S. FDA, 2025 · 2026Roadmap to Reducing Animal Testing in Preclinical Safety Studies; draft guidance on new approach methodologiesU.S. Food and Drug Administration. J.P. Morgan, 2026Q2 2026 Biopharma Licensing and Venture ReportJ.P. Morgan. J Telemed Telecare, 2005Centenary of tele-electrocardiography and telephonocardiographyHjelm NM, Julius HW. Clin Transl Sci, 2024From organs to algorithms: redefining cancer classification in the age of artificial intelligenceKhozin S. PLOS Biol, 2023Functional unknomics: systematic screening of conserved genes of unknown functionRocha JJ, et al., Freeman M, Munro S.

From Organs to Algorithms

Cancer is classified by organ and histology. Response to treatment follows disease mechanism.

For more than a century, cancer has been classified by organ of origin and microscopic appearance. Molecular profiling has shown that tumors in different organs can share a driver, while tumors in the same organ can differ substantially. The goal set out here is a classification based on disease mechanism. Molecular profiling of tumor cells is one input to it. Mechanism also depends on features extrinsic to the tumor cell, including the immune and stromal microenvironment and the host, and these are measured across sequencing, pathology, imaging and clinical data. This theme covers how the current categories arose, what they cost patients, and how AI could derive categories from multimodal data rather than reproduce the existing ones.

  1. 01

    The names are historical accidents

    Anatomy and histology were the best instruments of their time. They were never designed to predict which drug will work.

  2. 02

    Molecular selection already improves outcomes

    In FDA lung cancer trials, about 4 patients were treated for one to benefit when patients were selected by molecular profile, against 11 without selection.

  3. 03

    Mechanism is multimodal

    Tumor-cell genomics is one layer. Microenvironment and host factors also determine response. Models trained on inherited labels reproduce those labels; models trained on integrated data can define new categories.

Part I

How cancer got its names

Every classification is a record of the instruments available when it was written. Cancer’s was written in layers: first by what could be seen, then by where it was found, then by what its cells looked like through a lens.

Figure 1Layers of a name. Select a period.

Each layer added knowledge. Despite sequencing, transcriptomics and single-cell proteomics, the system in routine use remains anchored in anatomical location and histology, a framework based on historical practice rather than current biological understanding. It limits how fully newer diagnostic methods are used, and how precisely therapies are matched to patients.

Khozin, Clin Transl Sci 2024

Part II

The promise of 100%

The premise of precision medicine is the right therapy for the right patient, at the right time and dose. Taken literally, it implies that every correctly matched patient should respond.

In practice, responses in trials and in routine care vary widely, and a substantial share of patients with metastatic disease derive little or no benefit. The limitation is less in the principle than in the matching: classification by organ and histology does not resolve the biological differences that determine response.

“What if the ‘noise’ is the signal? What if what we have been calling variation is actually heterogeneity: meaningful biological differences that determine whether a patient thrives or dies on a given therapy?”

Sean Khozin · The Ghost in the Statistical Machine, 2026

The average patient, the figure every trial reports, exists only in aggregate. A classification that groups distinct diseases together increases the heterogeneity behind that average and obscures the subgroups a drug could help.

Part III

Defined by what it is not

“Lung cancer” denotes the organ of origin, a surgical observation. “Non-small cell” is defined by exclusion: tumors that do not show small-cell morphology. Together they define a category by negation, and group malignancies with distinct molecular drivers, immune microenvironments and resistance mechanisms under a label that encodes none of them.

Where it sits

“Lung”: the organ, from the surgeon’s map.

What it is not

“Non-small cell”: a negative, from the microscope.

What drives it

EGFR, ALK, ROS1, BRAF, KRAS, RET, MET, HER2, NTRK… each a different disease, each with different drugs.

Labels drawn from human perception are also unstable. In an FDA analysis of 13,677 patients in registration trials, two expert readers looking at the same scans disagreed on tumor response about 30% of the time.

When these categories are used as training labels, models inherit them. Most AI imaging work still validates model outputs against existing “gold standard” criteria, which carries their assumptions forward.

“When these inherited categories become training labels for foundation models, the models do not simply inherit their limitations; they amplify them.”

Sean Khozin · The Simulator’s Dilemma, Revisited, 2026

Part IV

The numbers say it already

Across 30 lung cancer trials that supported FDA decisions from 2003 to 2017, researchers counted how many patients had to be treated for one to benefit. The result tracked molecular selection of patients more closely than drug class.

Figure 2Patients treated for one to benefit, advanced non-small cell lung cancer
4Approved targeted therapies in molecularly selected patients
11Approved therapies without molecular selection
23Therapies withdrawn or never approved

Averages across subpopulations, number needed to benefit. Each row shows one patient who benefits among those treated. Raju, Khozin et al., JCO Clin Cancer Inform 2020

The same pattern shows up in response rates. With doublet chemotherapy in morphologically defined non-small cell lung cancer, response rates are about 30%. In molecularly defined populations they commonly range from 50% to 90%.

Khozin, The Simulator’s Dilemma, Revisited, 2026

When machine-learning models were trained on patient-level data from 17 FDA-submitted lung cancer trials, biomarker status was the strongest predictor of response, progression-free survival and overall survival. But single biomarkers had limited predictive value, especially PD-L1 for immunotherapy. The authors concluded that further gains require composite, multiomic signatures that integrate many measurements.

Siah, Khozin, Wong & Lo, JCO Clin Cancer Inform 2019

Part V

Lung cancer, split by split

“‘Lung cancer’ has decomposed into a taxonomy of molecular species.”

Sean Khozin · The Ghost in the Statistical Machine, 2026

Since 2011, FDA approvals have subdivided “non-small cell lung cancer” into biomarker-defined populations. Drag through the years to see the sequence.

Figure 3Biomarker-defined FDA approvals in non-small cell lung cancer
2010

    Selected approvals by year of first FDA approval for each biomarker-defined population. U.S. FDA approval records

    Once a population is defined by its driver, a small study can support approval. In March 2016, FDA approved crizotinib for ROS1-positive lung cancer on the strength of about fifty patients from a single-arm study, in which independent review found a 66% response rate.

    Khozin, Fifty Patients, 2026

    Part VI

    Re-sort the tumors

    Group tumors by organ and each group mixes different diseases, while the same driver turns up in many groups. Grouped by driver, tumors from different organs share a therapeutic target.

    Figure 4The same tumors, sorted two ways

    Illustrative: each tile is a tumor type in which the driver occurs, not its frequency. Approvals: U.S. FDA.

    Regulators have begun to classify this way. In May 2017, FDA approved pembrolizumab for any solid tumor with high microsatellite instability or mismatch-repair deficiency: the first approval defined by a biomarker rather than an organ. It came during Dr. Khozin’s tenure at FDA, at a time when moving away from anatomy was, in his words, “at best an ambiguous concept for many stakeholders.” Approvals for NTRK fusions, BRAF V600E, RET fusions and HER2 followed.

    An earlier precedent is imatinib. Developed against the BCR-ABL kinase in chronic myeloid leukemia, it was later approved for gastrointestinal stromal tumors, which are driven by KIT, another kinase it inhibits. It showed that a targeted drug can cross disease boundaries when the mechanism is shared.

    Khozin, Unearthing Mechanistic Foundations of Oncological Diseases with AI, 2023 · Khozin, The Discipline of Precision, 2026

    Part VII

    What algorithms can see

    In 2016, in the second game against Lee Sedol, AlphaGo played Move 37: a stone outside established human strategy. Commentators first read it as an error; it proved decisive.

    Dr. Khozin calls these non-obvious, or “alien,” constructions: solutions that are better than the human ones and that lie outside the set of answers humans would think of. In pathology and imaging, the alien constructions are features: patterns in tissue and scans, in tumor cells and in their surroundings, that predict biology and outcome and have not been described by human observers. The definition is formal.

    Let S be the set of all possible solutions, H ⊂ S the set of human-conceivable solutions, h : S → ℝ a measure of solution quality and d a distance on S. A solution s* ∈ S is an alien construction if all four hold:

    1. 1s* ∈ arg maxs∈S h(s)Optimal
    2. 2s* ∉ HNot human-conceivable
    3. 3mins∈H d(s, s*) > ε,  ε > 0Far from every human solution
    4. 4P(s* ∣ human knowledge) < δ,  δ smallImprobable under human priors

    Definition 1, Khozin, Non-Obvious Alien AI Constructions (preprint, not peer-reviewed), 2024

    Figure 5Testing Definition 1 on a solution landscape

    Drag H to move current practice; drag the white handle to widen human exploration (σ). The red band is the ε-neighborhood of H.

    1. 1Optimal
    2. 2Outside H
    3. 3Distance to H > ε
    4. 4P < δ

    Best human solution, h
    Global optimum, h(s*)
    Gain left outside H

    How the figure computes each condition. Shading is solution quality h, a mixture of Gaussian peaks; s* is its maximum on a 160 × 100 grid. Human knowledge is a Gaussian prior centered on current practice with spread σ; H is its 95% highest-density region, a disc of radius 2.45σ (dashed rings mark 1σ and 2σ). Distance is Euclidean. P(s* ∣ human knowledge) is the prior probability of proposing a solution at least as atypical as s*, exp(−r² / 2σ²) for a point r from the center. Widening σ models broader human exploration: at some point the optimum becomes conceivable and stops being alien.

    Construction for illustration; the definition is from the preprint. Silver et al., Nature 2016

    Already in the literature

    Reading slides as an expert does

    Expert-level grading

    A deep-learning system for Gleason grading of prostate biopsies agreed with expert uropathologists and outperformed general pathologists.

    Bulten et al., Nat Med 2022

    Faster, surer diagnosis

    AI assistance cut diagnosis time by up to 62% for plasma cell identification and 33% for lymph-node metastases, and reduced diagnostic uncertainty and requests for extra tests by over 60%.

    Huang et al., Nat Biomed Eng 2024

    Finding what no one named

    New prognostic markers

    Patterns in the stromal environment of colorectal tumors predicted survival; slide analysis found a distinct subset of endometrial cancers with markedly worse outcomes.

    Kather et al., PLoS Med 2019 · Darbandsari et al., Nat Commun 2024

    Features shared across cancers

    Four imaging features common to three malignancies, each with its own molecular character and prognostic meaning.

    Wu et al., Nat Mach Intell 2021

    Molecules from morphology

    Mutations from a stained slide

    Microsatellite instability and POLE mutations detected from routine H&E slides in colorectal cancer; across cancers, morphology predicted specific mutations.

    Gustav et al., NPJ Precis Oncol 2024 · Fu et al., Nat Cancer 2020

    Expression, and response

    Genome-wide tumor mRNA expression inferred from slides, then used to predict response to targeted and immune therapies.

    Hoang et al., Nat Cancer 2024

    Structure in space

    Tumor microregions

    Across 131 tumor sections from six cancer types, spatial transcriptomics, multiplexed imaging and single-nucleus sequencing revealed distinct microregions, each with its own genetic clones. Treatment-resistant subclones tended to form larger, metabolically active microregions.

    Mo et al., Nature 2024

    Imaging beyond the eye

    A lung-cancer screening model outperformed experienced radiologists in detection; image-based deep learning identified patients likely to respond better to radiotherapy.

    Ardila et al., Nat Med 2019 · Lou et al., Lancet Digit Health 2019

    Trained models also produce a learned representation, the latent space, in which data is compressed and organized into clusters that are not apparent to human observers. New disease categories are most likely to be found there, if the representation is analyzed directly rather than mapped back onto existing labels.

    Khozin, Unearthing Mechanistic Foundations of Oncological Diseases with AI, 2023

    Part VIII

    Beyond the molecular profile

    Current biomarker-defined categories rest mostly on one layer: genomic alterations in tumor cells. That layer is necessary but incomplete. The same driver mutation can produce different outcomes depending on the immune infiltrate, the stroma, tissue architecture and the host. A mechanism-based classification integrates features intrinsic to the tumor cell with features extrinsic to it.

    Figure 6Inputs to a mechanism-based classification

    Intrinsic to the tumor cell

    • GenomeDriver mutations, fusions, copy number, mutational signatures, tumor mutational burden
    • EpigenomeDNA methylation, chromatin state
    • TranscriptomeExpression programs, cell states, single-cell heterogeneity
    • ProteomeProtein and phosphoprotein levels, pathway activity
    • MorphologyNuclear and cellular features in tissue
    • Clonal architectureSubclones and their evolution under treatment

    Extrinsic: microenvironment

    • Immune contextureDensity, type and location of T cells, macrophages and other immune cells; PD-L1 expression
    • StromaCancer-associated fibroblasts, extracellular matrix
    • Vasculature and metabolismAngiogenesis, hypoxia, nutrient gradients
    • Spatial organizationMicroregions and cell neighborhoods within the tumor

    Extrinsic: host

    • GermlineInherited variants, including pharmacogenomic ones
    • Systemic stateSystemic immunity, inflammation, microbiome, comorbidity, prior therapy

    Measured by DNA and RNA sequencing, including single-cell · immunohistochemistry and multiplex imaging · H&E whole-slide images · spatial transcriptomics and proteomics · radiology · circulating tumor DNA · clinical records

    Highlighted, the layer behind most biomarker-defined approvals to date. Most of the profile sits outside it. After Khozin, Clin Transl Sci 2024; Mo et al., Nature 2024

    AI-driven classification reverses the current direction of diagnosis. It begins with integrated data from the tumor, its microenvironment and the host, together with computationally extracted features, and derives categories from them, rather than starting from the organ.

    Current cancer classification compared with AI-driven classification, ten features
    Feature Current classification AI-driven classification
    01Primary basisAnatomical location and histological featuresIntegration of molecular, imaging, histopathological and clinical characteristics
    02OriginHistorical medical practicesReflects the evolving understanding of biological processes
    03PrecisionLimited by inter- and intra-observer variabilityPotentially more objective and reproducible
    04Molecular insightLimited integration of molecular dataComprehensive integration of multi-omics data
    05Novel featuresLimited ability to identify new subtypesCan identify de novo features and patterns
    06Data integrationPrimarily a single modalityIntegrates imaging, pathology, genomics and more
    07Treatment matchingOften leads to suboptimal therapy matchingPotential for more precise treatment selection
    08Biological relevanceMay group heterogeneous cancers togetherAims to reflect underlying disease mechanisms more accurately
    09StandardizationVaries across institutionsPotential for standardized diagnostic criteria
    10Prognostic valueLimited by broad groupings of heterogeneous patientsCan reveal new prognostic indicators

    Table 1, Khozin, Clin Transl Sci 2024

    Part IX

    What it will take

    A computational classification must meet the same evidentiary standard as any other diagnostic claim. The main obstacles are known.

    1. 01
      Standardized, integrated data

      Robust multi-center datasets that link molecular, imaging, pathology and clinical records.

    2. 02
      Interpretability

      Models whose reasoning can be examined in biological terms.

    3. 03
      Clinical validation

      Evaluation guidelines and prospective trials, not only retrospective benchmarks.

    4. 04
      Ethics

      Attention to bias in who is represented in the data that defines a disease.

    5. 05
      Teams and infrastructure

      Clinicians, researchers and AI specialists working together, with the computing to match.

    Khozin, Clin Transl Sci 2024 · Elemento, Khozin & Sternberg, NEJM AI 2025

    “A mutation list is not a disease model. A pathway diagram is not an intervention strategy. A biomarker is not a mechanism unless the biology makes it one.”

    Sean Khozin · The Discipline of Precision, 2026

    Classify by mechanism, measured across modalities.

    When training data captures proxies, models learn proxies. A classification built on mechanism, drawing on the tumor cell, its microenvironment and the host, is more likely to identify the patients a therapy will help.

    Connections

    Further reading

    References

    Clin Transl Sci, 2024From organs to algorithms: redefining cancer classification in the age of artificial intelligenceKhozin S. JCO Clin Cancer Inform, 2020Patient-centered approach to benefit-risk characterization using number needed to benefit and number needed to harm: advanced non-small-cell lung cancerRaju GK, Khozin S, et al. JCO Clin Cancer Inform, 2019Machine-learning and stochastic tumor growth models for predicting outcomes in patients with advanced non-small-cell lung cancerSiah KW, Khozin S, Wong CH, Lo AW. NEJM AI, 2025The use of artificial intelligence for cancer therapeutic decision-makingElemento O, Khozin S, Sternberg CN. Preprints, 2024 · not peer-reviewedNon-obvious alien AI constructions: opportunities and implicationsKhozin S. Oncologist, 2014FDA approval summary: erlotinib for the first-line treatment of metastatic NSCLC with EGFR exon 19 deletions or exon 21 (L858R) substitution mutationsKhozin S, et al. Clin Cancer Res, 2015FDA approval: ceritinib for the treatment of metastatic anaplastic lymphoma kinase-positive non-small cell lung cancerKhozin S, et al. Clin Cancer Res, 2016Osimertinib for the treatment of metastatic EGFR T790M mutation-positive non-small cell lung cancerKhozin S, et al. Nature, 2024Tumour evolution and microenvironment interactions in 2D and 3D spaceMo CK, et al. Nat Med, 2022Artificial intelligence for diagnosis and Gleason grading of prostate cancer: the PANDA challengeBulten W, et al. Nat Biomed Eng, 2024A pathologist–AI collaboration framework for enhancing diagnostic accuracies and efficienciesHuang Z, et al. PLoS Med, 2019Predicting survival from colorectal cancer histology slides using deep learningKather JN, et al. Nat Cancer, 2020Pan-cancer computational histopathology reveals mutations, tumor composition and prognosisFu Y, et al. Nat Commun, 2024AI-based histopathology image analysis reveals a distinct subset of endometrial cancersDarbandsari A, et al. NPJ Precis Oncol, 2024Deep learning for dual detection of microsatellite instability and POLE mutations in colorectal cancer histopathologyGustav M, et al. Nat Cancer, 2024A deep-learning framework to predict cancer treatment response from histopathology images through imputed transcriptomicsHoang DT, et al. Nat Med, 2019End-to-end lung cancer screening with three-dimensional deep learning on low-dose chest computed tomographyArdila D, et al. Lancet Digit Health, 2019An image-based deep learning framework for individualising radiotherapy doseLou B, et al. Nat Mach Intell, 2021Radiological tumour classification across imaging modality and histologyWu J, et al. Nature, 2016Mastering the game of Go with deep neural networks and tree searchSilver D, et al.

    The Broken Instrument

    What we measure determines what we can find.

    For twenty-five years, nearly every solid-tumor drug approved on tumor-based endpoints has rested on one instrument: RECIST, a convention for measuring lesion diameters on CT. Two expert readers applying it to the same images disagree about a third of the time. This theme follows what that disagreement does in a trial: it attenuates true effects, discards information, makes the classification of borderline patients unstable, and influences which drugs are developed. It also describes how volumetric, machine-read measurement could replace it.

    1. 01

      The ruler is a convention

      RECIST certifies drugs, but almost no one uses it to treat patients. Its authority comes from agreement, not validation.

    2. 02

      Noise has a direction

      Random disagreement does not cancel out. It shrinks observed effects, inflates trial sizes and tilts results toward false negatives.

    3. 03

      The substrate is replaceable

      Volumetric, continuous, machine-read measurement can now be the same in the trial and in the clinic.

    Part I

    A ruler against a shadow

    RECIST (Response Evaluation Criteria in Solid Tumors) works like this. A radiologist picks up to five target lesions, no more than two per organ, and measures each along its longest diameter (lymph nodes along their short axis). The diameters are summed. Shrink by 30% or more from the baseline sum and the patient has a partial response. Grow by 20% or more from the smallest sum recorded on study, and by at least 5 mm, or develop a new lesion, and it is progressive disease. Anything in between is stable disease. Disappearance of all target lesions is a complete response.

    Figure 1Read a scan. Drag the change in the sum of diameters.
    0%Stable disease

    Objective response rate, the endpoint behind many accelerated approvals, is the share of patients read as complete or partial response. The figure assumes the baseline is also the smallest sum, and leaves out the 5 mm rule and new lesions. Eisenhauer et al., Eur J Cancer 2009 (RECIST 1.1)

    RECIST is used to evaluate drugs in trials, not to manage patients in routine care. Its history explains its design.

    Figure 2Fifty years of the ruler. Select a period.

    Part II

    An old warning

    The problem was measured before the modern criteria existed. In 1976, sixteen experienced oncologists each measured twelve simulated tumor masses with their usual clinical methods. Unknown to them, some masses were identical in size. Comparing their measurements of identical masses showed how many “responses” measurement error alone would produce.

    7–8%“Responses” from error alone at a 50% shrinkage criterion
    19–25%“Responses” from error alone at a 25% criterion

    Moertel & Hanley, Cancer 1976

    The authors advised investigators to expect a 5–10% response rate from human error in measurement alone. Since then the criteria have moved from two diameters to one; measurement is still performed manually by human readers.

    Part III

    Two readers, one scan

    Many registration trials read every scan twice: by the investigator and by a blinded radiologist at a central facility, each unaware of the other. In 2017, Dr. Khozin and colleagues at FDA pooled a decade of these paired reads.

    13,677Paired assessments
    69.5%Agreed on the response category
    ≤45%Disagreement in hard-to-measure cancers such as pancreatic and ovarian

    In the language of measurement theory, this corresponds to a weighted kappa (agreement beyond chance) of roughly 0.30 to 0.55: between “fair” and “moderate” on the scale most often used to grade it. That is what one expects from a diagnostic tool in early development, not from the instrument on which oncology drug development rests.

    Landis & Koch, Biometrics 1977 · Khozin, The Broken Instrument Beneath the Evidence, 2026

    The standard defense is procedural. Disagreement is random, spread evenly across both arms of a randomized trial, so the comparison stays fair and the p-value stays honest. That is partly true. It is also incomplete, and many cancer drugs are advanced on single-arm studies with no comparison arm at all.

    Part IV

    The attenuation

    When random misclassification enters a comparison of two proportions, the difference between them does not stay the same. It shrinks. If each patient is misclassified with probability m, the observed difference between arms is the true difference multiplied by (1 − 2m). The result goes back at least to Irwin Bross in 1954, and it is rarely invoked in trial design.

    Figure 3How much of a real effect survives the measurement
    Presets
    Observed difference
    Of the real effect lost
    Patients needed, if 200 would suffice with no error ()

    Two readings of “30%.” Treating the 30% disagreement between readers as each patient’s misclassification rate gives the essay’s illustration: a 20-point advantage appears as 8 points, and a 200-patient trial needs about 1,250. If instead 30% is the disagreement between two readers who each err independently, each reader’s own error rate is about 18%, and the loss is closer to 37%, with about 2.5 times the patients. Either way the direction is the same.

    Bross, Biometrics 1954 · simplified two-category model; sample size scales with the inverse square of the effect

    The bias has a direction. Random measurement error biases observed effects toward the null. At the level of the instrument, that produces a systematic excess of false negatives over false positives. A trial that misses statistical significance is not necessarily a trial of an inactive drug. Some fraction of negative trials in solid-tumor oncology are trials of active drugs measured with an instrument that could not see them.

    No one knows the size of that fraction. A negative trial does not record whether the drug or the measurement failed.

    Part V

    Four boxes for a continuous world

    Underneath the categories, the biology is continuous: the percent change in tumor burden can take any value. RECIST turns it into four ordered boxes, and response rate collapses those into two. A four-level category carries at most two bits of information; a yes-or-no outcome carries at most one. A continuous measurement carries several times more.

    Figure 4Three drugs, thirty patients each. Best change in tumor size.
    Objective response rate
    Patients whose tumors shrank
    Mean change in tumor size

    Simulated patients, for illustration

    A drug that shrinks every patient’s tumors by a quarter, an effect that implies real activity, can post a response rate of zero, the same as an inactive drug. Trial designs that keep tumor change continuous, with a control group, were proposed precisely to stop discarding that information.

    Karrison, Maitland, Stadler & Ratain, J Natl Cancer Inst 2007 · Shannon, Bell Syst Tech J 1948

    Part VI

    Instability at the cutoff

    Consider a patient whose true tumor burden fell by 28%: two points short of a partial response. If readers’ measurements vary by about ten percentage points, as the disagreement data suggest, how often will this patient be called a responder?

    Figure 5The same tumor, read many times
    Partial response
    Stable disease
    Progressive disease

    Share of reads falling in each category, assuming normally distributed reader error around the true change. Khozin, The Broken Instrument Beneath the Evidence, 2026

    Forty-two reads in a hundred call it a partial response; fifty-eight call it stable disease. Near the threshold, classification is close to random, and the reported effect depends on how many borderline patients fall on the responder side.

    The instability does more than add noise. Because response rate cuts sharply at minus 30%, sponsors are rewarded for drugs that produce deep responses in a few patients, and drugs that produce shallow but consistent shrinkage across many are structurally disfavored. The shape of the oncology pipeline, enriched for responses the human eye can see, follows from the endpoint. It selects for what the instrument can detect, not necessarily for what prolongs life.

    Part VII

    An instrument only trials use

    Community oncologists, who treat most patients with cancer, do not sum diameters. They read narrative radiology reports (“stable appearance of the right upper lobe mass”) and combine them with clinical judgment. When Dr. Khozin showed the reader-disagreement data to Richard Pazdur, FDA’s longtime head of oncology, Pazdur was not surprised. “You know, Sean, I never liked RECIST,” he said. What he trusted most was the “lipstick sign”: a frail patient returning to clinic after a few cycles of treatment, color back in her face, wearing bright red lipstick.

    23%Patients for whom RECIST progression could be determined from health-record data (6 of 26)
    87%With a real-world progression event using radiology- and clinician-anchored approaches

    Griffith, … Khozin, … Miksad, Adv Ther 2019

    So the variable on which approval rests is absent from the data routine care generates, and trial results cannot be checked against practice on the same scale.

    “Who has validated ORR as a surrogate?”

    A true surrogate, in Ross Prentice’s 1989 definition, must capture the treatment’s effect on the outcome that matters, so that once the surrogate is known, no residual effect remains. Response rate has never been validated this way. When Dr. Khozin, early in his FDA tenure, described response rate as a surrogate, Robert Temple, then deputy center director for clinical science, asked the question above. It has not been answered.

    0.89R², response rate vs progression-free survival, across FDA lung cancer trials
    0.09R², response rate vs overall survival
    0.08R², progression-free vs overall survival

    Blumenthal et al. (incl. Khozin), J Clin Oncol 2015 · 14 trials, 12,567 patients

    Across FDA approvals from 2009 to 2014 that rested on surrogate endpoints, more than half of accelerated approvals had no formal analysis of how well the surrogate tracked survival; where analyses existed, correlations were mostly low. Even the strongest precedent shows the difficulty: pathological complete response in early breast cancer predicts survival powerfully for individual patients, especially in aggressive subtypes, yet the pooled analysis could not validate it as a trial-level surrogate.

    Prentice, Stat Med 1989 · Kim & Prasad, Mayo Clin Proc 2016 · Cortazar et al., Lancet 2014 · Khozin, The Misnomer of Surrogate Endpoints in Oncology, 2025

    Part VIII

    Machines, readers, references

    Run one chest CT through a measurement algorithm a thousand times and it returns the same number a thousand times. Run it through a generative AI system and it returns a distribution of reports, mostly in agreement. Give it to a thousand radiologists and they disagree on the response category roughly 30% of the time.

    Algorithm

    Identical output on every run. Perfectly reproducible, which is not the same as correct.

    Generative AI

    A distribution of reports. Reliability can be measured run by run.

    Radiologists

    About 30% disagreement on the category that decides whether a drug stays in development.

    Two principles follow. Reliability is necessary for validity, but not sufficient: a system can return the same wrong answer every time, and consistency reads as confidence. And when machines are judged against a panel of clinicians, the panel itself has to be measured. A device that agrees with a three-person panel 85% of the time sounds deficient until one learns the panelists agree with each other only 78% of the time. Machines and readers should report reliability on the same scales: kappa for categories, intraclass correlation for continuous measures.

    Khozin, A Thousand Runs, 2026 · the 85% and 78% figures are an illustration

    Part IX

    Replacing the substrate

    In 2009, the RECIST working group considered moving to volumetric or functional measurement and concluded that there was not yet enough standardization or evidence to abandon the diameter. That condition has changed. Volumetric segmentation is now routine, imaging foundation models are trained on more scans than any radiologist reads in a career, and automated assessment can run at scale in trials and in community care.

    RECIST compared with a volumetric, continuous, machine-read measure
    Property RECIST Total Tumor Burden Index
    01GeometryLongest diameter of up to five lesionsVolume of all disease
    02ScaleFour categories, then a binaryContinuous, with no thresholds to straddle
    03ReaderHuman, about 30% disagreementMachine-generated, reproducible across sites and re-reads
    04SettingTrials onlyThe same variable in trials and routine care

    The regulatory path will take time: prospective comparison with RECIST, integration across imaging systems, agreement on validation. None of these barriers is technical.

    Replace the measurement, not the statistics.

    The choice of measurement determines which effects can be detected, which drugs are developed and, in turn, which patients are treated. Diameter-based RECIST is a convention; reproducible alternatives are now available.

    Connections

    Further reading

    References

    Cancer, 1976The effect of measuring error on the results of therapeutic trials in advanced cancerMoertel CG, Hanley JA. Cancer, 1981Reporting results of cancer treatmentMiller AB, Hoogstraten B, Staquet M, Winkler A. J Natl Cancer Inst, 2000New guidelines to evaluate the response to treatment in solid tumorsTherasse P, et al. Eur J Cancer, 2009New response evaluation criteria in solid tumours: revised RECIST guideline (version 1.1)Eisenhauer EA, et al. Lancet Oncol, 2017iRECIST: guidelines for response criteria for use in trials testing immunotherapeuticsSeymour L, et al. Biometrics, 1977The measurement of observer agreement for categorical dataLandis JR, Koch GG. PMID 843571. Biometrics, 1954Misclassification in 2 × 2 tablesBross I. Bell Syst Tech J, 1948A mathematical theory of communicationShannon CE. J Natl Cancer Inst, 2007Design of phase II cancer trials using a continuous endpoint of change in tumor sizeKarrison TG, Maitland ML, Stadler WM, Ratain MJ. Stat Med, 1989Surrogate endpoints in clinical trials: definition and operational criteriaPrentice RL. J Clin Oncol, 2015Overall response rate, progression-free survival, and overall survival with targeted and standard therapies in advanced non-small-cell lung cancer: FDA trial-level and patient-level analysesBlumenthal GM, et al., including Khozin S. Mayo Clin Proc, 2016Strength of validation for surrogate end points used in the US Food and Drug Administration’s approval of oncology drugsKim C, Prasad V. Lancet, 2014Pathological complete response and long-term clinical benefit in breast cancer: the CTNeoBC pooled analysisCortazar P, et al. Adv Ther, 2019Generating real-world tumor burden endpoints from electronic health record data: comparison of RECIST, radiology-anchored, and clinician-anchored approachesGriffith SD, … Khozin S, … Miksad RA.

    There Is No Average Patient

    A trial estimates the effect in the patients it enrolled. Clinical practice applies it to a broader population.

    Real-world outcomes for patients with cancer are often worse than those reported in the trials that led to a drug’s approval. Part of the gap comes from who gets into trials: narrow eligibility, a handful of specialized sites, and unequal access leave older patients, patients with other illnesses, and many minority groups underrepresented. This theme shows how much that gap can change a drug’s expected benefit, even when the drug works equally well for everyone, and gives clinicians and researchers a tool to make the adjustment themselves.

    1. 01

      Trials enroll a narrower world

      Fewer than half of exclusion criteria in major trials are strongly justified, and many groups enroll at a fraction of their share of the disease.

    2. 02

      The gap appears even without heterogeneity

      In a worked example, a trial reports 13 patients treated per benefit. The same drug, equally effective in everyone, needs 18 in the real-world mix.

    3. 03

      The adjustment is arithmetic

      Reweight the trial’s subgroup results to the patients you actually treat. The tool below does it in your browser.

    Part I

    The ghost of the average

    In 1747, aboard HMS Salisbury, James Lind divided twelve sailors with scurvy into pairs and gave each pair a different remedy. The pair given citrus recovered. The experiment is often cited as one of the first controlled clinical trials. It also introduced a construct that has shaped medicine since: the average patient, who exists only in aggregate.

    “The Average Patient does not exist, yet its statistical ghost has served us well. It has saved millions. But it has also exacted a heavy price: the systematic erasure of the particular in service of the universal.”

    Sean Khozin · The Ghost in the Statistical Machine, 2026

    Randomized trials are built for internal validity: making sure the observed effect is real for the people studied. What they trade away is external validity: whether the result holds for the people who will actually be treated. Clinicians at the point of care usually know their patients far more granularly than any trial’s average, yet are expected to accept its result at face value.

    Part II

    Who gets into trials

    47%Of exclusion criteria in high-impact trials graded “strongly justified”
    22%Black patients’ enrollment in FDA oncology approval trials, relative to their share of US cancer incidence
    48%Of real-world advanced lung cancer patients excluded by three traditional criteria

    Van Spall et al., JAMA 2007 · Loree et al., JAMA Oncol 2019 · Harvey, … Khozin, … Kim, Clin Cancer Res 2021

    The pattern is old and persistent. In National Cancer Institute cooperative group trials from 2000 to 2002, about 3% of patients aged 30 to 64 enrolled, against 1.3% of those aged 65 to 74 and 0.5% of those 75 and older; Black and Hispanic patients enrolled at lower rates than White patients. Across 230 trials behind FDA oncology approvals from 2008 to 2018, Black patients made up 3.1% of participants and Hispanic patients 6.1%.

    The renal cancer trial Dr. Khozin and Roger Stein use as an example enrolled patients who were overwhelmingly White or Asian, although about 12% of real-world patients with renal cell carcinoma are Black: an underrepresentation of at least 95%. In 2022, FDA declined to approve a PD-1 inhibitor whose pivotal trial was conducted in a single country, a signal that representativeness had become a regulatory question.

    Murthy, Krumholz & Gross, JAMA 2004 · Loree et al., JAMA Oncol 2019 · Khozin & Stein, BBA Rev Cancer 2022

    Part III

    Three kinds of error

    A trial’s estimate can miss the real-world effect for three distinct reasons. Trial design takes care of the first. The third is measurable, and usually ignored.

    1. 01
      Sampling variability

      Chance. Handled by powering the trial and setting a threshold such as p < 0.05.

    2. 02
      Unknown unknowns

      Confounders no one has measured yet. Only identifiable after new discoveries.

    3. 03
      Sampling-frame error

      Known, measurable differences between the trial’s population and the real one. This is what can be adjusted.

    After Figure 2, Khozin & Stein, BBA Rev Cancer 2022

    The usual account says external validity fails when a treatment works differently in different groups, those groups have different chances of entering the trial, and the two are correlated. Those conditions are sufficient. They are not all necessary. The trial’s answer can drift from the real world’s even when the drug works the same in every group.

    Part IV

    Same drug, different answer

    Consider a hypothetical cancer and three groups of patients with different death rates on the control therapy: 80% in group A, 16.7% in group B and 75% in group C. The real-world patient population is 1.1% A, 33.5% B and 65.3% C. A trial with restrictive criteria enrolls 30% A, no B and 70% C. The new drug cuts the risk of death by 10% in relative terms.

    Absolute risk reduction and number needed to treat in the trial and in the real world
    Measure Trial, as reported Real world, same effect for all Real world, no effect in group B
    Absolute risk reduction7.6%5.6%5.0%
    Treated per benefit1318 (+38%)20 (+54%)

    Examples 3 and 4, Table 2, Khozin & Stein, BBA Rev Cancer 2022 · hypothetical numbers

    The trial is not wrong about its own patients. It is answering a question about a population that does not exist. And the error can run the other way: a trial that excludes the patients most likely to benefit understates a drug’s value, and patients go without a treatment that would have helped them.

    Part V · Tool

    Reweight the trial

    Enter the groups that matter for your question: age bands, comorbidity, performance status, ancestry, or anything else a trial reports by subgroup. For each, give its share of the trial, its share of the population you treat, its event rate on the control therapy, and the treatment’s effect relative to that control. The tool recomputes what the trial would have reported for your population, and for one patient.

    Start from
    Treatment effect
    Effect expressed as
    Trial
    Real world

      Trial, as reported

      Event rate, control
      Event rate, new treatment
      Absolute risk reduction

      Your population

      Event rate, control
      Event rate, new treatment
      Absolute risk reduction

        One patient

        Event rate on control
        Event rate treated
        (absolute reduction )

        Everything is calculated in your browser. Nothing you enter is sent anywhere; the link you copy carries the numbers in its address.

        Part VI

        What the tool cannot do

        Invent evidence

        For a group the trial left out, the effect you enter is an assumption. Try a range: no effect, the trial’s effect, and something in between.

        See the unmeasured

        It corrects for differences you can measure, not for factors no one recorded.

        Model time

        It uses one outcome at one time point. Read survival at, say, two years off each arm’s curve and enter one minus that value as the event rate.

        The choice of scale matters. The paper’s worked examples hold the relative risk constant across groups; its shortcut formula holds the odds ratio constant. They agree when events are rare and diverge when they are common, which is often the case in advanced cancer. Formal methods exist for richer data, including standardizing to a target population and correcting for selection into the trial, but they need patient-level data. This tool is the first-order version: enough to see whether the gap is large.

        Cole & Stuart, Am J Epidemiol 2010 · Heckman, Econometrica 1979 · Rothwell, Lancet 2005

        Part VII

        Fixing the trial itself

        Statistical adjustment is an interim measure. The durable solution is trial populations that resemble the patients who will be treated.

        1. 01
          Broaden eligibility

          In 10,500 real-world patients with advanced lung cancer, three traditional criteria excluded 48%. Broadened versions admitted 98%, adding more women, more patients aged 75 and older, and more with stage IV disease.

        2. 02
          Bring trials to patients

          Decentralized and pragmatic designs reach patients beyond a few specialized centers.

        3. 03
          Borrow strength from real-world data

          Hybrid trials combine randomization with external control data from completed studies and health records.

        Harvey, … Khozin, … Kim, Clin Cancer Res 2021 · Khozin & Coravos, Clin Pharmacol Ther 2019 · Ventz, Khozin, … Trippa, Nat Commun 2022

        Real-world data already shows the gap. In the first year after approval, patients with metastatic lung cancer treated with PD-1 inhibitors in community clinics had a median overall survival of 8.0 months, shorter than in the pivotal trials, possibly because of narrow trial eligibility.

        Khozin et al., Oncologist 2019

        There is no average patient.

        The Belmont Report’s principle of justice asks that the benefits of research not flow only to those who can reach it. A trial that leaves patients out cannot tell us how they will fare. Until trials include them, their absence should at least be estimated and reported.

        Connections

        Further reading

        References

        BBA Rev Cancer, 2022A novel approach to reducing disparities in health outcomes by enhancing interpretation of cancer clinical trials for underrepresented patient groupsKhozin S, Stein RM. Lancet, 2005External validity of randomised controlled trials: “to whom do the results of this trial apply?”Rothwell PM. JAMA, 2007Eligibility criteria of randomized controlled trials published in high-impact general medical journals: a systematic sampling reviewVan Spall HGC, Toren A, Kiss A, Fowler RA. JAMA, 2004Participation in cancer clinical trials: race-, sex-, and age-based disparitiesMurthy VH, Krumholz HM, Gross CP. JAMA Oncol, 2019Disparity of race reporting and representation in clinical trials leading to cancer drug approvals from 2008 to 2018Loree JM, et al. Am J Epidemiol, 2010Generalizing evidence from randomized clinical trials to target populations: the ACTG 320 trialCole SR, Stuart EA. Econometrica, 1979Sample selection bias as a specification errorHeckman JJ. Clin Cancer Res, 2021Impact of broadening trial eligibility criteria for patients with advanced non-small cell lung cancer: real-world analysis of select ASCO-Friends recommendationsHarvey RD, et al., including Khozin S. Oncologist, 2019Real-world outcomes of patients with metastatic non-small cell lung cancer treated with programmed cell death protein 1 inhibitors in the year following U.S. regulatory approvalKhozin S, et al. Clin Pharmacol Ther, 2019Decentralized trials in the age of real-world evidence and inclusivity in clinical investigationsKhozin S, Coravos A. Nat Commun, 2022The design and evaluation of hybrid controlled trials that leverage external data and randomizationVentz S, Khozin S, et al. US DHEW, 1979The Belmont Report: ethical principles and guidelines for the protection of human subjects of researchNational Commission for the Protection of Human Subjects.

        The Patient-Centered Estimand

        From relative hazard to absolute benefit.

        Cancer drugs are approved on hazard ratios: how much faster events happen on one arm than the other. A patient asks something else: if I take this, what are the chances it helps me, and what are the chances it hurts me? With colleagues at FDA and MIT, Dr. Khozin re-expressed thirty lung cancer trials that supported FDA decisions from 2003 to 2017 as the number of patients treated for one to benefit and for one to be harmed. This theme shows why the translation changes what a result means, and gives clinicians a tool to make it for any trial.

        1. 01

          The hazard ratio is not the patient’s number

          It is relative and averaged over time. The same ratio can mean three patients treated per benefit or thirty.

        2. 02

          Benefit has a clock and a ruler

          The number changes with when you look and with what you measure: response, progression or survival.

        3. 03

          The translation is arithmetic

          Enter a hazard ratio, the control arm’s median and a time point. The tool below returns benefit and harm per patient.

        Part I

        Forty-two seconds

        When the results of RASolute 302 were first shown in May 2026, the audience applauded for 42 seconds. In patients with previously treated metastatic pancreatic cancer, the RAS inhibitor daraxonrasib had doubled median overall survival over chemotherapy, from 6.6 to 13.2 months, with a hazard ratio of 0.40.

        A hazard ratio of 0.40 is usually read as “a 60% reduction in the risk of death.” The statement is accurate, but it specifies neither an absolute effect nor a time frame. The trial also reported the answer a patient can use: at 12 months, 53.3% of patients on daraxonrasib were alive against 18.7% on chemotherapy. Treat three patients, and one more of them is alive a year later.

        0.40Hazard ratio for death
        +35More patients alive at 12 months, per 100 treated
        3Patients treated for one more to be alive at a year

        RASolute 302, RAS G12 population · as summarized in Khozin, 42 Seconds, 2026, and Gierulska et al., Cancers 2026

        Most trials are not like this one. Their effects are smaller, which is where the translation matters most and is done least.

        Part II

        What a hazard ratio is, and is not

        A hazard is the rate at which events happen among patients who have not yet had one. The hazard ratio compares those rates between the experimental arm and the control arm, and the Cox model averages the comparison over the whole follow-up. The control is whatever the trial chose as the comparator: in oncology usually the standard of care, sometimes placebo added to it, rarely no treatment. Every number on this page is relative to that comparator. It is an efficient statistic for detecting a difference. It is a poor one for describing it.

        1. 01
          It is relative

          A 40% reduction means very different things when 80% of patients on the control arm will die within a year and when 8% will. Risk communication research is consistent: patients and physicians overestimate benefit when it is framed in relative terms.

        2. 02
          It averages over time

          When hazards are not proportional, as when curves cross or separate late, a single ratio describes no moment of the trial. Its clinical meaning becomes difficult, if not impossible, to interpret.

        3. 03
          It conditions on survival

          Later in follow-up, the ratio compares the patients still at risk in each arm, who are no longer comparable groups. Even in a perfect randomized trial, the period-specific ratio has a built-in selection bias.

        Gigerenzer et al., Psychol Sci Public Interest 2007 · Uno et al., J Clin Oncol 2014 · Hernán, Epidemiology 2010

        “We recommend using frequency statements instead of single-event probabilities, absolute risks instead of relative risks, mortality rates instead of survival rates, and natural frequencies instead of conditional probabilities.”

        Gigerenzer, Gaissmaier, Kurz-Milcke, Schwartz & Woloshin · Helping Doctors and Patients Make Sense of Health Statistics, 2007

        The 2019 ICH E9(R1) addendum formalizes this question. It asks every trial to define its estimand: the treatment, the population, the outcome, how events such as switching therapy are handled, and the population-level summary that will describe the effect. The hazard ratio is one possible summary. It is not the only one, and it is rarely the one a patient would choose.

        ICH E9(R1), Estimands and Sensitivity Analysis in Clinical Trials, 2019

        Part III

        One in how many?

        In 1988, Andreas Laupacis, David Sackett and Robin Roberts proposed the number needed to treat: the inverse of the absolute difference in risk between arms. Douglas Altman later split it in two, the number needed to treat for one patient to benefit (NNB) and for one to be harmed (NNH), and in 1999, with Per Kragh Andersen, showed how to compute it for time-to-event outcomes such as survival.

        NNB(t) = 1 ⁄ ( Sc(t)HR − Sc(t) )

        Sc(t) is the share of control patients still event-free at time t. Under proportional hazards, the treated arm’s share is that number raised to the hazard ratio. At the control arm’s median, Sc = 0.5 and the formula becomes 1 ⁄ (0.5HR − 0.5).

        Laupacis, Sackett & Roberts, N Engl J Med 1988 · Altman, BMJ 1998 · Altman & Andersen, BMJ 1999 · Raju, Khozin, et al., JCO Clin Cancer Inform 2020

        The formula makes one thing plain that the hazard ratio hides: the same ratio produces a different number for every prognosis. Too little risk and there is little to prevent; too much and events happen before the drug can matter.

        Figure 1One hazard ratio, five prognoses. Benefit at 12 months.

        Each row is a patient group whose median survival on the control therapy differs. Bars show additional patients alive at 12 months per 100 treated; the number is how many must be treated for one of them. With a hazard ratio of 0.70, it ranges from about 8 to 27.

        Illustrative: constant hazards in each group, so survival at 12 months is 0.512/median

        Part IV

        Benefit has a clock

        Because the number depends on Sc(t), it depends on t. Raju, Khozin and colleagues plotted it continuously and found it can change sign. In IPASS, gefitinib against chemotherapy in East Asian patients with lung adenocarcinoma, the overall hazard ratio for progression was 0.74. Underneath, patients with EGFR mutations did far better on gefitinib (hazard ratio 0.48) and patients without them did far worse (2.85). The two groups’ curves add up to a drug that harms early and helps later, and the number crosses from harm to benefit at the moment the Kaplan–Meier curves cross.

        Mok et al., N Engl J Med 2009 · Raju, Khozin, et al., JCO Clin Cancer Inform 2020, Figure 3

        Figure 2A mixed population. Extra patients free of progression, per 100 treated, over time.
        Worst point: per 100 treated, at
        Months until harm turns into benefit
        At 12 months:

        About 60% of IPASS patients with a known mutation status were EGFR-positive. Move the slider to see how the population mix decides the overall answer: the fewer patients with the mutation, the deeper the early harm, the later it turns into benefit, and the smaller the benefit at a year.

        Illustrative model using the published subgroup hazard ratios and a 6-month control median for both groups; the observed IPASS curves crossed near 6 months

        A similar crossing appears in immunotherapy. In CheckMate 057, patients whose tumors expressed PD-L1 at 10% or more began benefiting from nivolumab early; the rest did not, and the overall survival curves crossed before they separated. A single hazard ratio averages these patterns into one number that corresponds to no specific subgroup or time point. The same arithmetic also connects to who is in the trial: change the mix of patients and the answer changes, the problem examined in Theme 04.

        Borghaei et al., N Engl J Med 2015 · Raju, Khozin, et al., 2020, Figure 3B

        Part V

        Which ruler?

        A trial reporting response rate, progression-free survival and overall survival yields three numbers needed to benefit, and they need not agree. Across the thirty trials, the endpoint behind each NNB tracks its size as closely as the drug does.

        Figure 3Thirty lung cancer trials. Patients treated for one to benefit, by the endpoint it was computed from.

        Select a trial to see its details.

        Response-based NNBs average 3.6, progression-based 6.1 and survival-based 14.7. Part of the difference is the drugs: most response-based results come from single-arm trials of targeted therapies in selected patients. Part is the ruler. Tumor shrinkage happens to more patients than survival gains do, and single-arm trials compare response against an assumed rather than a randomized control.

        Data: Table 1, Raju, Khozin, Gurumurthi, Domike & Woodcock, JCO Clin Cancer Inform 2020 · point estimates; logarithmic scale

        The paper is explicit that NNBs from different endpoints are not comparable, and that the purpose of the number has to guide which one is used. A response-based NNB answers “how many patients must be treated for one more tumor to shrink?” That is a question about the instrument examined in Theme 03, not about how long or how well a patient lives.

        Part VI

        The ledger: benefit and harm

        The same arithmetic applies to harm. The number needed to harm is the inverse of the difference in adverse-event rates between arms. Put next to the number needed to benefit, it gives both sides of the decision on one scale: patients, not ratios.

        Across the thirty trials, the numbers lined up with FDA’s decisions. On average, about 4 patients were treated for one to benefit with approved targeted therapies in molecularly selected populations, 11 with approved therapies in unselected populations, and 23 with therapies that were withdrawn or not approved (Theme 02 draws the same result). For approved targeted therapies, the number needed to harm was usually negative in the paper’s notation: fewer serious adverse events than the control arm.

        Cetuximab · FLEX21 treated per patient alive longer9 treated per extra serious adverse eventNot approved
        Bevacizumab · E459914 treated per patient alive longer9 treated per extra serious adverse eventApproved
        Nivolumab · CheckMate 05710 treated per patient alive longer2 treated per serious adverse event avoidedApproved

        Rounded from Table 1, Raju, Khozin, et al., 2020; harm is nonfatal serious adverse events

        The ledger does not make the decision. Cetuximab and bevacizumab had similar counts of harm and were judged differently: a serious adverse event is not a death, and the counts do not weigh severity, duration or what a patient values. What the ledger does is put the trade-off in a form a patient can weigh.

        Part VII

        The frontier

        The paper also asked how far lung cancer treatment had come. For each year, it took the best estimated median survival available to any type of patient and compared it with an estimated median of about 6 months for a patient receiving no treatment. This is a modeled reference, not a trial control: it measures cumulative progress against no therapy, where a trial measures one drug against its comparator. The best median rose from 9 months in 2003 to 39.7 months in 2017, for patients with EGFR mutations. Translated at that 6-month median, the number of patients treated for one to benefit fell from 7.7 to 2.5.

        Figure 4From median survival to patients treated per benefit, against no treatment (modeled median 6 months)
        Years
        Implied hazard ratio against no treatment
        More patients alive at 6 months, per 100
        Treated for one to benefit

        The floor is 2. At the reference median, half of the reference patients are still alive, so no therapy can add more than the other half: at most one extra survivor for every two treated. By 2017 the frontier was near that floor. Measured later in the disease, the same treatments show more room, and the paper notes the frontier shifts lower when calculated at a lower survival percentile.

        Raju, Khozin, et al., 2020, Figure 5 · hazard ratio estimated as 6 ÷ median, which assumes constant hazards

        Part VIII · Tool

        Translate the trial

        Enter what nearly every trial report contains: the hazard ratio, the control arm’s median and the adverse-event rates in each arm. Choose the time point your patient cares about. The tool returns benefit and harm per patient and checks its own estimate against any landmark rate the trial reported.

        Start from

        Benefit
        Harm

        At

        Difference, per 100 treated
        Median on treatment, if hazards stay constant

        One in how many

        One hundred patients, treated

        Patients treated for one to benefit at other time points
        Time pointControlTreatmentPer 100Treated per benefit

          Everything is calculated in your browser. Nothing you enter is sent anywhere; the link you copy carries the numbers in its address. Survival is modeled from the median, assuming a constant hazard in the control arm and a constant hazard ratio.

          Part IX

          What the translation assumes

          Constant hazards

          Turning a median into a curve assumes the control arm’s event rate is steady, and applying one hazard ratio assumes the effect is too. Immunotherapy curves with a long tail break both. When a trial reports landmark rates, use them.

          Uncertainty

          When the confidence interval of the absolute difference includes zero, the interval for the number needed to treat runs from benefit through infinity to harm. Altman suggested reporting it that way rather than as a single figure.

          Counts, not weights

          One more patient alive and one more serious adverse event are counted the same. They are not the same to the patient, and the tool leaves that weighing where it belongs.

          Checked against the trials themselves, the model is close. For CheckMate 057 it estimates 9 patients treated per additional survivor at 12 months, against 8 from the reported rates; for RASolute 302, 3 against 3; for ALEX, 4 against 5. Where hazards are clearly not proportional, statisticians recommend a model-free summary such as the difference in restricted mean survival time: the months of life gained, on average, within a fixed window.

          Altman, BMJ 1998 · Altman & Andersen, BMJ 1999 · Uno et al., J Clin Oncol 2014

          Part X

          The patient’s estimand

          The paper’s conclusion is modest: numbers needed to benefit and to harm complement the hazard ratio, and reporting them could support shared decisions. Because the estimand is chosen at the design stage, it can be chosen to answer the question patients ask.

          1. 01
            Report absolute effects at set times

            Survival in each arm at prespecified landmarks, with confidence intervals, alongside every hazard ratio.

          2. 02
            Put both sides on one scale

            Patients treated per benefit and per harm, by subgroup, in labels and in the clinic.

          3. 03
            Choose summaries that survive the data

            When hazards are not proportional, prespecify a measure such as restricted mean survival time instead of a ratio that describes no one.

          One in how many?

          It is the first question most patients ask and the last one most trial reports answer. A hazard ratio of 0.40 can also be stated in terms a patient can use: treat three people, and one more of them is alive at one year.

          Connections

          Further reading

          References

          JCO Clin Cancer Inform, 2020Patient-centered approach to benefit-risk characterization using number needed to benefit and number needed to harm: advanced non-small-cell lung cancerRaju GK, Khozin S, Gurumurthi K, Domike R, Woodcock J. N Engl J Med, 1988An assessment of clinically useful measures of the consequences of treatmentLaupacis A, Sackett DL, Roberts RS. BMJ, 1998Confidence intervals for the number needed to treatAltman DG. BMJ, 1999Calculating the number needed to treat for trials where the outcome is time to an eventAltman DG, Andersen PK. Epidemiology, 2010The hazards of hazard ratiosHernán MA. J Clin Oncol, 2014Moving beyond the hazard ratio in quantifying the between-group difference in survival analysisUno H, Claggett B, Tian L, et al. Psychol Sci Public Interest, 2007Helping doctors and patients make sense of health statisticsGigerenzer G, Gaissmaier W, Kurz-Milcke E, Schwartz LM, Woloshin S. ICH, 2019E9(R1) addendum on estimands and sensitivity analysis in clinical trialsInternational Council for Harmonisation. N Engl J Med, 2009Gefitinib or carboplatin-paclitaxel in pulmonary adenocarcinomaMok TS, et al. (IPASS) N Engl J Med, 2015Nivolumab versus docetaxel in advanced nonsquamous non-small-cell lung cancerBorghaei H, et al. (CheckMate 057) N Engl J Med, 2015Nivolumab versus docetaxel in advanced squamous-cell non-small-cell lung cancerBrahmer J, et al. (CheckMate 017) N Engl J Med, 2016Pembrolizumab versus chemotherapy for PD-L1-positive non-small-cell lung cancerReck M, et al. (KEYNOTE-024) N Engl J Med, 2017Alectinib versus crizotinib in untreated ALK-positive non-small-cell lung cancerPeters S, et al. (ALEX) Lancet, 2009Cetuximab plus chemotherapy in patients with advanced non-small-cell lung cancer (FLEX): an open-label randomised phase III trialPirker R, et al. Cancers, 2026Efficacy and safety of daraxonrasib (RMC-6236) in previously treated metastatic RAS-mutated pancreatic ductal adenocarcinoma: a systematic reviewGierulska N, et al. Drug Discov Ther, 2026Daraxonrasib: the first-in-class RAS(ON) multi-selective inhibitor approved for treatment of metastatic pancreatic adenocarcinomaSun Y, Pang C, Gao J.