Fluent but wrong: The gap at the centre of radiology AI evaluation
By Dr Suneeta Mall, Head of AI Engineering, Harrison.ai
A radiology report can read fluently, use all the right terminology, and closely resemble what a specialist would write – and still be dangerously wrong. It can deny an effusion that is present, or describe a pathology that isn’t there. In a clinical setting, a report that sounds right but isn’t, is worse than no report at all: risking a confident yet wrong, at worst unsafe, outcome.
This is the uncomfortable gap at the centre of how generative radiology AI is evaluated today. The first wave of radiology AI, task-specific models trained to flag a predefined set of findings, is evaluated on a simple question: did it detect the thing or not? But the latest generation of models, foundation models that read a scan and write a full report in natural language, raise a harder problem. Their output is free-form text, a multi-dimensional output that can be clinically wrong in ways yet still sound entirely coherent – easy to miss yet significant errors. Most benchmark scores reported for these models measure how a report is worded, not whether it is clinically right. A model can miss a finding or hallucinate one and still post competitive numbers. When we set out to build and evaluate Harrison.Rad 1.5, we decided that gap was the first thing we had to close, because you can’t trust a model further than the yardstick used to test it, so we rebuilt the yardstick.
What conventional metrics actually reward
The metrics that dominate the radiology-AI literature fall into two families, and both have the same blind spot.
The first family, scores such as BLEU and ROUGE-L, works by counting how much of the reference report’s wording a generated report reuses. That makes them measures of phrasing, not of meaning. A report that is clinically perfect but written in different words scores closer to zero, because it shares almost none of the reference’s vocabulary. And a report that reuses the reference’s phrasing while reversing the finding scores well, because nothing in the calculation knows what the words assert. The ranking these metrics produce can therefore run in the opposite direction to clinical correctness.
The second family goes deeper. BERTScore, CheXbert-cos and RadGraph don’t match words. They ask whether a report is about the same things as the reference, comparing meaning rather than phrasing.
RadGraph is the most sophisticated of the three. Rather than treating the report as one block of text, it converts it into a structured representation of its clinical content: it extracts the clinical entities, the findings and the anatomy they sit in, and links them into triplets of the form this finding, in this location. It then counts how many of those structured facts in the generated report match the reference, and turns that overlap into an F1 score. It is grading clinical concepts and the relationships between them, not the sentences carrying them.
That is a real step up on BERTScore and CheXbert-cos. But RadGraph is only as good as the model doing the extraction, and that model is a BERT variant trained on limited X-ray reports. It knows the vocabulary it was trained on and stumbles on the rest: a synonym it has not seen, a finding worded differently, the reporting conventions of another hospital or another country. When the extractor misreads a report, every score built on top of it inherits the mistake.
All three also share the same clinical blind spots. They stay high when a report flips a finding from present to absent, and they treat a life-threatening finding and a trivial one alike. A report can invent a significant finding, or miss one, and still score well. Composite measures such as RadCliQ are assembled from these same components, so they inherit the same blindness.
A newer wave of metrics tries to close exactly these gaps by handing the judging over to a large language model. GREEN reads both reports and flags the errors that would actually change a clinician’s decision, an invented finding, a missed one, a hedge turned into a certainty, and returns a score along with a plain-language list of what went wrong. RadFact takes a different angle: it breaks each report into single-finding sentences and asks whether each claim in one report is logically supported by the other, giving separate readings for how truthful the generated report is and how complete. Both finally take affirm-versus-deny and clinical significance seriously, which is what the earlier families missed. But they trade one weakness for another. These judges lean on generalpurpose language models, and the way radiology is actually written, its shorthand, and house conventions, is not the kind of text that shows up much on the open internet these models learn from. So the models are working slightly out of their depth, which shows up as a gap in how well they generalise.
The gap shows up on findings that differ by a single word. Change lytic to sclerotic in a lumbar spine report, 24 of 25 words identical, and RadFact still scores it 1.00, perfect on truthfulness and completeness. One finding swapped for another, asserted with identical confidence.
None of these metrics reliably separates a clinically correct report from a clinically wrong one. So we don’t treat them as a measure of clinical quality.
The failure is easy to see once you put the scores next to a clinician’s verdict. Take a spine study where the correct read is cervical spondylosis. A model that answers, tersely but correctly, “Cervical spondylosis” scores zero on BLEU, ROUGE-L and even RadGraph-F1, because it shares almost no wording with the long reference report. Meanwhile, uninformative outputs (a non-answer like “Cannot be determined from this image,” or a vague descriptor like “Pelvic asymmetry”) hold a BERTScore around 0.81–0.82, comparable to fully correct reports. And on the same chest study, a clinically correct report and a clinically wrong one can sit within a hundredth of each other on BERTScore, so the metric cannot tell them apart even though one denies an effusion that is present.
Measuring clinical content, not wording
If surface metrics reward wording, the alternative has to start from clinical content. Our primary evaluation is a Findings-Diagnosis framework built on three ideas.
First, a normal-versus-abnormal call: a negation-aware analysis of the text, backed by clinical entity extraction, determines whether the study is being reported as normal or abnormal – the call that most changes what happens to the patient next, and the one we designed the framework to get right first.
Second, findings comparison: clinical entities are extracted from both the model’s report and the ground truth, stripped of generic descriptors, and matched using fuzzy token matching with ontologybased synonym expansion drawn from a radiology knowledge graph. This is what lets the framework recognise that “cervical spondylosis” and a longer description of degenerative change are the same clinical finding, where a keyword metric sees no overlap.
Third, and this is the part surface metrics have no equivalent for, polarity-contradiction detection. Where a report asserts the opposite of the truth (definitely denying a finding that is present, or asserting one that is absent), it receives zero credit. Reversing the direction of a finding contradicts the reference report outright, so the framework gives it zero credit rather than partial marks. Measured against that reference, a reversed finding isn’t a near-miss, it is an error.
We are candid that this is not infallible. Because the normal/abnormal decision and the finding overlap are computed automatically, the score can still credit a report that misses a subtle finding, or accept a vague-but-incorrect descriptor as a match. And in one respect it shares the very weakness we criticise in conventional metrics: it can penalise a clinically richer answer for its wording.
Asked what complication of an oesophageal stent had occurred (ground truth: the stent has migrated) Harrison.Rad 1, the previous-generation model, answered, “The oesophageal stent appears migrated.” Harrison.Rad 1.5 answered that the stent “has slipped out of its original position and now lies horizontally in the fundus of the stomach.” The second answer carries more clinical information: it not only identifies the migration but localises the displaced stent. Yet under our own Findings-Diagnosis score it is marked wrong, only because it paraphrased “migrated” instead of reproducing the word. The more specific, more useful answer is penalised for its language, the same failure mode we object to in BLEU, arising in a metric we built to fix it. So where automated matching isn’t enough to adjudicate correctness, most notably on free-text report generation, we add a clinical LLM-as-a-judge adjudication of the same findings-versus-diagnosis comparison, and treat that as the more reliable signal. In the stent case, the clinical judge scores the second answer correct. We continue to tabulate the conventional metrics alongside, for continuity with the literature, but we read them as context, not as the verdict.
Where this leaves us, and where it points
Held to this standard, Harrison.Rad 1.5 was the only system we evaluated to meet the passing threshold on a simulation of the FRCR examination, the assessment the Royal College of Radiologists uses to certify practising radiologists, with every general-purpose and medical-domain model tested falling substantially below it. We think that result matters because of how it was measured: on clinical correctness, against a clinician-referenced bar, not on wording.
We’re also clear about the limits. On some open-ended chest-report benchmarks, frontier generalpurpose and chest-specialised models still post higher values on individual metrics, partly genuine headroom we intend to close, and partly the very metric limitations described above.
Even a well-aligned single score, though, can only tell you how much a report is right, not why it is adequate or inadequate – whether it caught the clinically important findings, whether it would be safe to act on, whether a radiologist would accept it. A report that omits a minor finding is scored much like one that hallucinates a dangerous pathology, and those are not the same error.
We are developing a framework we call C6, which scores a draft report along six interpretable clinical pillars: Coverage, Correctness, Comprehensiveness, Care, Coherence and Conformity, rather than collapsing everything into one number. The Care pillar in particular introduces explicit clinical-severity weighting, so that a safety-critical error such as a missed pneumothorax is penalised more heavily than a benign omission or a stylistic quibble. And because different settings need different emphases – emergency triage prioritises catching the acute finding, oncology surveillance prioritises precise characterisation – the weighting is tunable per deployment.
We intend to adopt C6 as the primary evaluation framework for our future models, and to release it as an open tool for the radiology community. We’ll release C6 openly, so the field can hold every model, including ours, to the same standard.
Dr Suneeta Mall, Head of AI Engineering, Harrison.ai
Harrison.Rad 1.5 and Harrison.Rad 1 are research-only foundation models, not medical devices regulatory-cleared for clinical use. Harrison.ai is seeking regulatory clearance for medical devices powered by these models in various markets.