AUROC vs. PPV: Why the Headline Number Doesn't Survive Deployment

Almost every AI-ECG paper opens with an AUROC in the 0.85–0.95 range. It's the number that gets the model published, funded, and onto a conference slide. It tells you almost nothing about whether the model will help a patient in your clinic.

I've spent years validating ECG algorithms across US and European hospital sites, through both FDA and CE-mark pathways, and it's the same failure mode I keep seeing: an excellent AUROC goes into a real workflow, most of what it flags turns out to be nothing, and people quietly stop acting on it. The model didn't get worse. The disease is just rarer where it runs than the headline number ever admits. Here's the arithmetic behind it.

The Same Model, Two Settings, Two Verdicts

AUROC measures one thing: given a sick patient and a healthy one, how often does the model rank the sick one higher. It's a property of the model on a held-out set. It doesn't move when disease prevalence changes.

Positive predictive value does. PPV is the question the clinician actually asks — the model flagged this patient, what are the odds they really have the disease? — and the answer depends almost entirely on how common the disease is where you deploy.

Take AI-ECG for hyperkalemia, dangerously high potassium. In a Mayo Clinic validation — a model from the lineage later commercialized by Anumana, where I used to work — it reached an AUROC of 0.88 with 80% sensitivity and 80% specificity. In the emergency department, where about 1% of patients had severe hyperkalemia, its PPV was 3%. The same model, on ICU patients at roughly 3% prevalence, reached 14% (Harmon et al., CJASN 2024). Same weights, same 0.88 AUROC — only the base rate moved, and the model went from about 32 false alarms per true case down to 6.

That's not a tuning problem. It's arithmetic.

High Specificity Doesn't Save You

The usual response is to chase specificity. Surely a model that's right 98% of the time on healthy patients can't be drowning in false positives?

Cleveland Clinic ran an AI-ECG screen for hypertrophic cardiomyopathy across 103,492 real-world ECGs, with specificity above 98%. It flagged 1,265 patients — and 691 of them, 54.6%, did not have HCM (Desai et al., JACC: Clinical Electrophysiology 2025). What it caught instead was often the other causes of a thick-walled heart: hypertensive hypertrophy, aortic stenosis.

That 54.6% is not the false-positive rate — at >98% specificity, the false-positive rate is under 2%. It's the share of flagged ECGs that were wrong. When a disease is rare, a tiny false-positive rate multiplied across a huge population of healthy people still produces more false alarms than true cases. The specificity was excellent and the flagged group was still mostly noise.

The Pattern Holds Across the Literature

This isn't cherry-picked. A 2025 review in EMBO Molecular Medicine lined up low-ejection-fraction screening (AI-ECG looking for a weakened heart) across settings. One group's own model returned a PPV of about 5–7% in an outpatient clinic and about 34% in hospitalized patients — the Liu and Tsai studies respectively, both catalogued in a 2025 EMBO Molecular Medicine review (Lin et al.). Across the wider literature the spread runs higher still, up to roughly 83% in risk-enriched screening populations, though those top numbers pool different models and cohorts, so read them as the shape of the effect, not one controlled experiment. Discrimination barely moved throughout; AUROCs stayed in the 0.92–1.0 band. PPV swung more than tenfold.

Three conditions, one story:

Low PPV Isn't Automatically Useless

Here's the counterargument a good cardiologist will make, and it's right: a low PPV doesn't mean a model is worthless. It depends on what a false positive costs and what a missed case costs. Hyperkalemia is confirmed by a single repeat blood draw and can kill if missed, so a 3% PPV screen that surfaces cases early can still be net-positive — the same logic behind serial troponin, cheap to repeat and dangerous to miss. Desai's 45% PPV against a low background prevalence is, on its own terms, a large risk-enrichment: a screening win, not a scandal.

So the argument isn't "low PPV, therefore bad." It's that PPV is the number that decides whether a deployment helps or drowns, it's invisible in the AUROC, and it belongs on the table before you turn the model on, not discovered afterward.

What This Means If You're Buying, Building, or Reviewing

I've written the validation reports regulators actually read. What I'd insist on:

• The right number: PPV and NPV at your site's real prevalence and at the operating point you'll actually run. PPV moves with both, and a vendor can shift the threshold independently of your base rate.
• Not the vendor's: a PPV from a 30%-prevalence tertiary cohort isn't transferable as-is. Recompute it at your prevalence from the reported sensitivity and specificity, exactly what the curve above does.
• A number people feel: false alarms per true catch. A 3% PPV lands differently once you say "about 32 unnecessary workups for every real case."
• The cost of being wrong: each false positive is a repeat test, a referral, an anxious patient, and a clinician one step closer to ignoring the next alert. Weighed against the cost of a miss, that is the actual decision.

AI-ECG works. But "does it work?" has no answer until you name the population, the threshold, and the cost of being wrong. Ask for the PPV at your prevalence before you ask about the AUROC — it's the number your clinic will live with.

Disclosure: I worked at Anumana — which commercializes Mayo Clinic's AI-ECG work, including the lineage behind the hyperkalemia study above — and before that at Idoven. The argument here cuts against products adjacent to my own past work, which is rather the point.

Sources: Harmon DM et al., "Validation of Noninvasive Detection of Hyperkalemia by AI-Enhanced ECG in High Acuity Settings," CJASN 2024;19(8):952–958. · Desai MY et al., "Real-World AI-Based ECG Analysis to Diagnose Hypertrophic Cardiomyopathy," JACC: Clinical Electrophysiology 2025;11(6):1324–1333. · Lin CS et al., "Artificial Intelligence–Enabled Electrocardiography from Scientific Research to Clinical Application," EMBO Molecular Medicine 2025 (review).