Study · September 2026

Does a 90% guarantee hold for the patients who are ill?

A conformal classifier returns a set of possible diagnoses instead of one label, with a guarantee on how often that set contains the right answer. The guarantee is an average over everyone. I measured what it delivers to the patients who are having a heart attack, and what becomes of it at two hospitals it was not calibrated on.

Code and numbers Report Data

3 public datasets · 45,923 tracings · 200 calibration draws
State: published, v1.0.0 · Every figure is drawn from a committed results file

In plain terms

An algorithm that reads electrocardiograms usually returns one answer. A conformal one returns the set of answers it cannot rule out, calibrated so that the set contains the right diagnosis in 90 cases out of 100, which is the whole of what the method guarantees. The set most often holds one label, and that settles the case. Where it holds two labels, or none at all, a specialist reads the tracing instead.

Three quarters of this population are not having a heart attack, and covering them well is enough to reach the 90%. Inside the infarction cases alone, on held-out patients at that same site, the same calibration covers only 73.2% of them — so more than one in four patients with infarction gets a set with no infarction in it. That threshold then goes, unchanged, to a hospital in Shandong and to an angiography cohort in Chongqing.

At the site the threshold was fitted on, PTB-XL in Germany, over 2,198 tracings:

73.2%

of the infarction cases

89.9%

of all cases, which is what the guarantee constrains

Fitting a threshold within each label brings coverage to 90.0% at this site.

The same threshold, carried unchanged to two Chinese hospitals:

93.6%

Shandong, above the target

25,770 tracings

72.5%

Chongqing, 17.5 points below the target

17,955 tracings

At Chongqing the same class-conditional correction reaches only 84.0%, short of the 90% asked for.

What was measured

The classifier reaches an AUROC of 0.932 on the infarction label against a published anchor of 0.930. The two are different quantities — the anchor averages five diagnostic classes rather than reporting infarction alone — so their agreement is a plausibility check on an ordinary baseline and not a reproduction. The model was frozen after that, so the coverage differences come only from where the threshold is placed.

Score distributions for infarction and non-infarction cases, with the pooled and per-label decision boundaries drawn across them. — Figure 1. Threshold placement on the model score for MI, PTB-XL fold 10. Grey: 1,648 non-MI cases. Coral: 550 MI cases. Dashed lines mark the fitted decision boundaries; the labelled bands give the resulting decision regions.

Stacked bars giving, for each label and each of the three schemes, the share of cases labelled correctly, deferred, or labelled wrongly. — Figure 2. Outcome decomposition by label at the source site, averaged over 200 patient-level calibration draws. Fold 10 holds 550 MI and 1,648 non-MI cases, of which each draw scores about half, the figure's own note giving the per-draw means. The two conformal schemes run at a 90% coverage target and the single threshold at 90% sensitivity. That places the single threshold and class-conditional calibration at a comparable MI miss rate by construction; pooled calibration is matched to neither and lands at 27%. Each share is of all cases carrying that label, and a deferred case counts as neither correct nor wrong.

A grid of coverage bars, one row per site and one column per calibration scheme, with the requested level drawn as a dashed line across each panel. — Figure 3. Coverage by site (rows: PTB-XL 2,198 tracings, Shandong 25,770, Chongqing 17,955) and calibration scheme (columns: no correction, class-conditional, label-shift weighted). Within each panel the horizontal axis is the confidence asked for. Grey: all cases. Blue: non-MI cases. Red: MI cases. Dashed line: requested coverage. Bars are means over 200 calibration draws of PTB-XL, and the whiskers are one standard deviation across those draws, which is the spread of the calibration and not the sampling error of the site.

Limits

The study covers one source dataset, one diagnosis and two target sites — and the two moved in opposite directions, so the direction and the size of the effect at a third site stay unknown. At Chongqing the positive label comes from a discharge diagnosis of acute infarction rather than from the tracing, and the hardware, the population and the recording years all differ at once, so the study measures the resulting gap without being able to attribute it.

Coverage is an average over the cases of a label and says nothing about one tracing. A clinician reading a positive result wants the chance that a case flagged as infarction really is one, which is a different quantity. The work was never tested in clinical use, and this is not a medical device.

Where to check the numbers

The repository holds the report and its references, and tests that fail when a published number no longer matches its source.

github.com/rabbou/ecg-conformal-shift Contact me