Data engineering · September 2026
An ECG data chain, from public downloads to one traceable database
Twelve public ECG distributions arrive in formats their publishers never agreed on. I built the chain that takes them as shipped and writes one Parquet database in which any tracing leads back to its source file in a single join. It runs on public corpora, and no model is trained on it.
12 distributions · 110,876 records · 87,609 distinct tracings
Numbers read from the results files at commit e066fde
How the chain runs
Each step writes a results file into the repository, and every number on this page is read from one of them.
What the checks found
Each of these defects was found by a step of the chain and is counted in a results file.
One unit, three spellings
The headers declare the same unit in three ways. Of the 12 distributions, 7 write mV, 3 write mv, and 2 declare no unit at all, which leaves the reader to supply one. The record scan keeps the spelling each header declared, so the difference shows before a single signal is read.
Flat and saturated leads at Ningbo
Each record is judged on the first ten seconds of its twelve standard leads, or on the whole record where it is shorter. At Ningbo 1,583 of 34,905 records fail at least one check. Among them, 1,480 have a flat lead, 20 a lead saturated at its own extreme and 97 non-finite samples, and a record can fail more than one check. Across all distributions 2,320 records fail, so Ningbo accounts for more of them than the other eleven together. The records stay in the database with their verdict in the quality table, and a reader filters on it.
The same tracing twice in CPSC-2018
Of the 6,877 CPSC-2018 records, 505 fall into 250 groups of identical tracings, so 255 records repeat a tracing the distribution already holds. I opened 30 repeat groups from CPSC-2018 and Georgia, drawn with a fixed seed, and compared them: every one is identical sample for sample, header comments included. A split drawn by patient does not guarantee that a recording and its copy land on the same side, and CPSC-2018 publishes no patient key anyway, so each record has a signal group to split on.
Two PTB-XL that do not agree
PhysioNet publishes PTB-XL with 21,799 records. The PhysioNet Challenge 2021 bundle includes it again with 21,837, under other record ids. Each original is found in the bundle by its name, and the 38 bundle records left over repeat a tracing the bundle already holds. The key of the database is the distribution and not the corpus name, so the two distributions stay apart and the link between them is recorded.
One file off its published digest
Beside each of the 222,301 files it points at, the database stores the digest its publisher declared. Of the 224,882 files re-hashed against their publisher's manifest, 224,881 match the published digest. The one that differs is a Ningbo signal file. It parses to twelve leads and passes every quality check, and only the manifest comparison detects it. The results file keeps the published digest and the observed one side by side.
How it is tested and rebuilt
The repository has 303 tests. CI runs 277 of them on every push to main and every pull request, after ruff, the format check and mypy. The other 26 need the corpora on disk and run on the machine where the downloads are.
The passes that read signals run one at a time, each under a memory cap and a lock. The duplicate screen — profiled phase by phase on one pair of packagings — holds 272 MB resident at its peak. The cap is set on what a pass holds, measured this way, and not on the cgroup counter, which also counts the page cache of the files being read.
Neither the corpora nor the Parquet sit in the repository. The corpora are downloaded from PhysioNet, the Parquet is rebuilt from them by the repository's scripts, and before it is read, each published code table is checked against the digest recorded when it was downloaded. From the rebuilt label table, all 240 cells of the PhysioNet Challenge 2021 scored-diagnosis table are recomputed and agree with the counts the organisers published.
Replay one query
Once the delivery is rebuilt, this DuckDB query traces a record to the file it came from. It is the repository's trace query, written against the Parquet files directly, and the repository's test counts its joins and fails if the query gains one.
SELECT r.record_id, r.source_id, f.relative_path, f.sha256_declared
FROM read_parquet('results/delivery/record.parquet') AS r
JOIN read_parquet('results/delivery/source_file.parquet') AS f
ON f.file_id = r.signal_file_id
WHERE r.record_id = 'challenge-2021/chapman_shaoxing:JS02202';What it returns, as recorded in results/database_report.json:
record_id challenge-2021/chapman_shaoxing:JS02202 source_id challenge-2021/chapman_shaoxing relative_path g3/JS02202.mat sha256_declared cd1e882875df52351103b3e9611133a5efad0fedfd93437b82ca53900a42b580
The last column is the digest PhysioNet released, read from the checksum file shipped with the download, and not one computed here.
The same chain in a hospital
This chain runs on public corpora, and it has never run in a hospital. Four things separate the two. It reads files published once, whereas a hospital sends tracings continuously from its carts and its ECG management system, often as vendor exports this repository has never parsed; that feed is built with the hospital's IT team and the vendor. Its delivery is open files on open data with no access control, whereas in a hospital who reads what is decided with the data protection officer, in the impact assessment a health data warehouse requires. In France, health data hosted by a third party must sit with an HDS-certified host, so the database would move to such a provider or to the hospital's own infrastructure rather than stay on the machine it runs on today. And nothing here watches the passes once they are launched; supervision and the on-call that goes with it belong to the hospital's operations team. The data work shown above would carry over, applied to a hospital's exports instead of PhysioNet's.
Where to check the numbers
The repository holds the code, the tests and the results files every number above is read from, at the commit cited at the top of this page.