English
Automated ECG interpretation has traditionally been divided into narrow models: one model for rhythm classification, another for left ventricular dysfunction, another for atrial fibrillation risk and another for report generation. DeepECG-Tok replaces this fragmented approach with a single instruction-following system that can answer different clinical questions directly from the same 12-lead ECG.
The key idea is to treat an ECG as a language the model can learn. DeepECG-Tok converts continuous cardiac waveforms into compact, discrete tokens, aligns those tokens with clinical text and uses them to condition a medical large language model. The resulting model can generate free-text interpretations, produce structured reports, identify diagnostic findings and estimate clinically important endpoints.
DeepECG-Tok uses a three-stage training framework:
The tokenizer was trained using 1.91 million ECGs from the Montreal Heart Institute, MIMIC-IV and CODE-15. Its frozen representations achieved a macro-averaged AUROC of 0.96 across 77 diagnostic conditions, outperforming the supervised and self-supervised DeepECG baselines. The representation transferred without retraining to the CLSA and Harvard-Emory cohorts, where macro-averaged AUROCs remained 0.88 and 0.90.
The instruction-tuned model was trained on more than 7.27 million question-answer pairs derived from over one million ECGs. The prompts covered free-text interpretation, structured JSON reporting, normal-versus-abnormal classification, rhythm and conduction disorders, ischemia and infarction, chamber enlargement, interval measurements and clinical endpoints.
Unlike conventional ECG classifiers, DeepECG-Tok can respond to questions such as:
This unified design removes the need to train and maintain a separate prediction head for every task.
The supplementary evaluation shows how the same model can move between focused diagnostic questions, structured outputs and open-ended interpretation. The examples below are adapted and shortened from Supplementary Table 20. They include both strong responses and an error to illustrate the model's present capabilities and limitations.
These examples are illustrative rather than a performance sample. They show that DeepECG-Tok can reproduce complex multi-finding interpretations and constrained JSON-like outputs, while also making clinically meaningful errors that require expert oversight.
DeepECG-Tok was evaluated without retraining on four independent external cohorts comprising 20,755 ECGs from 16,767 patients. These cohorts tested both conventional interpretation and linked clinical endpoints across different institutions, populations and acquisition systems.
Diagnostic transfer remained strong, with frozen-tokenizer macro-averaged AUROCs of 0.88 in CLSA and 0.90 in Harvard-Emory. For left ventricular ejection fraction (LVEF), the model supported both threshold-based detection and continuous estimation:
The continuous LVEF estimates were most accurate in the internal cohort and remained informative in both external datasets, although the higher MAE and lower correlation and agreement metrics show a measurable generalization gap. Structural heart disease prediction also transferred from the Montreal Heart Institute to EchoNext with comparable evaluation scores.
The model's clinical-endpoint capabilities extend well beyond ventricular function. From the same ECG representation and without separate task-specific prediction heads, DeepECG-Tok assessed future atrial fibrillation risk, structural heart disease, acute coronary occlusion and the likely culprit coronary artery.
These endpoints are particularly important because they span different clinical time horizons: immediate triage for acute coronary occlusion, detection of existing structural disease and longer-term prediction of incident atrial fibrillation. AF risk, acute coronary occlusion and culprit-artery localization were developed and evaluated using linked MHI data and therefore still require prospective, multicenter validation.
Across the full instruction-following evaluation, the ontology-grounded composite score was 0.71 internally and 0.50-0.53 in the external interpretation cohorts. Endpoint-style questions transferred more consistently than open-ended or rigidly structured report generation, highlighting both the model's generalization and the remaining effect of institutional reporting differences.
Free-text ECG reports cannot be evaluated reliably by word overlap alone. Clinically equivalent statements such as "atrial fibrillation with rapid ventricular response" and "AFib with RVR" may receive poor BLEU, ROUGE or METEOR scores despite expressing the same diagnosis. The study therefore introduced an open-source, ontology-grounded LLM-as-a-judge framework designed as a reusable evaluation standard for ECG-language models.
The framework routes each model response through one of two complementary pathways:
For each response, the judge separates the clinical content into findings that were identified, missed or hallucinated. The score is calculated as identified findings divided by the total number of identified, missed and hallucinated findings, then aggregated across 19 evaluation categories. This makes the score sensitive to both omissions and unsupported additions while avoiding the false penalties produced by literal word matching.
Two board-certified cardiologists independently re-scored a random sample of 500 question-answer pairs using the same finding-level rubric. Agreement between the automated framework and the cardiologists reached a mean Cohen's kappa of 0.82, supporting its use for scalable evaluation. The framework is publicly available at github.com/HeartWise-AI/ECG_LLM_Judge, and the planned PhysioNet release will provide 25,000 ECGs with standardized images, raw signals and clinically validated question-answer pairs for reproducible benchmarking.
Because the evaluator is independent of DeepECG-Tok's architecture, other groups can reuse it to compare ECG-language models, test new prompting strategies and measure cross-institutional generalization using clinically meaningful criteria rather than lexical similarity alone.
A separate blinded reader study compared DeepECG-Tok reports with reference clinician reports. In single-report review, the model report was rated better in 32% of comparisons, tied in 35% and rated lower in 33%. In forced-choice review, readers preferred the model in 31% of cases, judged the reports equivalent in 42% and preferred the reference in 28%. Among decided cases, the model was preferred 53% of the time, with no significant difference between model and reference reports.
DeepECG-Tok shows that a discrete ECG vocabulary can support more than classification. One shared representation can connect cardiac waveforms to natural-language interpretation, structured reasoning and endpoint prediction, providing a foundation for ECG systems that clinicians can query directly.
The study remains retrospective. Prospective, multicenter validation is needed before clinical deployment, particularly for endpoints derived from the Montreal Heart Institute cohort. The model also requires an upstream signal-quality gate and a calibrated abstention mechanism: when an ECG is absent or corrupted, structured prompts can still elicit a plausible but unsupported answer. These safeguards are important next steps toward safe clinical use.
The anonymized ECG evaluation dataset supporting the LLM-as-a-judge analysis is planned for release on PhysioNet following peer-reviewed publication, subject to credentialing and a data-use agreement.
This work was supported by the Fonds de recherche du Québec through grant FRQ 5232 and by FRQS career awards.