Comparison of prognostic accuracy of score scale and a machine learning model in predicting fatal cardiovascular complications
https://doi.org/10.25881/18110193_2025_4_86
Abstract
Identifying patients at high risk for fatal cardiovascular disease (CVD) complications is a critical task in reducing preventable CVD morbidity and mortality. Various risk assessment algorithms and scores are widely used for this purpose, but their limitations include a limited set of predictors and low accuracy. Machine learning methods offer the potential to address these shortcomings and personalize cardiovascular risk assessment.
Objective: To compare the accuracy of the SCORE scale and machine learning models in predicting fatal cardiovascular complications.
Materials and Methods: A multicenter retrospective study was conducted (1999–2018), including 3,891 treatment cases of 1,064 patients aged 40–69 years in the Russian Federation. Logistic regression, ensemble machine learning (ML) methods, and Multi-Layer Perceptron were used for forecasting. Comparison with SCORE was performed on an independent validation set consisting of 440 records.
Results: The CatBoost ML model demonstrated the best accuracy (AUROC 0.879; sensitivity 0.938; specificity 0.777). During validation, CatBoost demonstrated comparable discrimination to SCORE but outperformed the scale in specificity (0.653 vs. 0.408) and accuracy (0.673 vs. 0.45) when referencing patients to lowand intermediate-risk groups. Key predictors for the model were gender, age, smoking, systolic blood pressure, body mass index, heart rate, and lipid profile.
Conclusion: The machine learning model outperformed the SCORE scale in predicting fatal cardiovascular events. The use of machine learning in predicting cardiovascular risk can improve the effectiveness of CVD prevention and facilitate personalized patient care.
About the Authors
A. D. Ermak
K-SkAI LLC
Russian Federation
Russian Federation
Petrozavodsk
D. V. Gavrilov
K-SkAI LLC
Russian Federation
Russian Federation
Petrozavodsk
T. Yu. Kuznetsova
Petrozavodsk State University
Russian Federation
Russian Federation
DSc, Associate Professor
Petrozavodsk
A. E. Andreichenko
ITMO University
Russian Federation
Russian Federation
PhD
Saint Petersburg
E. A. Makarova
K-SkAI LLC
Russian Federation
Russian Federation
PhD
Petrozavodsk
R. E. Novitskiy
K-SkAI LLC
Russian Federation
Russian Federation
Petrozavodsk
A. V. Gusev
Federal Research Institute for Health Organization and Informatics
Russian Federation
Russian Federation
PhD
Moscow
References
1. Roth GA, Mensah GA, Johnson CO, Addolorato G, et al. Global Burden of Cardiovascular Diseases and Risk Factors, 1990-2019: Update From the GBD 2019 Study. Journal of the American College of Cardiology. Elsevier Inc. 2020; 76: 2982-3021.
2. Damen JAAG, Hooft L, Schuit E, Debray TPA, et al. Prediction models for cardiovascular disease risk in the general population: Systematic review. BMJ (Online). BMJ Publishing Group. 2016; 353.
3. Gavrilov DV, Gusev AV, Nikulina AV, Kuznetsova TY, Drapkina OM. Correctness of cardiovascular risk assessment in daily clinical practice. Profilakticheskaya Meditsina. 2021; 24(4): 69-75.
4. Boytsov SA, Pogosova NV. Cardiovascular prevention 2022. Russian national guidelines: Russian Society of Cardiology, National Society of Preventive Cardiology. Russian Journal of Cardiology. 2023; 28(5).
5. Global Effect of Modifiable Risk Factors on Cardiovascular Disease and Mortality. New England Journal of Medicine [Internet]. 2023; 389(14): 1273-85. Available from: http://www.nejm.org/doi/10.1056/NEJMoa2206916.
6. Conroy RM, Pyörälä K, Fitzgerald AP, Sans S, et al. Estimation of ten-year risk of fatal cardiovascular disease in Europe: The SCORE project. Eur Heart J. 2003; 24(11): 987-1003.
7. Tokgozoglu L, Torp-Pedersen C. Redefining cardiovascular risk prediction: Is the crystal ball clearer now? European Heart Journal. Oxford University Press. 2021; 42: 2468-71.
8. Kapoor S, Narayanan A. Leakage and the reproducibility crisis in machine-learning-based science. Patterns. 2023; 4(9).
9. Com YD, Simonoff JS. An Investigation of Missing Data Methods for Classification Trees Applied to Binary Response Data Yufeng Ding. Journal of Machine Learning Research. 2010; 11.
10. Cao XH, Stojkovic I, Obradovic Z. A robust data scaling algorithm to improve classification accuracies in biomedical data. BMC Bioinformatics. 2016; 17(1).
11. De Amorim LB V, Cavalcanti GDC, Cruz RMO. The choice of scaling technique matters for classification performance [Internet]. Available from: https://github.com/amorimlb/scaling.
12. Weiss GM. Foundations of imbalanced learning. 2012.
13. Akiba T, Sano S, Yanase T, Ohta T, Koyama M. Optuna: A Next-generation Hyperparameter Optimization Framework. 2019 Jul 25. Available from: http://arxiv.org/abs/1907.10902.
14. Sokolova M, Lapalme G. A systematic analysis of performance measures for classification tasks. Inf Process Manag. 2009; 45(4): 427-37.
15. Brodersen KH, Ong CS, Stephan KE, Buhmann JM. The balanced accuracy and its posterior distribution. In: Proceedings – International Conference on Pattern Recognition. 2010. Р.3121-4.
16. Barandela R, Sã Anchez B; JS, Garcã V, Rangel E. Rapid and Brief Communication Strategies for learning in class imbalance problems [Internet]. Pattern Recognition. 2003. Available from: www.elsevier.com/locate/patcog
17. Chicco D, Jurman G. The Matthews correlation coefficient (MCC) should replace the ROC AUC as the standard metric for assessing binary classification. BioData Min. 2023; 16(1).
18. Zoubir AM, Iskandler DR. Bootstrap methods and applications. IEEE Signal Process Mag. 2007; 24(4): 10-9.
19. Lundberg SM, Erion G, Chen H, Degrave A, et al. Explainable AI for Trees: From Local Explanations to Global Understanding [Internet]. Available from: https://github.com/suinleelab/treeexplainer-study.
20. Wester DB. Comparing treatment means: overlapping standard errors, overlapping confidence intervals, and tests of hypothesis. Biom Biostat Int J. 2018; 7(1): 73-85.
21. Fischer BG, Evans AT. SpPin and SnNout Are Not Enough. It’s Time to Fully Embrace Likelihood Ratios and Probabilistic Reasoning to Achieve Diagnostic Excellence. J Gen Intern Med. 2023; 38(9): 2202-4.
22. Baduashvili A, Guyatt G, Evans AT. ROC Anatomy — Getting the Most Out of Your Diagnostic Test. J Gen Intern Med. 2019; 34(9): 1892-8.
23. Thomas G, Kenny LC, Baker PN, Tuytten R. A novel method for interrogating receiver operating characteristic curves for assessing prognostic tests. Diagn Progn Res. 2017; 1(1).
24. Gill SK, Karwath A, Uh HW, Cardoso VR, et al. Artificial intelligence to enhance clinical value across the spectrum of cardiovascular healthcare. European Heart Journal. Oxford University Press. 2023; 44: 713-25.
25. Friedrich S, Groß S, König IR, Engelhardt S, et al. Applications of artificial intelligence/machine learning approaches in cardiovascular medicine: A systematic review with recommendations. European Heart Journal — Digital Health. Oxford University Press. 2021; 2: 424-36.
26. Gusev AV, Gavrilov DV, Novitsky RE, Kuznetsova TY, Boytsov SA. Improvement of cardiovascular risk assessment using machine learning methods. Russian Journal of Cardiology. 2021; 26(12): 171-80.
27. Song X MACJRK. Comparison of machine learning techniques with classical statistical models in predicting health outcomes. Stud Health Technol Inform. 2004; 107: 736-40.
28. Wu J, Roy J, Stewart WF. Prediction Modeling Using EHR Data Challenges, Strategies, and a Comparison of Machine Learning Approaches [Internet]. 2010. Available from: www.lww-medicalcare.com.
Review
For citations:
Ermak A.D., Gavrilov D.V., Kuznetsova T.Yu., Andreichenko A.E., Makarova E.A., Novitskiy R.E., Gusev A.V. Comparison of prognostic accuracy of score scale and a machine learning model in predicting fatal cardiovascular complications. Medical Doctor and Information Technologies. 2025;(4):86-98. (In Russ.) https://doi.org/10.25881/18110193_2025_4_86
Views:
31
Email this article 