Evaluation of the effectiveness of a medical decision support system for preliminary diagnosis in Moscow outpatient clinics
https://doi.org/10.25881/18110193_2025_3_36
Abstract
The implementation of the medical decision support system (MDSS) in clinical practice requires careful monitoring to ensure patient safety and track the performance of artificial intelligence technologies.
Goal: To assess the effectiveness of the TOP-3 MDSS in outpatient clinics of the Moscow Heath Care Department.
Materials and methods: The Moscow Heath Care Department monitored the TOP-3 operation between October 1, 2020 and March 03, 2024 (n = 63,809,360 people). The Hit-3 metric was used to determine if the MDSS needs retraining. An additional study involving medical experts included a retrospective analysis of data from 3,000 patients and calculation of the diagnostic agreement rate among the MDSS, doctors and the experts.
Results: The monitoring demonstrated mean Hit-3 of 63.5, 64.5 and 67.7 for MDSS ver. 1, 2, and 3, respectively. In cases where the clinicians disagreed with the MDSS (n = 2000), based on the patient complaints, experts agreed with the MDSS in 80.2% of cases, with clinicians in 11.5% of cases, and established a different diagnosis in 8.3% of cases. In cases where the clinicians’ and the MDSS’s conclusions matched (n = 1000), experts approved the diagnosis in 50.4% of cases, selected an alternative diagnosis suggested by the MDSS in 37.9% of cases, and established a different diagnosis for 11.7% of patients. Conclusion: The described monitoring methodology, supplemented by expert review, allowed for a comprehensive assessment of the MDSS to be implemented in the health care system. Based on the results of the TOP-3 effectiveness assessment, it was decided that the analyzed list of data from electronic health records should be expanded, which will be implemented in the upcoming version TOP-3+.
About the Authors
M. V. KirinaRussian Federation
Moscow
A. S. Bezymyannyy
Russian Federation
Moscow
YU. A. Vasilev
Russian Federation
PhD
Moscow
E. V. Blokhina
Russian Federation
Moscow
B. I. Karamov
Russian Federation
Moscow
A. S. Abrosimov
Russian Federation
Moscow
A. S. Abrosimov
Russian Federation
Moscow
K. M. Arzamasov
Russian Federation
DSc
Moscow
A. P. Pamova
Russian Federation
PhD
Moscow
V. E. Kazarinova
Russian Federation
Moscow
References
1. Указ Президента Российской Федерации от 10.10.2019 г. №490 «О развитии искусственного интеллекта в Российской Федерации». http://www.kremlin.ru/acts/bank/44731.
2. Vasiliev YuA, Vladzimirskij AV, Arzamasov KM. Komp`yuternoe zrenie v luchevoj diagnostike: pervy`j e`tap Moskovskogo e`ksperimenta. M.: Izdatel`skie resheniya. 2023. 376 р. (In Russ.)
3. Vasilev YA, Tyrov IA, Vladzymyrskyy AV, Arzamasov KM, et al. Doublereading mammograms using artificial intelligence technologies: A new model of mass preventive examination organization. Digital Diagnostics. 2023; 4(2): 93-104. (In Russ.) doi: 10.17816/DD321423.
4. Komkov AA, Ryazanova SV, Mazaev VP. Effective decision support systems in Clinical Practice and Prevention: Literature Review. CardioSomatics. 2023; 14(3): 177-185. (In Russ.) doi: 10.17816/CS569263.
5. Gusev AV, Tokarev SA, Gavrilov DV, Kuzneczova TYu. Primenenie avtomatizirovannoj sistemy` podderzhki prinyatiya vrachebny`x reshenij pri dispanserizacii vzroslogo naseleniyadlya kontrolya pravil`nosti ocenki urovnya serdechnososudistogo riska. Menedzhment kachestva v medicine. 2022; 1: 177-185. (In Russ.)
6. Taheri Moghadam S, Sadoughi F, Velayati F, Ehsanzadeh SJ, Poursharif S. The effects of clinical decision support system for prescribing medication on patient outcomes and physician practice performance: a systematic review and meta-analysis. BMC Med Inform Decis Mak. 2021; 21(1): 98. doi: 10.1186/s12911-020-01376-8.
7. Vasiliev YA, Lantux ZA, Shatyonok MP, Soldatov IV, et al. Metodika vy`bora sistemy` podderzhki prinyatiya vrachebny`x reshenij v luchevoj diagnostike dlya obespecheniya radiacionnoj bezopasnosti: metodicheskie rekomendacii. M.: Moscow Center for Diagnostics and Telemedicine; 2023. (In Russ.)
8. Edward HS, Bruce GB. A Model of Inexact Reasoning in Medicine. Mathematical Biosciences. 1975; 23: 351-379. doi: 10.1016/0025-5564(75)90047-4.
9. Pugachev PS, Gusev AV, Kobyakova OS, Kady`rov FN, et al. Mirovy`e trendy` cifrovoj transformacii otrasli zdravooxraneniya. Nacional`noe zdravooxranenie. 2021; 2(2): 5-12. (In Russ.)] 10. Регистрационное удостоверение на медицинское изделие от 24.05.2022 №РЗН 2022/17272. https://goszdravnadzor.ru/ru-roszdravnadzora-№-rzn-2022-17272-ot-24-maya-2022-go/
10. Blinov P, Avetisian M, Kokh V, Umerenkov D, Tuzhilin A. Predicting Clinical Diagnosis from Patients Electronic Health Records Using BERT-Based Neural Networks. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). 2020: 111-121. doi: 10.1007/978-3-030-59137-3_11.
11. Martinez-Franco AI, Sanchez-Mendiola M, Mazon-Ramirez JJ, et al. Diagnostic accuracy in Family Medicine residents using a clinical decision support system (DXplain): a randomized-controlled trial. Diagnosis (Berl). 2018; 5(2): 71-76. doi: 10.1515/dx-2017-0045.
12. Benditz A, Pulido LC, Grifka J, Ripke F, Jansen P. A clinical decision support system in back pain helps to find the diagnosis: a prospective correlation study. Arch Orthop Trauma Surg. 2023; 143(2): 621-625. doi: 10.1007/s00402-021-04080-y.
Review
For citations:
Kirina M.V., Bezymyannyy A.S., Vasilev Yu.A., Blokhina E.V., Karamov B.I., Abrosimov A.S., Abrosimov A.S., Arzamasov K.M., Pamova A.P., Kazarinova V.E. Evaluation of the effectiveness of a medical decision support system for preliminary diagnosis in Moscow outpatient clinics. Medical Doctor and Information Technologies. 2025;(3):36-49. (In Russ.) https://doi.org/10.25881/18110193_2025_3_36