Assessment of the maturity of artificial intelligence technologies for healthcare: methodology and its application based on the use of innovative computer vision technologies for medical image analysis and subsequent applicability in the healthcare system of Moscow

Aim: to develop and test a methodology for assessing the maturity of healthcare software based on artificial intelligence (AI).

Materials and methods. The methodology for developing a maturity matrix for AI-based healthcare software is based on published data and on an analysis of our own practical experience obtained during the «Experiment on the use of innovative technologies in the field of computer vision for the analysis of medical images and further application in the Moscow healthcare system « in 2021–2022. We studied study results from 35 separate software products based on AI, covering key areas of radiology.

Results. We developed a maturity matrix that takes into account the indicators of technical stability — the proportion of technological defects, and the diagnostic component — the area under the characteristic curve. This model has been tested in 35 software products based on AI, with 40% of the products having achieved maturity. The dynamics of development was assessed for 24 software products based on AI: 15 of them (62%) were in the zone of diagnostic stagnation; 8 (33%) — in the zone of high diagnostic and technical potential, 1 (4%) — in the zone of low diagnostic and technical potential, and 1 (4%) worsened the technical component with the increase in diagnostic potential.

Conclusion. A methodology for assessing the maturity of AI for healthcare has been developed based on the performance and quality assessment of 35 software products. This methodology includes a maturity matrix and a method for assessing the clinical and technical transformation of maturity, which makes it possible to evaluate an AI-based software product both discretely (simultaneously) and in dynamics.

1. Pugachev PS, Gusev AV, Kobjakova OS, Kadyrov FN, Gavrilov DV, Novickij RJe, Vladzimirskij AV. Global trends in the digital transformation of the healthcare industry. National Healthcare. 2021; 2(2): 5-12. (In Russ.)

2. Lisickij NN, Antohin JuN. Management of the digital development of the Russian healthcare organizational system: national and global agenda. Economy. Right. Innovation.2021; 4: 49-54. (In Russ.)

3. Gusev AV, Vladzimirskij AV, Golubev NA, Zarubina TV. Informatization of healthcare in the Russian Federation: history and results of development. National Health. 2021; 2(3): 5-17. (In Russ.)

4. Iyamu I, Xu AXT, Gómez-Ramírez O, Ablona A, Chang HJ, Mckee G, Gilbert M. Defining Digital Public Health and the Role of Digitization, Digitalization, and Digital Transformation: Scoping Review. JMIR Public Health Surveill. 2021; 7(11): e30399. doi: 10.2196/30399.

5. Orlov GM, Levin MB. Methodological approaches to the development of reference models of state information systems in the field of health care of the constituent entities of the Russian Federation. Information resources of Russia. 2021; 2(180): 20-27. (In Russ.)

6. Cresswell K, Sheikh A, Krasuska M, Heeney C, Franklin BD, Lane W, Mozaffar H, Mason K, Eason S, Hinder S, Potts HWW, Williams R. Reconceptualising the digital maturity of health systems. Lancet Digit Health. 2019; 1(5): e200-e201. doi: 10.1016/S2589-7500(19)30083-4.

7. Kouroubali A, Papastilianou A, Katehakis DG. Preliminary Assessment of the Interoperability Maturity of Healthcare Digital Services vs Public Services of Other Sectors. Stud Health Technol Inform. 2019; 264: 654-658. doi: 10.3233/SHTI190304.

8. Liaw ST, Zhou R, Ansari S, Gao J. A digital health profile & maturity assessment toolkit: cocreation and testing in the Pacific Islands. J Am Med Inform Assoc. 2021; 28(3): 494-503. doi: 10.1093/jamia/ocaa255.

9. Duncan R, Eden R, Woods L, Wong I, Sullivan C. Synthesizing Dimensions of Digital Maturity in Hospitals: Systematic Review. J Med Internet Res. 2022; 24(3): e32994. doi: 10.2196/32994.

10. Woods L, Eden R, Pearce A, Wong YCI, Jayan L, Green D, McNeil K, Sullivan C. Evaluating Digital Health Capability at Scale Using the Digital Health Indicator. Appl Clin Inform. 2022; 13(5): 991-1001. doi: 10.1055/s-0042-1757554.

11. Orlov GM. A method for measuring the digital maturity of a regional appointment system based on a reference service model. International Journal of Open Information Technologies. 2020; 8(11): 110-121. (In Russ.)

12. Shul’kin IM, Vladzimirskij AV. Data-driven management in radiology: assessment of the effectiveness of the Unified Radiological Information Service model. Health manager. 2022; 7: 68-80. (In Russ.)

13. Studzinski J. Evaluating the maturity of IT-supported clinical imaging and diagnosis using the Digital Imaging Adoption Model : Are your clinical imaging processes ready for the digital era? Radiologe. 2017; 57(6): 466-469. doi: 10.1007/s00117-017-0253-8.

14. Morozov SP, Vladzymyrskyy AV, Safronov DS. Benchmarking for assessing the quality of digitizationof radiology departments: methodology development Physician and information technology. 2019; 1: 40-45. (In Russ.)

15. Flott K, Callahan R, Darzi A, Mayer E. A Patient-Centered Framework for Evaluating Digital Maturity of Health Services: A Systematic Review. J Med Internet Res. 2016; 18(4): e75. doi: 10.2196/jmir.5047.

16. Khanbhai M, Flott K, Darzi A, Mayer E. Evaluating Digital Maturity and Patient Acceptability of RealTime Patient Experience Feedback Systems: Systematic Review. J Med Internet Res. 2019; 21(1): e9076. doi: 10.2196/jmir.9076.

17. Gusev AV, Astapenko EM, Ivanov IV, Zarubina TV, Kobrinskij BA. Principles of building trust in artificial intelligence systems for healthcare. Bulletin of Roszdravnadzor. 2022; 2: 25-33. (In Russ.)

18. Schoonenboom J, Johnson RB. How to Construct a Mixed Methods Research Design. Kolner Z Soz Sozpsychol. 2017; 69(2): 107-131. doi:10.1007/s11577-017-0454-1.

19. Computer vision in radiology: the first stage of the Moscow experiment: monograph. Y.A. Vasilyev, A.V. Vladzimirsky, editors. 2022. 388p (In Russ.)

20. Decree of the Government of Moscow 21.11.2019 г. № 1543-ПП (Ed. 22.02.2022 №225-ПП) (In Russ.)

21. Morozov SP, Vladzymyrskyy AV, Sharova DE, Akhmad ES, Zinchenko VV. The first russian federation national standards for artificial intelligence systems in medicine. Quality management in medicine. 2022; 1: 58-62(In Russ.)

22. Order of the Moscow Department of Health of 24.02.2022 №160 «On approval of the Procedure and conditions for conducting an experiment on the use of innovative technologies in the field of computer vision for the analysis of medical images and further application in the healthcare system of the city of Moscow». (In Russ.)

23. Morozov SP, Vladzimirskij AV, Kljashtornyj VG, et all. Clinical trials of software based on intelligent technologies (radiation diagnostics). Moscow: Scientific and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow City Health Department, 2019. (In Russ.)

24. Sadiq RB, Safie N, Abd Rahman AH, Goudarzi S. Artificial intelligence maturity model: a systematic literature review. PeerJ Comput Sci. 2021; 7: e661. doi: 10.7717/peerj-cs.661.

25. Coates DL, Martin A. An instrument to evaluate the maturity of bias governance capability in artificial intelligence projects. IBM Journal of Research and Development. 2019; 4(5): 1–7; 15. doi: 10.1147/JRD.2019.2915062.

26. Kreutzer RT, Sirrenberg M. AI challenge — how artificial intelligence can be anchored in a company. Understanding Artificial Intelligence 2020; Cham: Springer; 2020: 235-273.

27. Lichtenthaler U. Five maturity levels of managing AI: from isolated ignorance to integrated intelligence. Journal of Innovation Management. 2020; 8(1): 39-50. doi: 10.24840/2183-0606_008.001_0005.

28. Ellefsen APT, Oleśków-Szłapka J, Pawłowski G, Toboła A. Striving for excellence in AI implementation: AI maturity model framework and preliminary research results. LogForum. 2019; 15: 363-376.

29. Tu H, Lin Z, Lee K. Automation With Intelligence in Drug Research. Clin Ther. 2019; 41(11): 2436-2444. doi: 10.1016/j.clinthera.2019.09.002.

30. Lamberti MJ, Wilkinson M, Donzanti BA, Wohlhieter GE, Parikh S, Wilkins RG, Getz K. A Study on the Application and Use of Artificial Intelligence to Support Drug Development. Clin Ther. 2019; 41(8): 1414-1426. doi: 10.1016/j.clinthera.2019.05.018.

31. Tariq A, Purkayastha S, Padmanaban GP, Krupinski E, Trivedi H, Banerjee I, Gichoya JW. Current Clinical Applications of Artificial Intelligence in Radiology and Their Best Supporting Evidence. J Am Coll Radiol. 2020; 17(11): 1371-1381. doi: 10.1016/j.jacr.2020.08.018.

32. Kelly BS, Judge C, Bollard SM, Clifford SM, Healy GM, Aziz A, Mathur P, Islam S, Yeom KW, Lawlor A, Killeen RP. Radiology artificial intelligence: a systematic review and evaluation of methods (RAISE). Eur Radiol. 2022 Apr 14. doi: 10.1007/s00330-022-08784-6.

33. Jin C, Chen W, Cao Y, Xu Z, Tan Z, Zhang X, Deng L, Zheng C, Zhou J, Shi H, Feng J. Development and evaluation of an artificial intelligence system for COVID-19 diagnosis. Nat Commun. 2020; 11(1): 5088. doi: 10.1038/s41467-020-18685-1.

34. Namiri NK, Lee J, Astuto B, Liu F, Shah R, Majumdar S, Pedoia V. Deep learning for large scale MRIbased morphological phenotyping of osteoarthritis. Sci Rep. 2021; 11(1): 10915. doi: 10.1038/s41598-021-90292-6.

35. Stemmer A, Shadmi R, Bregman-Amitai O, Chettrit D, Blagev D, Orlovsky M, Deutsch L, Elnekave E. Using machine learning algorithms to review computed tomography scans and assess risk for cardiovascular disease: Retrospective analysis from the National Lung Screening Trial (NLST). PLoS One. 2020; 15(8): e0236021. doi: 10.1371/journal.pone.0236021.

36. Kuo RYL, Harrison C, Curran TA, Jones B, Freethy A, Cussons D, Stewart M, Collins GS, Furniss D. Artificial Intelligence in Fracture Detection: A Systematic Review and Meta-Analysis. Radiology. 2022 Jul; 304(1): 50-62. doi: 10.1148/radiol.211785.

37. Murray NM, Unberath M, Hager GD, Hui FK. Artificial intelligence to diagnose ischemic stroke and identify large vessel occlusions: a systematic review. J Neurointerv Surg. 2020; 12(2): 156-164. doi: 10.1136/neurintsurg-2019-015135.

38. Gavrilov PV, Gavrilova OP, Smol’nikova UA. Identification of peripheral formations in the lungs using a program for automated analysis of fluorographic images. Radiation diagnostics and therapy. 2020; S1: 77. (In Russ.)

39. Drokin IS, Ericheva EV, Buhvalov OL, Piljus PS, Malygina TS, Sinicyn VE. Experience in the development and implementation of a system for searching for oncological formations using artificial intelligence on the example of X-ray computed tomography of the lungs. Physician and information technology. 2019; 3: 48-57. (In Russ.)

40. Pavlovich PI, Bronov OJu, Kapninskij AA, Abovich JuA, Rychagova NI. Comparative study of the results of digital mammography data analysis of the system based on artificial intelligence «Celsus» and radiologists. Digital Diagnostics. 2021; 2(S2): 22-23. (In Russ.)

41. Komar’ PA, Dmitriev VS, Ledjaeva AM, Shaderkin IA, Zelenskij MM. Rating of artificial intelligence startups: prospects for healthcare in Russia. Russian journal of telemedicine and e-health. 2021; 7(3): 32-41. (In Russ.)