Prospective internal validation of an XGBoost-based machine learning model for sepsis prediction in patients with prolonged ICU stays

Introduction: to prospectively validate a previously developed machine learning model for predicting sepsis onset in this patient population.

Methods: Prospective validation was conducted in accordance with TRIPOD+AI recommendations using data from the Russian Intensive Care Dataset (RICD v3.0). The XGBoost-based model estimated the risk of sepsis onset on an hourly basis with a 6-hour prediction window, using demographic, clinical, laboratory, and vital parameters aggregated into dynamic time windows. Model performance was evaluated using the area under the receiver operating characteristic curve (AUROC) with a 95% confidence interval (CI), as well as diagnostic metrics at the optimal cutoff point.

Results: Validation was performed on a cohort of patients admitted to the ICU of the Federal Research and Clinical Center of Intensive Care Medicine and Rehabilitology between October 2024 and September 2025 (n = 103; 53,561 patient-hours). The median ICU length of stay was 32 (27; 38) days. The XGBoost model achieved an AUROC of 0.737, consistent with the internal testing results (AUROC 0.752; 95% CI 0.730; 0.767). The optimal cutoff (≥0.5651) yielded a sensitivity of 60.3%, specificity of 81.5%, and a high negative predictive value (97.4%). Although the model showed a high rate of false positives, it demonstrated strong clinical utility for ruling out sepsis within the next 6 hours. Moreover, a multilevel interpretation approach allows risk stratification across low, intermediate, and high probability zones of sepsis development.

Conclusion: Prospective validation confirmed the stability and reproducibility of the machine learning model for sepsis prediction in patients with prolonged ICU stays. The results support the feasibility of integrating the model into an electronic health system as a component of a decision support framework for subsequent real-world evaluation.

1. Singer M, Deutschman CS, Seymour CW, et al. The third international consensus definitions for sepsis and septic shock (sepsis-3). JAMA - J Am Med Assoc. 2016; 315: 801-10. doi: 10.1001/jama.2016.0287.

2. La Via L, Sangiorgio G, Stefani S, et al. The Global Burden of Sepsis and Septic Shock. Epidemiologia. 2024; 5: 456-78. doi: 10.3390/epidemiologia5030032.

3. Arbous SM, Termorshuizen F, Brinkman S, et al. Three-year mortality of ICU survivors with sepsis, an infection or an inflammatory illness: an individually matched cohort study of ICU patients in the Netherlands from 2007 to 2019. Crit Care. 2024; 28: 374. doi: 10.1186/s13054-024-05165-x.

4. Fleischmann C, Scherag A, Adhikari NKJ, et al. Assessment of global incidence and mortality of hospital-treated sepsis current estimates and limitations. Am J Respir Crit Care Med. 2016; 193: 25972. doi: 10.1164/rccm.201504-0781OC.

5. Sakr Y, Jaschinski U, Wittebole X, et al. Sepsis in intensive care unit patients: Worldwide data from the intensive care over nations audit. Open Forum Infect Dis. 2018; 5: ofy313. doi: 10.1093/ofid/ofy313.

6. Lyanguzov AV, Luchinin AS, Ignat'ev SV, et al. Predictors of ICU mortality in patients with hemoblastosis and infectious complications. Anesteziologiya i Reanimatologiya. 2023; 1: 33-38. (In Russ.) doi: 10.17116/anaesthesiology202301133.

7. Kochkin AA, Yadgarov MY, Berikashvili LB, et al. Predictors of lethal outcomes in patients with refractory septic shock. Vestnik anesteziologii i reanimatologii. 2021; 18(3): 30-35. (In Russ.) doi: 10.21292/2078-5658-2021-18-3-30-35.

8. Abe T, Ogura H, Shiraishi A, et al. Characteristics, management, and in-hospital mortality among patients with severe sepsis in intensive care units in Japan: The FORECAST study. Crit Care. 2018; 22: 322. doi: 10.1186/s13054-018-2186-7.

9. Mohamed AKS, Mehta AA, James P. Predictors of mortality of severe sepsis among adult patients in the medical Intensive Care Unit. Lung India. 2017; 34: 330-5. doi: 10.4103/lungindia.lungindia_54_16.

10. Vincent JL, Marshall JC, Ñamendys-Silva SA, et al. Assessment of the worldwide burden of critical illness: The Intensive Care Over Nations (ICON) audit. Lancet Respir Med. 2014; 2: 380-6. doi: 10.1016/S2213-2600(14)70061-X.

11. Raghavan M, Marik PE. Management of sepsis during the early “golden hours”. J Emerg Med. 2006; 31: 185-99. doi: 10.1016/j.jemermed.2006.05.008.

12. Im Y, Kang D, Ko RE, et al. Time-to-antibiotics and clinical outcomes in patients with sepsis and septic shock: a prospective nationwide multicenter cohort study. Crit Care. 2022; 26: 19. doi: 10.1186/s13054-021-03883-0.

13. Evans L, Rhodes A, Alhazzani W, et al. Surviving sepsis campaign: international guidelines for management of sepsis and septic shock 2021. Intensive Care Med. 2021; 47: 1181-247. doi: 10.1007/s00134-021-06506-y.

14. Vincent J-L, Beumier M. Diagnostic and prognostic markers in sepsis. Expert Rev Anti Infect Ther. 2013; 11: 265-75. doi: 10.1586/eri.13.9.

15. He RR, Yue GL, Dong ML, et al. Sepsis Biomarkers: Advancements and Clinical Applications—A Narrative Review. Int J Mol Sci. 2024; 25. doi: 10.3390/ijms25169010.

16. Adegbite BR, Edoa JR, Ndzebe Ndoumba WF, et al. A comparison of different scores for diagnosis and mortality prediction of adults with sepsis in Low-and-Middle-Income Countries: a systematic review and meta-analysis. EClinicalMedicine. 2021; 42: 101184. doi: 10.1016/j.eclinm.2021.101184.

17. Konjety P, Chakole VG. Beyond the Horizon: A Comprehensive Review of Contemporary Strategies in Sepsis Management Encompassing Predictors, Diagnostic Tools, and Therapeutic Advances. Cureus. 2024; 16: e64249. doi: 10.7759/cureus.64249.

18. Sakib N, Ahamed SI, Khan RA, et al. Unpacking prevalence and dichotomy in quick sequential organ failure assessment and systemic inflammatory response syndrome parameters: Observational data–driven approach backed by sepsis pathophysiology. JMIR Med Informatics. 2020; 8: e18352. doi: 10.2196/18352.

19. Islam MM, Nasrin T, Walther BA, et al. Prediction of sepsis patients using machine learning approach: A meta-analysis. Comput Methods Programs Biomed. 2019; 170: 1-9. doi: 10.1016/j.cmpb.2018.12.027.

20. Yadgarov MY, Landoni G, Berikashvili LB, et al. Early detection of sepsis using machine learning algorithms: a systematic review and network meta-analysis. Front Med. 2024; 11: 1491358. doi: 10.3389/fmed.2024.1491358.

21. Fleuren LM, Klausch TLT, Zwager CL, et al. Machine learning for the prediction of sepsis: a systematic review and meta-analysis of diagnostic test accuracy. Intensive Care Med. 2020; 46: 383-400. doi: 10.1007/s00134-019-05872-y.

22. Likhvantsev VV., Berikashvili LB, Yadgarov MY, et al. The Tri-Steps Model of Critical Conditions in Intensive Care: Introducing a New Paradigm for Chronic Critical Illness. J Clin Med. 2024; 13. doi: 10.3390/jcm13133683.

23. Hawkins RB, Raymond SL, Stortz JA, et al. Chronic Critical Illness and the Persistent Inflammation, Immunosuppression, and Catabolism Syndrome. Front Immunol. 2018; 9: 1511. doi: 10.3389/fimmu.2018.01511.

24. Bagshaw SM, Stelfox HT, Iwashyna TJ, et al. Timing of onset of persistent critical illness: a multi-centre retrospective cohort study. Intensive Care Med. 2018; 44: 2134-44. doi: 10.1007/s00134-018-5440-1.

25. Yadgarov MY, Rebrova OY, Berikashvili LB, et al. Machine Learning Model for Sepsis Prediction in Prolonged and Chronic Critical Illness: Development and Validation Using Retrospective Real-World ICU Data. J Clin Med. 2026; 15: 777. doi: 10.3390/jcm15020777.

26. Grechko AV, Yadgarov MY, Yakovlev AA, et al. Russian Intensive Care Dataset - RICD. Obshchaya Reanimatologiya. 2024; 20(3): 22-31. (In Russ.)] doi: 10.15360/1813-9779-2024-3-22-31.

27. RICD - Open dataset [updated 2025 Oct 16; cited 2025 Dec 9. Available from: https://fnkcrr-database. ru.

28. Collins GSS, Reitsma JBB, Altman DGG, et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): The TRIPOD Statement. BMC Med. 2015; 13: 1. doi: 10.1186/s12916-014-0241-z.

29. Collins GS, Moons KGM, Dhiman P, et al. TRIPOD+AI statement: Updated guidance for reporting clinical prediction models that use regression or machine learning methods. Bmj. 2024; 385. doi: 10.1136/bmj-2023-078378.

30. Wang Z, Wang W, Sun C, et al. A methodological systematic review of validation and performance of sepsis real-time prediction models. Npj Digit Med. 2025; 8: 190. doi: 10.1038/s41746-025-01587-1.

31. Schvetz M, Fuchs L, Novack V, et al. Outcomes prediction in longitudinal data: Study designs evaluation, use case in ICU acquired sepsis. J Biomed Inform. 2021; 117: 103734. doi: 10.1016/j.jbi.2021.103734.

32. Liu Z, Khojandi A, Mohammed A, et al. HeMA: A hierarchically enriched machine learning approach for managing false alarms in real time: A sepsis prediction case study. Comput Biol Med. 2021; 131: 104255. doi: 10.1016/j.compbiomed.2021.104255.

33. Yu SC, Gupta A, Betthauser KD, et al. Sepsis Prediction for the General Ward Setting. Front Digit Heal. 2022; 4: 848599. doi: 10.3389/fdgth.2022.848599.

34. Rockenschaub P, Akay EM, Carlisle BG, et al. External validation of AI-based scoring systems in the ICU: a systematic review and meta-analysis. BMC Med Inform Decis Mak. 2025; 25: 5. doi: 10.1186/s12911-024-02830-7.

35. Wong A, Otles E, Donnelly JP, et al. External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients. JAMA Intern Med. 2021; 181: 1065-70. doi: 10.1001/jamainternmed.2021.2626.

36. Amland RC, Haley JM, Lyons JJ. A Multidisciplinary Sepsis Program Enabled by a Two-Stage Clinical Decision Support System: Factors That Influence Patient Outcomes. Am J Med Qual. 2016; 31: 501-8. doi: 10.1177/1062860615606801.

37. Narayanan N, Gross AK, Pintens M, et al. Effect of an electronic medical record alert for severe sepsis among ED patients. Am J Emerg Med. 2016; 34: 185-8. doi: 10.1016/j.ajem.2015.10.005.

38. Tafelski S, Nachtigall I, Deja M, et al. Computer-assisted Decision Support for Changing Practice in Severe Sepsis and Septic Shock. J Int Med Res. 2010; 38: 1605-16. doi: 10.1177/147323001003800505.

39. Liu VX, Fielding-Singh V, Greene JD, et al. The timing of early antibiotics and hospital mortality in sepsis. Am J Respir Crit Care Med. 2017; 196: 856-63. doi: 10.1164/rccm.201609-1848OC.

40. Agnello L, Vidali M, Padoan A, et al. Machine learning algorithms in sepsis. Clin Chim Acta. 2024; 553: 117738. doi: 10.1016/j.cca.2023.117738.

41. Boussina A, Shashikumar SP, Malhotra A, et al. Impact of a deep learning sepsis prediction model on quality of care and survival. Npj Digit Med. 2024; 7: 14. doi: 10.1038/s41746-023-00986-6.

42. Sendak MP, Ratliff W, Sarro D, et al. Real-world integration of a sepsis deep learning technology into routine clinical care: Implementation study. JMIR Med Informatics. 2020; 8: e15182. doi: 10.2196/15182.

43. Adams R, Henry KE, Sridharan A, et al. Prospective, multi-site study of patient outcomes after implementation of the TREWS machine learning-based early warning system for sepsis. Nat Med. 2022; 28: 1455-60. doi: 10.1038/s41591-022-01894-0.

44. Valan B, Prakash A, Ratliff W, et al. Evaluating sepsis watch generalizability through multisite external validation of a sepsis machine learning model. Npj Digit Med. 2025; 8: 350. doi: 10.1038/s41746-025-01664-5.