Comparing Time-Fixed Mortality Prediction Models and Their Effect on ICU Performance Metrics Using the Simplified Acute Physiology Score 3

Lars Engerström, Andrew A. Kramer, Thomas Nolin, Folke Sjöberg, Göran Karlström, Mats Fredrikson, Sten M. Walther

To examine ICU performance based on the Simplified Acute Physiology Score 3 using 30-day, 90-day, or 180-day mortality as outcome measures and compare results with 30-day mortality as reference. Retrospective cohort study of ICU admissions from 2010 to 2014. Sixty-three Swedish ICUs that submitted data to the Swedish Intensive Care Registry. The development cohort was first admissions to ICU during 2011-2012 (n = 53,546), and the validation cohort was first admissions to ICU during 2013-2014 (n = 57,729). None. Logistic regression was used to develop predictive models based on a first level recalibration of the original Simplified Acute Physiology Score 3 model but with 30-day, 90-day, or 180-day mortality as measures of outcome. Discrimination and calibration were excellent for the development dataset. Validation in the more recent 2013-2014 database showed good discrimination (C-statistic: 0.85, 0.84, and 0.83 for the 30-, 90-, and 180-d models, respectively), and good calibration (standardized mortality ratio: 0.99, 0.99, and 1.00; Hosmer-Lemeshow goodness of fit H-statistic: 66.4, 63.7, and 81.4 for the 30-, 90-, and 180-d models, respectively). There were modest changes in an ICU's standardized mortality ratio grouping (< 1.00, not significant, > 1.00) when follow-up was extended from 30 to 90 days and 180 days, respectively; about 11-13% of all ICUs. The recalibrated Simplified Acute Physiology Score 3 hospital outcome prediction model performed well on long-term outcomes. Evaluation of ICU performance using standardized mortality ratio was only modestly sensitive to the follow-up time. Our results suggest that 30-day mortality may be a good benchmark of ICU performance. However, the duration of follow-up must balance between what is most relevant for patients, most affected by ICU care, least affected by administrative policies and practically feasible for caregivers.