Author + information
- Received October 15, 2012
- Revision received January 22, 2013
- Accepted January 24, 2013
- Published online June 1, 2013.
- Zubin J. Eapen, MD∗∗ (, )
- Li Liang, PhD∗,
- Gregg C. Fonarow, MD†,
- Paul A. Heidenreich, MD, MS‡,
- Lesley H. Curtis, PhD∗,
- Eric D. Peterson, MD, MPH∗ and
- Adrian F. Hernandez, MD, MHS∗
- ↵∗Reprint requests and correspondence:
Dr. Zubin J. Eapen, Duke Clinical Research Institute, PO Box 17969, Durham, North Carolina 27715.
Objectives The study sought to derive and validate risk-prediction tools from a large nationwide registry linked with Medicare claims data.
Background Few clinical models have been developed utilizing data elements readily available in electronic health records (EHRs) to facilitate “real-time” risk estimation.
Methods Heart failure (HF) patients ≥65 years of age hospitalized in the GWTG-HF (Get With The Guidelines-Heart Failure) program were linked with Medicare claims from January 2005 to December 2009. Multivariable models were developed for 30-day mortality after admission, 30-day rehospitalization after discharge, and 30-day mortality/rehospitalization after discharge. Candidate variables were selected based on availability in EHRs and prognostic value. The models were validated in a 30% random sample and separately in patients with reduced and preserved ejection fraction (EF).
Results Among 33,349 patients at 160 hospitals, 3,002 (9.1%) died within 30 days of admission, 7,020 (22.8%) were rehospitalized within 30 days of discharge, and 8,374 (27.2%) died or were rehospitalized within 30 days of discharge. Compared with patients classified as low risk, high-risk patients had significantly higher odds of death (odds ratio [OR]: 8.82, 95% confidence interval [CI]: 7.58 to 10.26), rehospitalization (OR: 1.99, 95% CI: 1.86 to 2.13), and death/rehospitalization (OR: 2.65, 95% CI: 2.44 to 2.89). The 30-day mortality model demonstrated good discrimination (c-index 0.75) while the rehospitalization and death/rehospitalization models demonstrated more modest discrimination (c-indices of 0.59 and 0.62), with similar performance in the validation cohort and for patients with preserved and reduced EF.
Conclusions These predictive models allow for risk stratification of 30-day outcomes for patients hospitalized with HF and may provide a validated, point-of-care tool for clinical decision making.
Heart failure (HF) is the top discharge diagnosis for Medicare and leading cause of rehospitalization within 30 days of an index hospitalization (1). With reductions in hospital length of stay, more deaths within 30 days of admission for HF now occur post-discharge (2). As a result, short-term outcomes for HF are a central focus for patients, providers, payers, and policy makers. Recognizing the need to curb rehospitalization rates, the Centers for Medicare and Medicaid Services (CMS) initiated public reporting of risk-adjusted rehospitalization rates for hospitals to encourage improvements in local care and will reduced payments for hospitals with high rehospitalization rates for HF beginning in October 2012 (3,4).
Several options are available to hospitals to prevent rehospitalizations for HF patients (5–10). A risk-specific approach toward the deployment of these interventions could target these services to patients who will derive the greatest benefit. Models have been created to predict outcomes for inpatients or prognosis for outpatients (11–18). Some were developed using local HF populations to provide automated risk calculation within a single electronic health record (EHR) (19). However, these models often incorporate data elements, such as comorbidities, that are not systematically and consistently captured across different information systems (20). As a result, their adoption and impact on real-world clinical practice may remain limited. To date, models using routinely collected inpatient data to predict post-discharge outcomes and capable of implementation across multiple EHR systems are not available. Using Get With The Guidelines-Heart Failure (GWTG-HF) linked with the Medicare database, we derived and validated risk-prediction tools from a large nationwide registry that would: 1) apply to Medicare beneficiaries with HF similar to those encountered in real-world clinical practice; 2) leverage existing data in EHRs; and 3) provide clinicians and health systems with a validated risk-stratification tool for short-term outcomes of clinical, economic, and policy significance.
We linked data from the GWTG-HF (Get With The Guidelines-Heart Failure) registry with enrollment files and inpatient claims from the CMS from January 1, 2005, through December 31, 2009. We followed patients through the end of 2009. The design, inclusion criteria, and data collection methods have been published (21). Patients were eligible for inclusion if they were admitted for an episode of worsening HF or developed significant HF symptoms during a hospitalization for which HF was the primary discharge diagnosis. Hospital teams used HF case-ascertainment methods similar to those used by the Joint Commission. Data on medical history, signs and symptoms, medications, contraindications for or intolerance to medications, and diagnostic test results were collected via a web-based registry. All regions of the United States were represented and a variety of centers participated, from community hospitals to large tertiary centers.
All participating institutions were required to comply with local regulatory and privacy guidelines and, if applicable, to obtain institutional review board approval. Because the data were used primarily at the local site for quality improvement, sites were granted a waiver of informed consent under the Common Rule. Outcome Sciences, Inc. (Cambridge, Massachusetts) served as the registry coordinating center. The Duke Clinical Research Institute (Durham, North Carolina) served as the data analysis center.
The CMS files included data for all fee-for-service Medicare beneficiaries age ≥65 years hospitalized with a diagnosis of HF (International Classification of Diseases [ICD]-Ninth Revision-Clinical Modification 428.x, 402.x1, 404.x1, and 404.x3). We linked patient data in the registry with Medicare Part A inpatient claims, matching by admission and discharge dates, hospital, date of birth, and sex (22).
Consistent with metrics publicly reported by CMS (23), the 3 outcomes of interest were mortality within 30 days of admission, all-cause rehospitalization within 30 days after discharge, and all-cause mortality or rehospitalization within 30 days after discharge. As with other studies of Medicare beneficiaries, we obtained dates of death from CMS enrollment files and rehospitalization dates from Part A inpatient claims.
There were 2 analysis populations based on eligibility for the mortality and rehospitalization outcomes. Overall, the linked data set included 45,410 HF patients according to clinical and ICD-9 diagnoses age ≥65 years for whom Medicare data were available at 244 fully participating hospitals. For patients with multiple hospitalizations recorded in the registry, we used information from the earliest hospitalization. We excluded hospitals that had fewer than 15 cases (N = 225; 25 hospitals), high missingness (≥25%) on admission lab tests (N = 7,565; 32 hospitals), and high missingness on discharge lab tests (N = 4,271; 27 hospitals). We excluded patients with less than 30 days from index admission to the end of the study (n = 515), leaving 32,834 patients and 160 hospitals in the mortality analysis. To evaluate post-discharge outcomes, we also excluded in-hospital deaths (n = 1,274), index discharge date to end of study <30 days (n = 589), and patients who transferred to an acute care facility (n = 658), leaving 30,828 patients and 160 hospitals in the post-discharge analyses.
We described baseline characteristics using percentages for categorical variables and medians with 25th and 75th percentiles for continuous variables. Multivariable logistic regression models were developed using the generalized estimating equations method to account for within-hospital clustering. A 70% sample was randomly selected to construct derivation sample and derive the predictor model for each outcome, and the remaining 30% was used as validation sample to validate each model. Based upon their clinical importance, likely availability in EHR data, and significance of the statistical tests, variables for the reduced model were chosen from the derivation sample and validated in the validation sample. C statistics were used in the validation to evaluate the model performance on discrimination. The plots of observed versus predicted outcomes were used to evaluate model performance on calibration. The validated model that included the variables selected was then refitted in the whole study sample. The reduced models were also validated in the clinically important HF subgroups: HF-reduced ejection fraction (REF) (patients with a documented left ventricular [LV] EF <40%) and HF-preserved EF (PEF) (patients with a documented LV EF ≥40%). Finally, patients were classified into high-, moderate-, and low-risk groups based upon their predicted risk using the validated models. Each outcome was compared across the risk tertiles using low-risk patients as the referent.
Overall missing data was ≤8% with the exception of discharge labs (<10%), BNP (22%), and troponin abnormal (vs. normal) (22%). Missingness of specific variables was imputed as follows: sex as male and race as white. Missingness for the continuous vital signs and lab variables were imputed using the corresponding median value. Weight missing was imputed to sex-specific median weight values. Plots displaying the relationship of each continuous variable with the log odds of each outcome were used to assess appropriateness of the linearity assumption. When appropriate, knots were determined from the plots to create splines. Age, respiratory rate, and blood urea nitrogen (BUN) were included as continuous variables because of their linear relationship with log odds of outcomes, and the remaining continuous variables were modeled as splines. Only significant splines were included in the reduced models. Continuous variables were truncated outside their significant spline intervals. A sensitivity analysis was performed using only patients who had complete data for each model.
All p values were 2-sided, with p <0.05 considered statistically significant. All analyses were performed using SAS software (version 9.2, SAS Institute, Inc., Cary, North Carolina).
A final study population of 33,349 patients at 160 hospitals was obtained. The median age of the overall cohort was 80 years (25th, 75th: 74, 86), 45.6% were men, and 81.0% were white (Table 1). Ischemic heart disease occurred in 60.9% of the cohort and the median EF was 43% (25th, 75th: 30, 55). Among patients with a documented EF, 12,685 patients were classified as having HF-REF and 18,123 as having HF-PEF. Comorbidities were common with diabetes in 39.7%, atrial fibrillation in 36.1%, chronic obstructive pulmonary disease or asthma in 28.3%, hypertension in 75.3%, and chronic renal insufficiency in 18.6%. Among the 32,834 patients who had at least 30 days of follow-up after index admission, 3002 (9.1%) died within 30 days of admission. Among the 30,828 patients who survived until discharge, 7,020 (22.8%) were readmitted within 30 days of discharge, and 8,374 (27.2%) either died or were readmitted within 30 days of discharge.
Multivariate predictors of 30-day outcomes were derived in the 70% derivation sample, validated in the 30% independent validation sample, and refit using the whole cohort (Table 2). Model calibrations were retained when testing the models in the independent validation cohort (Fig. 1). To assess inpatient and post-discharge mortality within 30 days of admission, a model of 13 independent predictors within domains of data routinely collected and available in EHRs—demographics, vital signs, laboratory data—was derived and validated. Admission BUN value (chi-square test: 441.2, odds ratio [OR]: per 10 mg/dl 1.23, 95% confidence interval [CI]: 1.21 to 1.26) and admission systolic blood pressure (chi-square test: 402.9, OR per 10 mm Hg: 0.82, 95% CI: 0.81 to 0.84) were the strongest in this model. Using the cohort of patients who survived until discharge from an index hospitalization, a 30-day rehospitalization model of 10 independent predictors available in EHR data was derived and validated. The admission hemoglobin value (chi-square test: 66.5, OR per 1 g/dl: 0.91, 95% CI: 0.89 to 0.93) and discharge sodium value (chi-square test: 58.0, OR per 5 mEq/l: 0.78, 95% CI: 0.73 to 0.83) were the strongest predictors of 30-day rehospitalization in this model. With this same cohort, a model composed of 11 independent predictors was developed for the prediction of 30-day mortality or rehospitalization. Similar to the 30-day mortality model, admission systolic blood pressure (chi-square test: 138.6, OR per 10 mm Hg: 0.91, 95% CI: 0.89 to 0.92) and admission BUN value (chi-square test: 102.7, OR per 10 mg/dl: 1.10, 95% CI: 1.08 to 1.12) were strongest in this model.
In both the derivation and validation cohorts, the 30-day mortality model demonstrated good discrimination with a c-index of 0.75 while the rehospitalization and mortality/rehospitalization models demonstrated more modest discrimination with c-indices of 0.59 and 0.62, respectively (Table 3). Similar c-indices and plots of observed versus predicted outcomes were obtained when validating the models among only patients with complete data who did not require imputation. When evaluated in patients with only HF reduced ejection fraction (HF-REF) or preserved ejection fraction (HF-PEF), the models discriminated similarly well (HF-REF: C statistic for mortality 0.76, rehospitalization 0.60, mortality or rehospitalization 0.64; HF-PEF: mortality 0.73, rehospitalization 0.58, mortality or rehospitalization 0.61). For the 30-day rehospitalization outcome, a simple model using a history of prior all-cause hospitalizations within 6 months had similar performance (C statistic 0.62) compared with the EHR model.
Mean 30-day outcomes varied according to risk classification for patients (Table 4). Compared with patients classified as low risk, medium-risk patients had significantly higher odds of death (OR: 2.81, 95% CI: 2.38 to 3.32), rehospitalization (OR: 1.35, 95% CI: 1.26 to 1.44), and death or rehospitalization (OR: 1.54, 95% CI: 1.41 to 1.68). Patients classified as high risk had significantly higher odds of death (OR: 8.82, 95% CI: 7.58 to 10.26), rehospitalization (OR: 1.99, 95% CI: 1.86 to 2.13), and death or rehospitalization (OR: 2.65, 95% CI: 2.44 to 2.89).
Using over 30,000 patient records from a diverse and large group of hospitals, we developed simple tools that utilize data routinely collected during inpatient care to predict 30-day rehospitalization and mortality. We derived and validated these models in Medicare beneficiaries, the patients whose short-term outcomes are the shared focus of CMS and health systems. Importantly, we developed models using variables selected according to their potential availability in an EHR (24), an ideal user interface by which predictive analytics can be more deeply integrated with clinical workflow to offer point-of-care decision support. This repurposing of clinical data for research purposes and quality improvement creates a “learning health system” in which providers both generate and consume evidence as part of routine health care delivery (25).
Hospitals may invest considerable resources to mitigate post-discharge risks for their local patient population. Personnel-intensive disease management, more frequent outpatient follow-up, and home monitoring are among the potential interventions at a hospital’s disposal (5–7,9,10,26); but few analytical tools exist to inform their strategic deployment. When applied indiscriminately without regard to the patient’s risk of rehospitalization, the cost-effectiveness of these interventions may be reduced. Identifying patients at higher risk can help clinicians determine who may benefit from additional resources and interventions in the post-discharge period. Enabling point-of-care risk stratification can also help inpatient providers engage patients and their families to strengthen shared decision-making capabilities.
This study has several important findings. First, our findings reinforce findings from previous studies that early outcomes for both mortality and rehospitalization are poor among Medicare beneficiaries. Nearly 1 in 10 patients died within 30 days of admission and nearly 1 in 4 patients are readmitted within 30 days after discharge. Other studies using administrative claims have shown similarly poor outcomes among Medicare beneficiaries (3,12,27). Second, it demonstrates that prediction models with fair discriminative capacity can be developed from clinical data—demographics, vital signs, laboratory values—obtained as part of routine clinical care during an index hospitalization for HF. In contrast to administrative claims, which are typically coded post-discharge, this study provides the key clinical variables that can identify patients at risk for short-term adverse outcomes before they are discharged. Beyond ensuring clinical stability, such decision support can improve inpatient care by providing point-of-care predictive analytics to customize care at the time of discharge. Third, this study demonstrates that these models discriminate similarly well in patients with HF-REF and HF-PEF. Fourth, we found that a history of prior hospitalizations within 6 months of an admission has similar discriminatory power compared with existing 30-day rehospitalization models.
These models are comparable in their discriminatory power with previous models, including those currently espoused by CMS (30-day mortality: c-index 0.70; 30-day rehospitalization: c-index 0.60) (18,28). Similar to these models, the discriminatory power of the models in this study are better for mortality than rehospitalization, suggesting that important predictors of rehospitalization are either not captured in contemporary datasets or not incorporated in models to date. Other clinical, psychosocial, or economic factors likely play an important role in predicting the risk of rehospitalization within 30 days and need to be included in future studies. However, given that many of these variables may not be captured by large nationwide registries like GWTG-HF, they are also likely not available in EHRs. Models should be prospectively tested against clinical gestalt to understand whether they improve risk stratification for patients hospitalized with HF.
The patient population studied comprised Medicare fee-for-service beneficiaries enrolled in GWTG-HF and might not be representative of all patients hospitalized with HF. However, the outcomes are likely conservative given the voluntary participation of hospitals in this quality improvement registry. Analyses of OPTIMIZE-HF (Organized Program to Initiate Lifesaving Treatment in Hospitalized Patients with Heart Failure), a similar clinical registry, indicated that enrolled patients were reasonably representative of CMS patients (29). The data in this study are dependent on the quality of medical record documentation and chart abstraction. Study results can be influenced by differences in assessment, treatment, and documentation patterns at participating hospitals. There may be additional variables that were either not considered or were considered and rejected because of limited data availability in EHRs that could ultimately improve the risk discrimination if available in a sufficient portion of patients. Residual confounding, measured and unmeasured, may account for some or all of these observations. The correlations of these composite measures with outcomes beyond 30 days, or their association with other outcomes such as functional status and quality of life, were not evaluated.
In older patients hospitalized at GWTG participating institutions with both HF-PEF and HF-REF, the GWTG-HF predictive models stratify risk for 30-day outcomes using data routinely available in EHRs. Integration of these models at the point of care may allow efficient, appropriate risk stratification and may also be used to inform patients and families at risk for early morbidity or mortality. Further study is needed to test the effectiveness of implementing these clinical risk models within EHRs to inform clinical decision making.
The authors thank Elizabeth Cook for editorial contributions.
This work was supported by an award from the American Heart Association Pharmaceutical Roundtable and David and Stevie Spina. The GWTG-HF (Get With The Guidelines-Heart Failure) program is provided by the American Heart Association. The GWTG-HF program is supported in part by Medtronic, Ortho-McNeil, and the American Heart Association Pharmaceuticals Roundtable. The GWTG-HF was funded in the past by GlaxoSmithKline. Dr. Eapen has received funding from an American Heart Association Pharmaceutical Roundtable outcomes training grant (0875142N). Dr. Fonarow is a significant consultant to Novartis and Gembro Research; and a modest consultant to Medtronic. Dr. Curtis has received support from Johnson & Johnson and GlaxoSmithKline. Dr. Peterson was co-principal investigator of the Data Analytic Center for the AHA GWTG Program; and has received research support from Eli Lilly and Janssen Pharmaceuticals. Dr. Hernandez has received research support from Johnson & Johnson, Amylin, and Portola Pharmaceuticals; and was co-principal investigator of the Data Analytic Center for AHA GWTG Program. All other authors have reported that they have no relationships relevant to the contents of this paper to report. Wayne Levy, MD, served as Guest Editor for this article.
- Abbreviations and Acronyms
- blood urea nitrogen
- confidence interval
- Centers for Medicare and Medicaid Services
- electronic health record
- heart failure
- left ventricular ejection fraction
- odds ratio
- preserved ejection fraction
- reduced ejection fraction
- Received October 15, 2012.
- Revision received January 22, 2013.
- Accepted January 24, 2013.
- American College of Cardiology Foundation
- Krumholz H.M.,
- Merrill A.R.,
- Schone E.M.,
- et al.
- Medicare Payment Advisory Commission
- Krumholz H.M.,
- Amatruda J.,
- Smith G.L.,
- et al.
- Kasper E.K.,
- Gerstenblith G.,
- Hefter G.,
- et al.
- Bui A.L.,
- Fonarow G.C.
- Philbin E.F.,
- DiSalvo T.G.
- Levy W.C.,
- Mozaffarian D.,
- Linker D.T.,
- et al.
- Keenan P.S.,
- Normand S.L.,
- Lin Z.,
- et al.
- ↵Medicare.gov. 30-day death and readmission measures data. Available at: http://www.medicare.gov/HospitalCompare/Data/RCD/30-day-measures.aspx. Accessed January 17, 2013.
- American College of Cardiology Foundation/American Heart Association Task Force on Clinical Data Standards
- Olsen L.A.,
- Aisner D.,
- McGinnis J.M.
- Krumholz H.M.,
- Wang Y.,
- Mattera J.A.,
- et al.
- Curtis L.H.,
- Greiner M.A.,
- Hammill B.G.,
- et al.