Author + information
- Received April 17, 2015
- Revision received July 20, 2015
- Accepted July 20, 2015
- Published online January 1, 2016.
- Harlan M. Krumholz, MD, SM∗,†,‡,§∗ (, )
- Sarwat I. Chaudhry, MD‖,
- John A. Spertus, MD, MPH¶,
- Jennifer A. Mattera, MPH, DrPH‡,
- Beth Hodshon, JD, MPH, RN‖ and
- Jeph Herrin, PhD∗,#
- ∗Section of Cardiovascular Medicine, Department of Internal Medicine, Yale School of Medicine, New Haven, Connecticut
- †Robert Wood Johnson Foundation Clinical Scholars Program, Department of Internal Medicine, Yale School of Medicine, New Haven, Connecticut
- ‡Center for Outcomes Research and Evaluation, Yale-New Haven Hospital, New Haven, Connecticut
- §Department of Health Policy and Management, Yale School of Public Health, New Haven, Connecticut
- ‖Section of General Internal Medicine, Department of Internal Medicine, Yale School of Medicine, New Haven, Connecticut
- ¶Mid America Heart Institute, University of Missouri-Kansas City School of Medicine, Kansas City, Missouri
- #Health Research and Educational Trust, Chicago, Illinois
- ↵∗Reprint requests and correspondence:
Dr. Harlan Krumholz, Yale School of Medicine, 1 Church Street, Suite 200, New Haven, Connecticut 06510.
Objectives This study sought to determine whether a model that included self-reported socioeconomic, health status, and psychosocial characteristics obtained from patients recently discharged from hospitalizations for heart failure substantially improved 30-day readmission risk prediction compared with a model that incorporated only clinical and demographic factors.
Background Existing readmission risk models have poor discrimination and it is unknown whether they would be markedly improved by the inclusion of patient-reported information.
Methods As part of the Tele-HF (Telemonitoring to Improve Heart Failure Outcomes) trial, we conducted medical record abstraction and telephone interviews in a sample of 1,004 patients recently hospitalized for heart failure to obtain clinical, functional, and psychosocial information within 2 weeks of discharge. Candidate risk factors included 110 variables divided into 2 groups: demographic and clinical variables generally available from the medical record; and socioeconomic, health status, adherence, and psychosocial variables from patient interview.
Results The 30-day readmission rate was 17.1%. Using the 3-level risk score derived from the restricted medical record variables, patients with a score of 0 (no risk factors) had a readmission rate of 10.9% (95% confidence interval [CI]: 8.2% to 14.2%), and patients with a score of 2 (all risk factors) had a readmission rate of 32.1% (95% CI: 22.4% to 43.2%), a C-statistic of 0.62. Using the 5-level risk score derived from all variables, patients with a score of 0 (no risk factors) had a readmission rate of 9.6% (95% CI: 6.1% to 14.2%), and patients with a score of 4 (all risk factors) had a readmission rate of 55.0% (95% CI: 31.5% to 76.9%), a C-statistic of 0.65.
Conclusions Self-reported socioeconomic, health status, adherence, and psychosocial variables are not dominant factors in predicting readmission risk for patients with heart failure. Patient-reported information improved model discrimination and extended the predicted ranges of readmission rates, but the model performance remained poor. (Telemonitoring to Improve Heart Failure Outcomes [Tele-HF]; NCT00303212)
Preventing readmissions after heart failure hospitalizations is a national priority, but the risk of readmission is difficult to predict. In a survey of readmission risk scores, Kansagara et al. (1) reported that most risk models have poor discrimination and predictive ability. For the model that is publicly reported by the Centers for Medicare & Medicaid Services and that is part of the Hospital Readmissions Reduction Program, the discrimination using administrative claims as well as using medical records was less than 0.70 (2).
A potential explanation for the poor discrimination of these models is that patient factors beyond clinical and basic demographic characteristics, which are the principal components of these models, may play an important role in readmission risk. Most models have not included information from patient interviews that could characterize information about their socioeconomic, health status, adherence, and psychosocial characteristics. Whether this information would markedly improve the model performance is not known.
Accordingly, we sought to determine whether a readmission risk model that incorporated information obtained from the patient, including clinical, socioeconomic, health status, and psychosocial characteristics, could improve risk prediction compared with a model that incorporated only clinical and demographic factors. We supplemented information available from the medical record at the time of discharge with information from a patient interview and used it to develop a risk score that could be compared with a model built only from data available at discharge.
Data for these analyses were derived from our published trial to assess the effect of telemonitoring on patients with heart failure (Tele-HF [Telemonitoring to Improve Heart Failure Outcomes]) (3,4). The primary outcome of Tele-HF was readmission or death from any cause within 180 days, and there were no differences between study arms (telemonitoring vs. usual care) in rates of readmission, death, or the combined endpoint of either death or readmission. Because there was also no difference in readmission rate at 30 days, the current analysis included patients from both arms combined of Tele-HF. Tele-HF enrolled 1,653 patients who had been hospitalized for heart failure in the previous 30 days (“index admission”) at 33 study sites in the United States. Exclusion criteria included <18 years of age; long-term nursing home residence; being a prison inmate; inability to participate in the study protocol, including irreversible medical conditions likely to affect 6-month survival, inability to stand on a scale, severe cognitive impairment (Folstein score <20) (5), and no access to telephone service; chronic hemodialysis; severe aortic or mitral valve heart disease; enrollment in another disease management study; and, because the primary outcome included all-cause hospitalization, plans for an inpatient cardiac procedure. In addition to Tele-HF exclusions, for this study we excluded patients who were not interviewed between 3 and 30 days post-discharge (n = 574; all patients except 1 had a baseline interview; this exclusion was due to interviews outside of the window established for this study) or who were readmitted between discharge and interview (n = 36) and, to ensure that patients could be scored, those who were missing >15 of the 110 candidate variables (n = 39). The final sample for this study included 1,004 participants. The Human Investigation Committee at the Yale School of Medicine approved the study.
We obtained baseline data through medical record review and patient interview. Site coordinators abstracted medical records for clinical information. The Coordinating Center at Yale University sought to conduct interviews with patients to obtain clinical, functional, and psychosocial information. The median time from discharge to the interview was 12 days (interquartile range: 6 to 19 days).
The outcome was hospital readmission for any cause within 30 days after the interview. Readmission was assessed through medical record review, patient interviews conducted at 3 and 6 months post-enrollment, and direct contact with area hospitals, including the index admission hospital. We used these 3 sources to identify discrepancies concerning readmission status or date and resolved them by contacting the relevant hospitals.
We ascertained mortality status for enrolled patients after the conclusion of the 180-day follow-up period. For patients who did not have a record of death in the medical chart and who were not able to be contacted directly for the follow-up survey after 180 days, we determined vital status by searching the Social Security Death Index, contacting other residents of the patients’ households, and searching online obituaries for patients of the same name and date of birth in the same city. We used date of death to censor patients in time-to-readmission analyses; all surviving patients were censored at 30 days.
Tele-HF included collection of several hundred clinical, demographic, treatment, and psychosocial data elements for each patient, as described previously (3). For scales that were made up of multiple items (e.g., the Kansas City Cardiomyopathy Questionnaire [KCCQ]) (6), we included individual items rather than summary scores and, as a secondary analysis, replicated the analyses using the full scales. We further excluded variables that were missing in >20% of the study sample (income category, number of previous admissions for heart failure, brain natriuretic peptide, physician follow-up scheduled, and 8 of the KCCQ items). The remaining set of candidate risk factors included 110 variables (Online Appendix). These risk factors were divided into 2 groups: demographic and clinical variables that are generally available from the medical record; and socioeconomic, health status, and psychosocial variables that are not generally available but might improve the predictive power of a risk model and be collectable, if clinically important.
We summarized the characteristics of the included and excluded patients and compared the 2 groups using chi-square tests of independence. We next sought to develop the most parsimonious model of the highest predictive value from the available patient variables; first, using only the demographic and clinical variables, and then using all available patient variables. For each of the 110 included variables, we estimated a single Cox proportional hazards model with time-to-readmission as the outcome, censored for death. We used these results to collapse multi-category responses into fewer categories, where appropriate, on the basis of frequency of the response, the face validity of a combination, and similarity of the association with the outcome.
Then, to reduce the resulting set of variables to a subset that was most predictive of 30-day readmission, we used a random forest (RF) algorithm (7,8). In an RF algorithm, an iterative process involving random selection is used to assign weights to each variable considered. First, a random bootstrap sample is drawn from the full set of observations; then, random subsets of 10 variables are drawn and compared on some metric. In our case, we used a Cox proportional hazards model with time-to-readmission as the outcome and assigned a score to each variable according to the standardized effect size. At each step, the best-scored variable moved on to the next stage, until a final set of weights was calculated for each variable. This was repeated over random bootstrap samples, and the weight for each variable was averaged over all random samples to produce an importance weight (IW). The advantages of the RF algorithm include: the IW assigned a variable by RF is not sensitive to correlation or interaction with other variables; many more variables can be scored using RF than can be assessed using multivariable or stepwise regression techniques; the RF algorithm incorporates split-sample validation at each step; and the random sample and random variable selection provide a robust treatment of missing data.
To assign an IW to each variable, we used a version of RF known as a random survival forest (RSF) algorithm (9). For each random subset of variables, a Cox proportional hazards model is estimated, with time-to-readmission as the outcome and censoring for death at 30 days. Weight is then determined by the absolute magnitude of the coefficient from the regression model (9). For our analysis, we selected 10 random variables at each step and used multiple imputation with 20 imputations to account for missing data in each Cox regression; this was repeated for 2,000 randomly selected samples and variables.
The result of the RSF analysis was a relative IW for each variable under consideration, reported as a percentage of the IW of the most important variable. For further consideration, we retained variables with relative IWs of at least 20%, indicating that they were at least one-fifth as important in predicting 30-day readmission as the most important variable. Because importance weights are calculated independently of each other, we further reduced this set of variables by applying forward stepwise selection to a Cox regression model, including at each step the variable with the greatest t value (most significant) as long as the level of significance was <5%. For stepwise selection, we restricted to only those patients with no missing data for the retained variables. Stepwise regression is known to produce over-narrow confidence intervals and artificially small p values (10,11) and applying stepwise regression applied after RSF may furthermore bias the p values up or down. For this reason, although we used the p values to identify predictors, we caution against using them to make inferences. Using the final set of variables, we estimated a Cox regression model using multiple imputation to account for missing values (12). Finally, to construct the score, we assigned for each of the final risk factors a number of points consistent with the magnitude of the corresponding hazard ratio from this final model. We replicated the entire RSF analysis, stepwise selection, final model, and score construction using: 1) only demographic and clinical risk factors available on hospital discharge; and 2) the set of demographic and clinical risk factors plus all additional psychometric and socioeconomic measures. As a secondary analysis, we replicated the analysis using psychometric scales rather than individual items.
We evaluated each of the 2 risk scores by reporting the observed 30-day readmission rate for each value of the risk score and calculating the C-statistic for each. To assess whether the probability of readmission increased with increasing risk score, we performed a test for trend. We compared any nested models by calculating the integrated discrimination improvement (13). Finally, because these data were for patients enrolled in a trial, we compared the final scores by intervention group using a rank sum test. We also created a score for those individuals who had interviews within 2 weeks.
We performed the analyses using R version 3.0.1 software (9,14) and Stata version 13.1 software (StataCorp, College Station, Texas).
Description of the study sample
There were 1,653 patients enrolled in the study, 574 of which were not interviewed between 3 and 30 days post-discharge; 36 were readmitted before their interview; and 39 were missing more than 15 of the 110 variables, leaving 1,004 patients in the sample. The included patients were similar to the excluded patients (Table 1) with the exception that the included patients were older (p < 0.001). Rate of readmission within 30 days of baseline interview did not differ (p = 0.09). The mean age of the group was 62 years, with 341 (34.0%) younger than age 55 years. The sample had just over 41% women, and almost 40% were African-American (Table 1). The majority of the patients had New York Heart Association functional class II or III heart failure on admission and approximately 70% had a left ventricular ejection fraction <40%. Comorbidities were common; three-quarters of the subjects had hypertension, and nearly one-half had diabetes mellitus. The 30-day mortality rate from the time of the interview was 1.7%, and the 30-day readmission rate from the time of the interview was 17.1%.
Of the final set of 110 variables considered for potential inclusion in the risk model, 27 variables were classified as demographic or clinical (Online Appendix). After applying the RSF algorithm to the set of 27 variables for the 1,004 patients, 5 variables had a relative importance of at least 20% (Table 2). Forward stepwise Cox regression using these variables found that only 1 obtained at the index admission (blood urea nitrogen [BUN] level) had an independent effect on readmission with a significance level of p <0.05 (Table 3). Repeating this process using all 110 variables identified 7 with a relative importance of at least 20%; forward stepwise Cox regression retained 3 of these: BUN, reported swelling (KCCQ-3), and reported shortness of breath (Tables 3 and 4).
Considering the magnitude of the hazard ratios in Table 4, we assigned each patient 1 point for reporting that his/her health was an economic burden; reporting swelling in the last 2 weeks; reporting health status of “poor”; systolic blood pressure ≤90 mm Hg; and BUN >20. We assigned an additional point to patients with BUN >50. To reflect real-world applications in which not all information might be available, we assigned 0 to a risk factor that was either negative or missing.
Table 5 illustrates the 30-day readmission rate for each category of risk score derived from each set of risk factors. Using the risk score derived from the restricted set of commonly available variables, patients with a score of 0 (no risk factors) had a readmission rate of 10.9% (95% confidence interval [CI]: 8.2% to 14.2%) whereas patients with a score of 2 had a readmission rate of 32.1% (95% CI: 22.4% to 43.2%) with a C-statistic of 0.62. In comparison, using the risk score derived from all variables, patients with a score of 0 (no risk factors) had a readmission rate of 9.6% (95% CI: 6.1% to 14.2%) and patients with a score of 4 had a readmission rate of 55.0% (95% CI: 31.5% to 76.9%) with a C-statistic of 0.64. The test for trend found a positive trend for both risk scores (p <0.001).
Our principal finding was that even with the inclusion of a number of patient-centered variables obtained shortly after admission, there was only minor improvement in the discrimination of a risk model to predict 30-day readmissions after a post-discharge interview following a heart failure hospitalization. Although our potential predictors were much more extensive than those used in previous studies and our outcomes were validated, we were unable to develop models with high discrimination. Our results reveal that the limitations in predicting readmission do not stem from not having information about the patient’s symptoms, health status, psychosocial characteristics, access to health care, or economic status.
The discrimination in our model that was on the basis of all available variables is much lower than what has been achieved in mortality models. For example, our discrimination was much lower than that reported by Lee et al. (15), who developed a mortality model for patients hospitalized with heart failure that was derived from basic demographic and detailed clinical variables and had a C-statistic of 0.80 for 30-day mortality. Previous reviews of readmission models for patients hospitalized with heart failure have reported discrimination performance that is comparable to that of the 2 models presented in this study (1,16). Our previous efforts with basic demographic and detailed clinical data yielded similar results even though we used different methods (2,17).
The explanation for the poor discrimination of the models is not known. Unmeasured factors related to health system quality may play a prominent role, as many system interventions have been shown to reduce readmission and gaps in the quality of transitional care are common (18–20). Discharged patients may have an acquired, transient syndrome of generalized risk, which is not represented well by the characteristics that we included and may depend more on the allostatic stress experienced during hospitalization (21). The pronounced variation in the causes of readmission suggests that the severity of the condition leading to the hospitalization is not the only factor that influences the risk of readmission (22). The inherent propensity of a system to admit patients, which is not incorporated into these models, might be the dominant influence (23) and bed supply, which may be a mediator of the propensity for admission, may also play a role (24). The inclusion criteria of first the randomized controlled trial and then of this study likely resulted in a more homogeneous sample than most that are typically used to develop risk models; if so, then there would be less variation in risk factors and outcomes, and consequently reduced discrimination. However, it is worth noting that the study population included almost 40% African Americans and 10% other non-white groups and had substantial diversity with respect to socioeconomic status. We enrolled patients from 33 sites across the country and our event rates are similar to nationally reported rates. Despite the breadth of variables that we included, other unidentified patient-level variables, such as the quality of the discharge summary, may be responsible for the readmission risk as has been recently reported (25,26).
Our model did not perform as well as a single-center model, developed by Amarasingham et al. (27), which used data from an electronic health record but not from patient interviews. The model included non-clinical factors such as number of home address changes and missed clinic visits. Their discrimination, at 0.72, was higher with these variables but still not as high as mortality models. Their model may be conveying information about utilization behavior and barriers to health care access—or may also carry quality of care information.
In our model on the basis of all the variables, self-reported lower extremity swelling and health status were identified as important predictors. The reason why these variables were more important than heart failure severity is not clear. Because all patients had decompensated heart failure requiring hospitalization, it may be that severity of heart failure was not a discriminating risk among the sample. Interestingly, age, race, and other prominent socioeconomic variables were not sufficiently predictive to be included in the models, including reported medication adherence. Our study sample included a diverse range of patients and had good representation by age, race, and socioeconomic status. Our findings suggest that these socioeconomic variables do not carry much weight in predicting readmission when viewed with other detailed information about the patient.
One strength of our study is the novel RSF algorithm applied to avoid the known bias in stepwise and other automated variable selection processes, and validation of the final subset of variables selected from the large number of variables collected. This method, robust to the presence of nonlinear effects and complex interactions, has been found to produce highly predictive models (28,29).
Nevertheless, our findings should be interpreted in the context of several potential limitations. The sample was derived from a clinical trial population consisting of individuals who agreed to participate and who may be more adherent than patients who were not enrolled in a clinical trial. Although the score should be validated in different populations, the factors are consistent with what has been reported in other groups. The interview was conducted either during the hospitalization or shortly thereafter and the reference time was different across the sample. Nevertheless, we assessed the outcomes from the time of the patient-reported information so the patients were stratified at the point that they were providing feedback about themselves. There is also the limitation of sample size. In this cohort, a risk factor that is present in 30% of the patients would only be detectable in bivariate analysis with 80% power if it elevated the risk of readmission by 7.5%. However, though smaller effects may be clinically meaningful, it is arguable that very small effects would not be of interest in a prognostic tool.
We failed to demonstrate that expanded demographic and patient-reported information could markedly improve the performance of readmission risk models. The patient factors related to health and demographics seem inadequate and there is a need for further understanding of the factors that dominantly influence readmission risk. These factors may include health system quality of care, hospitalization stress, and propensity to admit.
COMPETENCY IN MEDICAL KNOWLEDGE: Readmission risk for patients is difficult to predict from demographic, clinical, and patient self-reported information. After hospitalization, clinicians should be aware that the risk of readmission is high and it is difficult to stratify this risk further with conventionally available data.
TRANSLATIONAL OUTLOOK 1: In practice, there is a need to recognize that risk-stratification of patients for their risk of readmission is challenging. Even the lowest risk patients have a substantial risk.
TRANSLATIONAL OUTLOOK 2: Clinicians should recognize the limitations of the current readmission models and appreciate that there are likely unmeasured factors that may be providing a strong influence on patient recovery.
For supplemental tables please see the online version of this article.
Beth Hodshon was affiliated with the Section of Cardiovascular Medicine, Department of Internal Medicine, Yale School of Medicine, New Haven, Connecticut, during the time the work was conducted.
This work was supported by the Center for Cardiovascular Outcomes Research at Yale University grant U01 HL105270-05 and Telemonitoring to Improve Heart Failure Outcomes (Tele-HF) grant R01 HL080228 (from the National Heart, Lung, and Blood Institute). Dr. Krumholz has research agreements with Medtronic and Johnson & Johnson (Janssen), through Yale University, to develop methods of clinical trial data sharing and chairs a cardiac scientific advisory board for UnitedHealth. Dr. Spertus holds the copyright to the Kansas City Cardiomyopathy Questionnaire, has an equity interest in Health Outcomes Sciences, and is a member of a cardiac scientific advisory board for UnitedHealth. The authors have reported that they have no relationships relevant to the contents of this paper to disclose.
- Abbreviations and Acronyms
- blood urea nitrogen
- importance weight
- Kansas City Cardiomyopathy Questionnaire
- random survival forest
- random forest
- Received April 17, 2015.
- Revision received July 20, 2015.
- Accepted July 20, 2015.
- American College of Cardiology Foundation
- Keenan P.S.,
- Normand S.L.,
- Lin Z.,
- et al.
- Green C.P.,
- Porter C.B.,
- Bresnahan D.R.,
- Spertus J.A.
- Derksen S.,
- Keselman H.J.
- R Core Team. A language and environment for statistical computing. Foundation for Statistical Computing, Vienna, Austria 2013. Available at:http://www.R-project.org/. Accessed July 17, 2015.
- Krumholz H.M.,
- Currie P.M.,
- Riegel B.,
- et al.
- Sochalski J.,
- Jaarsma T.,
- Krumholz H.M.,
- et al.
- Al-Damluji M.S.,
- Dzara K.,
- Hodshon B.,
- et al.
- Salim Al-Damluji M.,
- Dzara K.,
- Hodshon B.,
- et al.
- Hsich E.,
- Gorodeski E.Z.,
- Blackstone E.H.,
- Ishwaran H.,
- Lauer M.S.