Machine Learning Prediction of Mortality and Hospitalization in Heart Failure with Preserved Ejection Fraction

Study population

TOPCAT was a multicenter, international, randomized, double-blind, placebo-controlled trial sponsored by the U.S. National Heart, Lung, and Blood Institute (NCT00094302), which randomized patients 50 years of age or older with at least 1 sign and at least 1 symptom of HF and a left ventricular ejection fraction (LVEF) of 45% or higher to receive spironolactone or placebo therapy (16). The trial enrolled patients from the United States, Canada, Brazil, Argentina, Russia, and Georgia, from 2006 through 2013, and assessed the incidence of the primary composite outcome of death from cardiovascular causes, aborted cardiac arrest, or hospitalization for HF exacerbation. For the current study, patients from Russia and Georgia were excluded due to regional discrepancies in the reported and actual use of spironolactone and improper application of the selection criteria during enrolment (17).

Outcomes

The outcomes of interest were all-cause mortality and HF hospitalization through 3 years of follow-up. All-cause mortality was defined as death due to any cause, and HF hospitalization was defined as an unexpected presentation to an acute care facility requiring overnight hospitalization with exacerbation of HF. HF as the cause of hospitalization was confirmed using a prespecified list of signs and symptoms present upon admission requiring treatment and was adjudicated by an independent endpoints committee. Kaplan-Meier survival plots were constructed to assess mortality and HF hospitalization in the study population.

Candidate variables

All baseline demographic and clinical data available from patients were used in addition to laboratory data, electrocardiography data, Kansas City Cardiomyopathy Questionnaire (KCCQ) scores (physical limitation score, symptom stability score, symptom frequency score, symptom burden score, total symptom score, self-efficacy score, quality of life score, social limitation score, overall summary score, and clinical summary score) to predict the outcomes of all-cause mortality and HF hospitalization.

Model development

To develop derivation and validation cohorts, a stratified, 5-fold cross-validation was used (Figure 1). The study population was divided randomly into 5 subsets with similar event rates. To form the derivation cohort, 4 subsets (80%) were combined, and the remaining subset (20%) was reserved as the validation set. This process was repeated 5 times for each outcome, such that every subset served as the validation set, thereby accounting for variability among patients and providing risk estimates for all cases. With low missing data, mean imputation was used; 30 variables had no missing data, 44 variables had <10% data missing, and the remainder (30 variables) had >10% missing data. Any variable with >50% missing data (18 of the 30 variables) was removed. A total of 86 variables were included in the models. The list of candidate variables along with their definitions are included in Online Table 1.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Analysis Flow for the Development and Evaluation of Models

TOPCAT = Treatment of Preserved Cardiac Function Heart Failure with an Aldosterone Antagonist trial.

Four commonly used machine learning methods and conventional logistic regression were used to train the models for assessing risk of mortality and HF hospitalization through 3 years of follow-up. The methods included logistic regression with a forward selection of variables, logistic regression with a lasso regularization for variable selection, random forest (RF), gradient descent boosting decision trees, and support vector machine (SVM). Various R packages (R software: R Core Team, Vienna, Austria) were used to conduct this analysis. The base GLM function was used for logistic regression. The glmnet package was used for logistic regression with lasso regularization (18). randomForest package was used for the RF model (19); xgboost was used for the gradient descent boosting (20); and e1071 (libSVM) software was used for the SVM (21).

Logistic regression was used with forward stepwise selection to choose variables with statistically significant results in a likelihood ratio test (p < 0.05). A forward stepwise selection through this ordered list was run to identify the mean incremental C-statistic improvements for each added variable (22). In high-dimension problems, backward selection techniques may be susceptible to greater noise (23). Forward selection has strong theoretical guarantees and excellent empiric behavior (24). Logistic regression with lasso regularization builds a similar logistic regression model, but it retains the variables using more stringent thresholds, thus helping to select a parsimonious, predictive subset of variables to train the logistic regression model. Pre-built default lambda parameters in glmnet package were used through 10-fold cross-validation (18). RF is a method by which a number of decision trees (tree-like graph of decisions and their possible consequences, including event outcomes) are built from the variable set. These decision trees were used to divide patients into similar subgroups by using the most important variables. The prediction is then generated using a “voting” scheme across all decision trees. Gradient descent boosting similarly picks variables across decision trees that best help predict correct outcomes in their training sets. Both RF and gradient descent boosting internally validate their selection of variables and cases within the training set. A total of 1,000 trees were used for RF and 100 trees for boosting (to avoid overfitting), with a learning rate of 0.1 and a maximum depth of 6 for each tree. Finally, SVM is a predictive method that tries to find a separable space between 2 classes in order to generate positive or negative predictions. The SVM in this study was given the entirety of a dataset and trained with 2 different kernels (or patterns to find the separation function), which were linear and radial-based functions. These kernels identify relationships of variables within the dataset by comparing linear separation between the outcomes of interest or a Gaussian separation between the outcomes of interest.

Model performance

Receiver operating characteristic (ROC) curves were used to estimate model discrimination by calculating the C-statistic or area under the curve (AUC). The best performing model assessed by highest AUC was chosen and analyzed further. The accuracy of probability of the best performing model was assessed using the Brier score, which is defined as the mean squared difference between the observed and predicted outcome. Brier scores range from 0 to 1.00, with 0 representing the best possible calibration. The 2 primary components decomposed from Brier score are reliability and resolution, which measure how close the prediction probabilities are to the true probabilities and how much the conditional probabilities differ from the prediction average, respectively. Calibration plots were used to plot the mean risk score relative to the observed outcome rate for a given decile of predicted risk. The prediction for every patient was plotted in order of their risk to assess the prediction distribution from the model.

Top predictors

The importance of each variable in a model was evaluated by using a variety of metrics. For logistic regression with forward selection, importance was ranked by ordering the variables that were selected. A standard statistical procedure for the ordering of variables was followed. At each iteration, 1 additional variable was added, and the maximum likelihood was measured by using a likelihood ratio test. For any variable with a p value < 0.05, the largest maximum likelihood variable was selected. This process was repeated until no variables had p significance <0.05. RF uses a mean decrease in accuracy to rank the importance of its variables. Boosting determines variable importance by the number of trees it appears in and the information gain that variable provides. The importance of each variable was evaluated in the best performing model. To avoid overfitting, model discrimination was evaluated through cross-validation, demonstrating a good fit by identifying a narrow confidence interval (25). Each training iteration can present a different order of variables; thus, variable importance was calculated from a model trained on all data. It was ensured that this model was representative of the important variables without overfitting based on the good fit demonstrated in the cross-validation. These selected variables were listed in the order of importance, and were evaluated by the incremental improvement of each variable in the 5-fold cross-validation data. This may overestimate the gain each variable provides, but the final values are similar to those obtained through cross-validation.

Sensitivity analyses

Sensitivity analyses were performed for all patients enrolled in the trial from the Americas. The models were developed on this population for the prediction of mortality and hospitalization, followed for the entirety of the study period. Variable importance was calculated, and the incremental improvement of each variable was evaluated in the 5-fold cross-validation. Furthermore, a 1-year prediction of mortality and HF hospitalization was assessed to find out how the models’ performance varied over different shorter follow-up times. All analyses were conducted using R version 3.4.0.