Author + information
- Tariq Ahmad, MD, MPH∗ (, )
- Jeffrey M. Testani, MD, MTR and
- Nihar R. Desai, MD, MPH
- Section of Cardiovascular Medicine, Yale New Haven Hospital and Yale University School of Medicine, New Haven, Connecticut
- ↵∗Reprint requests and correspondence:
Dr. Tariq Ahmad, Section of Cardiovascular Medicine, Yale University School of Medicine, 333 Cedar Street, New Haven, Connecticut 06510.
Everything should be made as simple as possible, but not simpler.
—Albert Einstein (1)The delivery of high-quality and patient-centered care relies on accurate risk prediction and phenotyping of disease (2). Cardiologists pride themselves on being particularly data-driven; appropriate management of patients with heart disease relies heavily on analysis of large clinical trials and registries rather than clinical gestalt and mechanistic feasibility. As a consequence, numerous promising therapeutic strategies have fallen by the wayside under the weight of contradictory data from large unbiased studies, and the bar for approval of new treatments is arguably higher than it is for other disease states (3). Despite this, contemporary clinical care and investigation continues to lag behind the needs of patients or providers because of a reliance on crude risk prediction models and traditional statistical methods that fail to capture disease complexity. To illustrate, stroke prediction for patients with atrial fibrillation is based on a handful of variables that comprise the CHA2DS2-VASc score; the predictive capabilities of this instrument fail to take into account the complexity of the stroke phenotype, and the dynamic interplay between multiple biological and clinical variables that occur on the patient level (4). Instead of acceding to such simplified constructs that are designed for ease of use at the bedside, there as a growing demand for use of increases in computing power to assist with more accuracy and precision in the diagnosis and treatment of heart disease.
“Big data” is a catch phrase applied to the combination of logarithmic increases in the quantity of available information on patients and the sophisticated tools available to analyze it—it is purported that if enough data points are known about an individual patient, advanced analytical methods could enable the delivery of higher quality individualized care (5). In an effort to show the feasibility of this approach, this issue of JACC: Heart Failure includes a thought-provoking article by biomedical engineers from Carnegie Mellon University who partnered with the advanced heart failure team at the University of Pittsburgh to use a machine learning approach—Bayesian network analysis—to predict a complex outcome (right ventricular failure) in a heterogeneous patient population (stage D heart failure undergoing left ventricular assist device implantation) (6). A Bayesian network is a data structure that maps out the conditional probability of patient variables in terms of nodes (independent variables) and edges (statistical associations between nodes) (7). The dynamic and nonlinear nature of Bayesian networks make them especially suited to the study of complex disease, in which accurate prediction of outcomes is predicated on dependence and independence among multiple variables, something that cannot be modeled using traditional regression methods. The investigators used 176 variables from 10,909 adult patients enrolled in the INTERMACS (Interagency Registry for Mechanically Assisted Circulatory Support) to predict 3 discreet clinical outcomes after left ventricular assist device (LVAD) placement: acute, early, and late right ventricular failure. They had two major findings: 1) their prediction models, which each included >30 pre-operative variables, showed an exquisite degree of sensitivity and specificity for all outcomes of interest (C-statistic >0.90, all); and 2) 2 established risk scores—the Right Ventricular Failure Risk Score (RVFRS) and the Drakos score—yielded results that were no better than a coin toss (C-statistic ∼0.50). The investigators acknowledged the difficulties in integrating this model into the clinical workflow and are in the process of creating a decision support tool that is interactive, easily accessible, and integrated into the electronic medical record.
This effort is a component of a larger project based out of Carnegie Mellon University named Cardiac Outcomes Risk Assessment that aims to be a clinical decision support tool designed for LVAD patients and their physicians (8). Although this is certainly a clinical question deserving of study, could these methods be applied to more widespread questions in heart failure and cardiology? For example, is there a role for using advanced analytical approaches to construct interactive decision tools for treatments such as coronary artery bypass surgery, implantable cardioverter-defibrillator placement, and anticoagulation for atrial fibrillation? Certainly, but in our view, the current enthusiasm for these promising approaches must be tempered by an understanding of their limitations and an explicit framework for future studies. This is a vital point because large medical data sets and machine learning algorithms have been available for decades, and have yielded thousands of papers, but almost none have contributed meaningfully to clinical care (9). Therefore, this paper provides a good opportunity to review some of the important questions and obstacles pertaining to the use of big data approaches on large clinical data sets (Figure 1).
What Variables Should Be Included in These Analyses?
The most robust of statistical methodologies cannot circumvent the fact that faulty, incomplete, and imprecise data will lead to nonsensical output: the “garbage in, garbage out” problem sometimes sardonically referred to as “garbage in, gospel out” when this limitation is not appropriately realized. Complex algorithms applied to large data sets can compound this issue by rapidly churning out information that is overwhelming and difficult to verify. This study provides a case in point by relying on INTERMACS registry data that are large (N =∼10,000) but saddled with a high number of missing variables and rudimentary phenotyping of individual patients. Moving forward, efforts should be made to collect more comprehensive and objective data on patients, such as biomarker levels, imaging parameters, and geospatial mapping, so that results are anchored in more concrete information. Furthermore, there is a need for consensus with regard to variable selection: 3 key papers on this topic, including 1 by our group, used divergent approaches for similar stated goals (10–12). In the absence of more clarity about this question, we run the risk of not translating these ideas into clinical practice.
Are There Statistical “Best Practices” for Analysis of Large Clinical Data Sets?
Considerable debate already exists about how traditional statistical methodologies should be used to address clinical questions; advanced analytical methodologies add another level of complexity to these discussions. Loghmanpour et al. (6) applied tree-augmented naive Bayesian classification to clinical data from the INTERMACS registry in an effort to predict right ventricle failure after LVAD placement. This is only 1 of several predictive data mining methods that could have been used for this purpose; others range from variations of logistic regression to neural networks. The investigators do not discuss in detail how their approach compares with that of other methodologies. The next phase of such applications of novel analytics to clinical databases will require clearer guidelines on the benefits and drawbacks of alternative approaches. (13).
What is the Role of Generalizability and Validation?
In a landmark study, Snyder et al. (14) at Stanford University analyzed a massive data set that combined genomic, transcriptomic, proteomic, metabolomic, and autoantibody profiles from him (a single individual) over a 14-month period, and showed his dynamic risk of disease. Although this was an extreme example of the potential of personalized medicine, the application of advanced analytical methods to enable individualized decision-making will surely become routine in the near future. In such a setting, patient heterogeneity would be magnified, and insights gained from 1 similar group of patients may not apply to a dissimilar population. An example of this is reflected in this paper. The investigators, not unexpectedly, showed that a machine-learning algorithm performed well for prediction of their outcome of interest. In contrast, other risk scores that were derived from single centers (RVFRS, University of Michigan Health System, Ann Arbor, Michigan, and the Drakos score, Intermountain Health System, Salt Lake City, Utah) performed poorly, with the accuracy of a coin toss. Similar findings have been noted elsewhere with the prediction of LVAD outcomes and are likely reflective of the significant differences in patient populations, environment, and practices among centers (15). In the future, greater benefit would result from creating local risk scores based on deep phenotyping of patients than trying to inadequately apply global risk scores on disparate populations.
In addition to challenges surrounding generalizability, the validation of published findings are an essential stride on the path toward clinical application. Beyond the need for investigators to demonstrate the worth of their algorithm on a training and test data set, there is an enormous need for transparency that centers on sharing of data and methodology. This will require a paradigm shift in the conduction and publishing of research, but this is necessary for independent verification of findings and improvements in coding. These practices have been embraced outside of medicine where data sharing and competitive challenges for prediction models are commonplace. Hopefully, ongoing synergy between computer engineers and cardiologists will catalyze the transition from “desktop to bedside.”
How Will Improved Risk Prediction Affect Clinical Practice?
At present, there are several highly effective risk scores for heart failure patients (e.g., Seattle Heart Failure Model), but none are used routinely to guide medical therapy or help with difficult and costly decisions, such as implantable cardioverter-defibrillator implantation (16). This divorce between risk prediction and therapeutics is in part due to a dearth of intervention studies that examine the effects of a treatment based on an objectively defined subgroup of patients in a randomized, controlled manner. Although this study presents us with an approach toward prediction of right ventricular failure, it remains unclear as to how this information might be clinically useful. Evaluating the myriad of possible therapeutic interventions in response to the information is not feasible using current approaches and will require innovative clinical trial designs.
In his autobiography, The Youngest Science: Notes of a Medicine-Watcher (17), renowned physician Dr. Lewis Thomas described the evolution of medicine during the turn of the 20th century from a hodgepodge of ineffective practices based on anecdotes to one rooted in scientific reductionism. Although this approach has yielded innumerable gains in our fight against disease, physicians have long realized that the complexity of human disease cannot be aptly recapitulated in the sterility of a research laboratory, nor taught in a lecture hall. Therefore, the well-trained clinician not only has a nuanced understanding of disease grounded in pathobiology but has enriched this with countless hours spent directly involved in the care of patients, and is mindful of the interconnected clinical observations that dictate the course of illness. This past turn of the century has been defined by a revolution in computing power that allows us to supplant 1-dimensional approaches to clinical research with those that account for the multiple components of perturbed pathways and regulatory networks that lead to illness. This work represents an initial but timely step toward a more intelligent and learning health care system that will require innovative bonds among patients, clinicians, data scientists, and health care systems. The ultimate goal is to make the delivery of health care simpler and better—where simplicity does not supersede precision.
The authors gratefully acknowledge Dr. Pooja Rao for her input in regards to this editorial.
↵∗ Editorials published in JACC: Heart Failure reflect the views of the authors and do not necessarily represent the views of JACC: Heart Failure or the American College of Cardiology.
The authors have reported that they have no relationships relevant to the contents of this paper to disclose.
- American College of Cardiology Foundation
- ↵Brainyquotes. Albert Einstein quotations. Available at: http://www.brainyquote.com/quotes/quotes/a/alberteins103652.html. Accessed June 22, 2016.
- Committee on a Framework for Development a New Taxonomy of Disease and Board on Life Sciences
- Fordyce C.B.,
- Roe M.T.,
- Ahmad T.,
- et al.
- January C.T.,
- Wann L.S.,
- Alpert J.S.,
- et al.
- Krumholz H.M.
- Loghmanpour N.A.,
- Kormos R.L.,
- Kanwar M.K.,
- Teuteberg J.J.,
- Murali S.,
- Antaki J.F.
- Loghmanpour N.A.,
- Druzdzel M.J.,
- Antaki J.F.
- Deo R.C.
- Shah S.J.,
- Katz D.H.,
- Selvaraj S.,
- et al.
- Ahmad T.,
- Pencina M.J.,
- Schulte P.J.,
- et al.
- Yancy C.W.,
- Jessup M.,
- Bozkurt B.,
- et al.
- ↵Thomas L. The youngest science: notes of a medicine-watcher. London: Penguin Books; 1995.