On November 13, in a joint report of the American College of Cardiology and the American Heart Association (ACC/AHA), updated guidelines for the assessment of CVD risk were released. The report included a new prediction tool for primary prevention of CVD, which is prevention in persons with no history of vascular disease or high cholesterol. The tool predicts the absolute risk that a patient will experience a serious atherosclerotic event (chronic heart disease mortality, non-fatal myocardial infarction, and stroke) within the next 10 years, taking into account the patient’s age, total and HDL cholesterol, systolic blood pressure, diabetes status and smoking status. The purpose of the new online tool is to help clinicians to make treatment recommendations for patients who may need to lower their cholesterol. The ACC/AHA recommends that Americans between 40 and 79 years who have a calculated 10-year risk of 7.5% be put on statin therapy to keep their cholesterol under greater control.
The stakes of the new recommendations couldn’t be higher. If put into practice, it is estimated that as many as 45 million Americans could meet the 7.5% risk criterion for statin therapy. Shockingly, problems with the performance of the model have subsequently emerged, which show that its risk estimates are far too high, overestimating an individual’s risk by two-fold in some cases. That means, if the CVD risk calculator tells you your 10-year risk is 10%, it might actually be closer to 5%. The implications of the tool’s bias are profound. Left unaddressed, millions could end up inappropriately recommended for statin treatment.
Equally troubling is that the risk calculator’s developers knew about the problems of the model before it was endorsed by the ACC/AHA. In a 184-page online supplement to the new guidelines, the creators of the model describe the results of a series of investigations to test the performance of the model, what is known as a 'validation study'. Validation involves applying the model to an independent sample of patients whose outcomes are already known and comparing the predicted number of events to the actual number of events observed. When the CVD risk calculator was put to the test in several contemporary cohorts, the predicted number of serious atherosclerotic events was consistently and substantially greater than the actual number observed. The overprediction persisted even after the authors excluded individuals with a history of statin or lipid-lowering therapy, so the difference could not be attributed to medication use.
What could cause such a bias? It appears that the problem is age, namely, the age of the data used to construct the model. The cohorts underlying the model are over a generation old, with enrollment years beginning in late 1980s and early 1990s. Because major population changes have taken place in the past two decades, including decreased rates of smoking and increased awareness about the harms of high cholesterol, risk levels from this period are unlikely to represent the risk of contemporary 40-79 year-olds. This likelihood seems to have been born out by the discrepancies found in the validation, discrepancies downplayed by the authors who state that 'secular tends of declining [atherosclerotic cardiovascular disease] incidence may lead to an overestimation of the predicted risks' (p. 41).
The shortcomings of the new risk calculator would likely have remained buried in a supplemental document if Dr. Paul Ridker and Dr. Nancy Cook of Brigham and Women’s Hospital in Boston had not brought them to the public’s attention in a Lancet commentary published on November 19, which was announced by Gina Kolata in a story for the New York Times. Although the creators of the model should be commended for conducting a thorough validation of the CVD risk model and discussing the model’s limitations, these measures do not excuse their failure to bring the model’s problems more to the forefront; neither does it excuse the irresponsible decision to endorse a prediction tool that has not been shown to be reliable.
The authors conclude their discussion of the validation study in the supplemental report with the statement: 'Secular tends in incidence also present challenges for risk prediction in which quantitative assessment for risk will continue to be an evolving process'. In other words, population changes make long-term CVD risk prediction hard, and we will try to do better in the future. This response might be acceptable if this were merely an academic exercise but not when treatment choices are on the line. Because the CVD risk model is intended to guide who will and will not get statin therapy, a higher standard is required.
Risk calculators can be valuable resources for making medical decisions that are more evidence-based and better tailored to a patient’s specific needs. But, just as new drugs have to be demonstrated to be safe and effective, calculators that are going to be used by patients and doctors need to be proven to be predictive and reliable. To ensure that calculators that are being put into practice are trustworthy, validation studies need to become the phase III trials for new clinical prediction tools and their results should be treated with equal gravity. A continued lack of such standards could put us all at risk.
