False positives: When the health conscious adopt the latest advice

Jun 29, 2020

Sections Economics

A pair of articles in the New England Journal of Medicine suggested, in 1993, that taking vitamin E could prevent heart disease. Eleven years later, new evidence from a meta-analysis of randomized trials demonstrated that instead it might increase mortality. Both findings received widespread media attention, and in response, consumption of the supplement surged and then plunged. 

Brown’s Emily Oster used this case and similar phenomena involving vitamin D and other dietary recommendations to examine a research question related to measuring consumers’ responses to health findings. If only those who exercise, eat right, and don’t smoke adopt the latest recommendations, would that skew the results of subsequent research to magnify the positive effects, even if the suggested behavior turns out to be fruitless or even dangerous?

Oster analyzed behavioral responses to the vicissitudes of health advice. She used data from the National Health and Nutrition Examination Survey, a sample of children and adults in the United States that has been run in various forms since the 1960s, and from the Nurses’ Health Study, a survey funded by the National Institutes of Health of more than 100,000 nurses since 1976 that focuses on health behaviors, outcomes, and mortality. She also tapped into the Nielsen Homescan data, part of the Nielsen Datasets at Chicago Booth’s Kilts Center for Marketing, using household purchasing data from 2004 through 2016 to track health behaviors.

In the decade after the 1993 studies, Oster finds, use of vitamin E nearly quadrupled, and its association with health outcomes increased dramatically. Before 1993, taking vitamin E was associated with a 10 percent reduction in death risk. Between 1993 and 2003, this jumped to a 25 percent reduction in mortality. After 2004, the reduction in risk of death returned to 10 percent, and the number of people using vitamin E as a supplement returned to pre-1994 levels. 

She finds that as purchases of vitamin E increased between 1994 and 2004, the characteristics of the buyers changed as well. Before, those taking the vitamin were 0.7 percentage points less likely to smoke. After the studies suggested a heart benefit, those taking it were 4 percentage points less likely to smoke, a dramatic change. After 2004, when the findings about increased mortality came out, that number fell to 1.6 percentage points. During the years when vitamin E was thought to be beneficial, Oster finds a much stronger positive relationship between consumption of the supplement and education, income, exercise, and diet quality. 

Oster finds similar patterns with the popularity of vitamin D, which rose and fell from the beginning of the century to the 2010s, and recommendations of various diets that changed over time. Such patterns correspond to who decides to follow the latest advice, she argues. For example, those who limit their sugar consumption are also more likely to exercise, less likely to smoke, and more likely to have higher educations and incomes. These behaviors and demographics are strongly associated with lower body mass index and improved heart health. 

The findings have important ramifications for researchers, Oster argues. “These results suggest [variable] bias may be dynamic and, indeed, may respond to research findings,” she writes. “The data clearly points to the conclusion that current approaches in this literature . . . are very unlikely to yield causal results.”

Oster suggests researchers use more gold-standard randomized controlled trials, as expensive and difficult as they can be. Researchers can also create better research designs for nonrandomized data, taking into account biases caused by changes in selection patterns, Oster suggests. In the face of null results, which are too often ignored, “it seems even more crucial to consider the scope for improved research designs,” she argues.