More on the Harvard study on saturated versus polyunsaturated fats

This is a follow up on this post, which addressed the main argument put forth in a recent BBC article. The BBC article argued that people should replace saturated with polyunsaturated fats to reduce their risk of heart disease.

Let us take a look at the actual Harvard study itself (i.e., the study discussed in the BBC article). The Harvard study is linked here.

This post, by Stephan Guyenet, already pointed out several problems with the study. Stephan actually reviewed the studies used in the meta-analysis, and also some that were excluded in the meta-analysis and that he believes should have been included.

Here are a few other problems, in addition to the ones already pointed out by Stephan:

One thing that looks suspicious about this Harvard meta-analysis study is that they say that: “Statistical evidence for substantial between-study heterogeneity was not present (Q-statistic p = 0.13; I2 = 37%).”

A meta-analysis is a study that essentially summarizes, in a statistically sophisticated way, a bunch of other studies (the “sourced” studies). Too much between-study heterogeneity (i.e., widely disparate results among sourced studies) is undesirable, because it can bias the results.

The problem is similar to that of trying to summarize net worth figures (e.g., by calculating their average) in a middle class neighborhood that happens to have a few billionaires living in it. The heterogeneity in wealth may lead to a wildly overestimated average.

Now, we know that p values go down with sample size, and are usually high with small samples unless the effect measured by the statistic is very strong, regardless of the statistic used.

Well, with a sample of only 8 studies, their p value (associated with the Q statistic) is close to being significant at the 0.05 level!

If this sample of sourced studies were a little higher (say, 20), there would be significant between-study heterogeneity, which would call the meta-analysis into question. This is a big problem, since a good meta-analysis is expected to include a large number of studies (e.g., greater than 100), and this one included only 8 studies.

Moreover, to the best of my knowledge, the Q statistic is not very reliable when used with small samples, due to its low power as a test of heterogeneity. This makes the p value reported even more problematic.

Finally, the sourced study with the largest sample (n = 9,057; thus possibly the most credible), indicated as “Minnesota CS” on Figure 2 of the Harvard study, found increased risk of heart disease associated with increased consumption of polyunsaturated fats and reduced consumption of saturated fats.

Reference:

Mozaffarian, D., Micha, R., & Wallace, S. (2010). Effects on Coronary Heart Disease of Increasing Polyunsaturated Fat in Place of Saturated Fat: A Systematic Review and Meta-Analysis of Randomized Controlled Trials. PLoS Med., 7(3): e1000252. doi: 10.1371/journal.pmed.1000252.