Believe the Results, Not the Conclusion

Early in my writing career I was fortunate to be able to spend three hours interviewing Linus Pauling, the only person in history to win two unshared Nobel Prizes. One of many things I learned during that interview process that has stayed with me ever since has to do with interpretation of scientific results.

Pauling was roundly criticized, in his later years, for his controversial stance on vitamin C. (He came to believe that vitamin C could not only prevent and cure the common cold but protect people from more serious diseases as well.) I interviewed Pauling at length on the subject. He told me that when he first began researching the role of vitamin C in protecting against the common cold, he was struck by how many papers he encountered in the scientific literature that showed a divergence between the results obtained and the conclusions drawn. Authors tended to fashion their conclusions around their own belief biases as much as around the actual data they reported. Pauling found studies in which the data clearly showed a positive protective effect for vitamin C, but because the effect was judged insignificant by the studies' authors, it wasn't discussed in the conclusion (nor the abstract, usually) of the paper in question.

What's so insidious about this is that other researchers who cite a given study will tend to quote from the conclusions section of a study and thereby propagate any biases in the conclusion.

I've seen this myself, over and over again. Propagation of incorrect conclusions has played an important role in, for example, the misplaced belief in the greater readability of serif fonts over sans-serif fonts. (See my earlier post on "The Serif Readability Myth.") Another example that comes readily to mind is the National Cancer Institute report that is so often cited to show that low-tar cigarets are just as harmful as regular cigarets: "Risks Associated With Smoking Cigarettes With Low Machine-Measured Yields of Tar and Nicotine" (National Cancer Institute; 2001. Smoking and Tobacco Control Monograph 13). Chapter 4 of this monograph summarizes a great deal of convincing evidence that today's filtered low-tar cigarets are much less toxic than the unfiltered high-tar cigarets of 60 years ago. And yet the monograph is cited as showing the opposite.

I bring all this up because tomorrow I will be presenting a lengthy blog on a controversial topic, and the support for it rests on a variety of studies in which authors attempt to show the opposite of what their own data shows.

The bottom line, in any case, is simple. Read studies carefully to see what the results are. Don't just read the Abstract and the Conclusion. You may very well be misled.