Re: Why Most Published Research Findings Are False

From: David C Campbell <amblema@bama.ua.edu>
Date: Tue Sep 06 2005 - 18:26:16 EDT

One major problem with the article is the assumption that all
scientific papers use the statistical methods discussed in the paper.
Many are more descriptive. For example, my papers on the fossil
mollusks of the Carolinas list the species found and provide
descriptions of some of them. Although there may be errors in my
identifications, this does not fall under the issues raised by that
article.

Others papers use different statistical methods (with their own
weaknesses). Although 5% is a classical cutoff, the number of papers
that actually have a 5% error (as opposed to something rather less than
5%) is relatively small.

Although it is true that the statistics are often a weak point for
scientific papers, this does not necessarily totally invalidate the
papers that use them. Furthermore, the papers that use a 5% (or other,
usually smaller) cutoff in the way envisioned in the summary will give
the percent probability, so the reader can judge.

Ironically, a basic claim of the article itself involves bad
statistics. Use of the 95% confidence level does not mean that 1 in 20
papers that use it are wrong, on average. It means that each result
that has a 95% support level individually has a 1 in 20 chance of being
wrong. The probability that x independent items, each with 95%
probability of being correct, are all correct, is (19/20)^x. This is
less than 50% when x=14. However, it never reaches zero. Thus, you
cannot be sure that any of them are incorrect, but there is a greater
than 50% chance that at least one out of 14 samples is incorrect.

Furthermore, additional studies affect the statistics. The odds that
both of two results with 95% confidence are wrong is only 0.28%.
Assuming that all of the studies that point to global warming as a
serious problem are wrong, for example, is unreasonable, even though
there are some people who make unsupported claims about the magnitude
of imminent change.

A serious problem in the way the article is interpreted by some
(refering to the responses posted on the website with the article) is
that it serves as an excuse to ignore scientific papers that you don't
like. In fact, it is the papers that support what you think that
require the greatest scrutiny, because the natural tendency is to
readily accept what sounds good.

Finally, I would note that the low funding for most science makes the
accusation of making up results for financial gain a bit dubious. In
particular, denying global warming has obvious financial benefits,
whereas accepting it as a problem that needs serious attention has
financial costs. That does not prove which is right, but it does point
to a definite risk of bias or pressure.

----------------------------------------
Dr. David Campbell
425 Scientific Collections
University of Alabama, Box 870345
Tuscaloosa AL 35487
"James gave the huffle of a snail in
danger But no one heard him at all" A.
A. Milne
Received on Tue Sep 6 18:27:46 2005

This archive was generated by hypermail 2.1.8 : Tue Sep 06 2005 - 18:27:46 EDT