Science II

Let's assume that the observations have been repeated many times and that they can be trusted. Let's further assume that we have formulated a plausible hypothesis that can explain these observations. Is that enough to attach the label of scientific discovery? It is not. To validate we need to do a series of experiments. We cannot rely on after-the-fact, or post-hoc explanations, however plausible they may seem. We need a predictive test. We need to elicit the event and see it unfolds the way we predict. We use the hypothesis to predict what will happen. If we are right our prediction should be very close to reality. How close? Close to within the limits of measurement error of our best possible measurement capability.

You may have heard this situation described another way. You may have heard we should have a null hypothesis. The null hypothesis states nothing will happen. All our observations were due to random chance only. When we set out to do an experiment, we try to disprove the null hypothesis. We show that the probability the events happened by chance alone is very small. Therefore we can accept the alternative hypothesis. This is the way statisticians like to describe the situation.

In any case, if you think we are done now, think again. We have carefully observed some new phenomena. We have hypothesized a cause for these observations that is both parsimonious and plausible. We have done repeated experiments in carefully controlled conditions to show that our predictions are correct to within measurement error. We have documented our findings and written a paper. The paper has been submitted to a scientific journal. There it was reviewed by a panel of peers and published. Surely, we can now claim a scientific discovery has been made?

If you said yes, let me assure you that you are not alone. As a matter of fact you are in good company. Many -way too many- scientists would agree with you. FDA would agree with you. Most universities and grant agencies agree with you. The whole American and Western European value system agrees with you. The whole publish-or-perish culture agrees with you. The money people agree with you.

But you are nonetheless wrong. Nothing has been proven. Not until another independent party is able to repeat the experiments and confirm the results. Many "important" scientific papers, written by eminent scientists, employed at the world's most prestigious institutions, reviewed by panels of equally well-known and eminent peers, and published in journals of the highest professional standing, have later been withdrawn because nobody could replicate the results. In some cases, deliberate fraud was the reason, but that is far from always the case. Scientists do make mistakes. They are also susceptible to wishful thinking as the rest of us are. The issue is more problematic in life sciences.

It is actually extremely likely that many "scientific discoveries," thus announced are false. In most cases, replication by an outside party is never done, either because it is too time-consuming, too costly, or simply because nobody is too interested in doing so. There is little to be gained by repeating other people's work. It is a thankless job. Furthermore, so many discoveries evoke so little outside attention that nobody really bothers to replicate them. Even in cases that were later recalled, it often took years before the withdrawal happened. And it often happened quietly and may have gone unnoticed.

Extensive reviews by mathematicians and statisticians have found significant errors in many peer reviewed publications. This is especially true in medical and health related areas where the errors are often significant enough to invalidate the results. Yet such cases almost never get recalled. Recently, the Wall Street Journal reported that medical scholar Dr. John Ioannidis, an epidemiologist documented how thousands of peer-reviewed research papers every year are seriously flawed. His conclusions were published in the Journal of the American Medical Association (JAMA) in 2007. He said: "The hotter the field of research, the more likely its published findings should be viewed skeptically. A new claim about a research finding is more likely to be false than true."

Caveat emptor!