Misunderstanding science

I just received a publication for coaches with an article on "how to read a research paper." Although well-intentioned and containing some good points, the article also shows how little people know about science and how distorted their understanding really is. I have often remarked that even practicing scientists -or I should say card-carrying scientists, if that is what one can call PhD's- promote ideas that are very unscientific indeed. Their track record is often as poor as that of the general public.

There are erroneous statements throughout the article I mentioned, but here is the creme de la creme. First, about "proving," the author states, "science does not really "prove" anything: what science does is to find the truth." As I have remarked many times before, equating science with truth is pervasive in Western societies and it is wrong. What science really does is build models and prove that those models are internally consistent and work to approximate the truth within certain boundaries. 

Newtonian mechanics is a model that works very well under normal circumstances of every day life. Within those boundaries, it approximates the "truth" to within several decimal places. It no longer functions adequately when speeds are approximating the speed of light. Similarly, the Ptolemian model -with the earth in the center- is useful for everyday navigation on the high seas and for those who find themselves with dead batteries in the GPS. It is sorely inadequate for space applications.

Another great example is the "body of evidence" argument. Here the author basically argues that a single study should never be used to overturn a body of evidence, meaning -in this case- a number of previous studies. This idea, that science is about democratically voting what is right and what is wrong, is also very pervasive. It is very wrong too. Even a single observation can invalidate centuries of consensus. Science is not democratic in that sense.

While it is true that practitioners in a scientific discipline hold a consensus -what Kuhn called a paradigm- and even interpret new findings within that context, that does not mean that such a consensus is in any way "scientific." As a matter of fact, many historians of science have shown that precisely such a consensus often prevents new breakthroughs ("revolutions") from happening sooner. The consensus is a modus operandi of the community, but it is not part of the scientific method. 

The third kicker is the assumption that publication in "peer reviewed journals" means something is scientifically proven. This too is deeply ingrained. It is one area where the scientific community is highly complicit and insincere. This loathsome behavior is endorsed financially as well and scientists are often awarded jobs and promotions based on the number of peer-reviewed publications they have. Unfortunately, the average peer-reviewed publication is no better and no more valuable than the average subprime mortgage.

Nothing should be called scientifically proven until it has been duplicated by at least one independent outsider. No amount of peer-review can substitute for a single, well executed experimental confirmation by another researcher.

There are many more problems with the article, but I will limit my review to just one more, very common misunderstanding. This one has to do with statistics. Once again the author can be forgiven because the vast majority of "card-carrying" scientists do not understand statistics either. It is probably fair to say that statistics is misused in about 3/4 of the peer reviewed papers. The numbers are even higher in biology, medicine, and related disciplines.

The argument made here is about statistical significance and it shows how easily people can be misled. The author argues that sometimes, an improvement of 1% can make the difference between a gold medal and no medal. He shows that in some disciplines, less than 1% separates the fourth placed athlete from the winner. No problem here.

However, then he goes on to equate that 1% difference with a 1% difference between experimental and control groups in a study. In such cases, 1% may not be enough to reach significance. What that means is the 1% is very likely due to chance. I.e. there is no difference. That is not how the article reads. It claims the 1% is real but somehow ignored because researchers think it is too small to matter.

He says:"In the real world of coaching, one has to look beyond the issue of chance and look at the actual effect. For instance, if an intervention can increase performance by 1%, it might be statistically unimportant. However, in real life a 1% increase could mean victory."

How is that for mixing apples and oranges? And believe me, it is an argument I have seen in many times in many places. If the 1% is real then surely it will matter in some circumstances. The problem here is that the study where the 1% difference occurred is not powered to show that that 1% is for real. The 1% is likely a fluke. What we need in such cases is not "expert judgement" to resolve an issue. In this case, there is no information to base an opinion on.

What we need is another study. Our current study simply does not have the information we seek. 

Science II

Let's assume that the observations have been repeated many times and that they can be trusted. Let's further assume that we have formulated a plausible hypothesis that can explain these observations. Is that enough to attach the label of scientific discovery? It is not. To validate we need to do a series of experiments. We cannot rely on after-the-fact, or post-hoc explanations, however plausible they may seem. We need a predictive test. We need to elicit the event and see it unfolds the way we predict. We use the hypothesis to predict what will happen. If we are right our prediction should be very close to reality. How close? Close to within the limits of measurement error of our best possible measurement capability.

You may have heard this situation described another way. You may have heard we should have a null hypothesis. The null hypothesis states nothing will happen. All our observations were due to random chance only. When we set out to do an experiment, we try to disprove the null hypothesis. We show that the probability the events happened by chance alone is very small. Therefore we can accept the alternative hypothesis. This is the way statisticians like to describe the situation.

In any case, if you think we are done now, think again. We have carefully observed some new phenomena. We have hypothesized a cause for these observations that is both parsimonious and plausible. We have done repeated experiments in carefully controlled conditions to show that our predictions are correct to within measurement error. We have documented our findings and written a paper. The paper has been submitted to a scientific journal. There it was reviewed by a panel of peers and published. Surely, we can now claim a scientific discovery has been made?

If you said yes, let me assure you that you are not alone. As a matter of fact you are in good company. Many -way too many- scientists would agree with you. FDA would agree with you. Most universities and grant agencies agree with you. The whole American and Western European value system agrees with you. The whole publish-or-perish culture agrees with you. The money people agree with you.

But you are nonetheless wrong. Nothing has been proven. Not until another independent party is able to repeat the experiments and confirm the results. Many "important" scientific papers, written by eminent scientists, employed at the world's most prestigious institutions, reviewed by panels of equally well-known and eminent peers, and published in journals of the highest professional standing, have later been withdrawn because nobody could replicate the results. In some cases, deliberate fraud was the reason, but that is far from always the case. Scientists do make mistakes. They are also susceptible to wishful thinking as the rest of us are. The issue is more problematic in life sciences.

It is actually extremely likely that many "scientific discoveries," thus announced are false. In most cases, replication by an outside party is never done, either because it is too time-consuming, too costly, or simply because nobody is too interested in doing so. There is little to be gained by repeating other people's work. It is a thankless job. Furthermore, so many discoveries evoke so little outside attention that nobody really bothers to replicate them. Even in cases that were later recalled, it often took years before the withdrawal happened. And it often happened quietly and may have gone unnoticed.

Extensive reviews by mathematicians and statisticians have found significant errors in many peer reviewed publications. This is especially true in medical and health related areas where the errors are often significant enough to invalidate the results. Yet such cases almost never get recalled. Recently, the Wall Street Journal reported that medical scholar Dr. John Ioannidis, an epidemiologist documented how thousands of peer-reviewed research papers every year are seriously flawed. His conclusions were published in the Journal of the American Medical Association (JAMA) in 2007. He said: "The hotter the field of research, the more likely its published findings should be viewed skeptically. A new claim about a research finding is more likely to be false than true."

Caveat emptor!