A Scientific Theory is Not Confirmed Merely Because It Seems to Make a Few Correct Predictions

In discussions of scientific theories, it is often argued that this or that result will confirm some scientific theory because such a result was predicted by that theory. But such reasoning is often mistaken. The fact that a theory may seem to make some correct predictions does not necessarily show that the theory is likely to be true.

Below are some of the reasons why this is true.

A theory can be a mixture of true and false assumptions, and correct predictions can be made by the true assumptions.

Theories are often a mixture of correct assumptions and mistaken assumptions. Correct assumptions in a theory may imply certain predictions, which may prove successful. But the theory may still contain incorrect assumptions, which did not imply the predictions that turned out true. The correct predictions only tend to confirm (perhaps to at least some degree) those parts of a theory that implied those correct predictions, not other assumptions that did not imply those predictions.

For example, some people advanced the theory in 2002 that the Bush administration had secretly orchestrated the September 11 attacks, to create a pretext for war because it wanted to invade Iraq. Perhaps some of those people then said in 2003 that their theory was confirmed, because the Bush administration really did invade Iraq in that year. But in this case we have a theory making two assumptions: (1) the assumption that the Bush administration orchestrated the September 11 attacks; (2) the assumption that the Bush administration wanted to attack Iraq. The invasion of Iraq in 2003 may tend to confirm the second of these assumptions, but not the first.

So when a particular scientific theory seems to be confirmed by some prediction that eventually matches observations, we need to ask: which parts of the theory tend to predict the prediction that matched observations? Only such parts – if any-- should be considered as having been put (possibly) in a favorable light by the observations.

Multiple theories may make a particular prediction, so a confirmation of the prediction may not really support a particular theory that makes that prediction.

It is not necessarily true that a confirmed prediction tends to show that the theory that predicted it is true, because there may be many other reasonable theories that make the same prediction. For example, let's imagine a person in 2007 arguing that sinister forces on Wall Street were trying to orchestrate a sharp economic downturn, so that they could make lots of money on certain types of stock market bets called puts (which increase in value when a stock goes down). In 2008 (when such an economic downturn occurred) such a person would no doubt say, “Look, we did have a sharp economic downturn; my theory is confirmed.” But such reasoning would be invalid, because the same sharp economic downturn was predicted by various other theories, such as the theory that a housing bubble would produce such an economic downturn, and the theory that too much consumer credit would produce an economic downturn.

It is too easy to selectively present data in a way that makes a theory's predictions look true, either by massaging the “observed data,” by massaging the “predicted data,” or by massaging both, either deliberately or through unrecognized bias.

The favorite device of a theory advocate is a “predicted versus actual” line graph. Here is a very simple example of this type of graph, with the blue line showing predicted results and the red line showing observed results:

This type of graph can be used to try to show that a particular theory is matching observations. But be distrustful when you see such a graph. Why? Because it is easy to cherry-pick either the data used as the “observed data” or the data used as the “predicted data,” or both.

This is particularly true in any case where the data points are not some simple thing (depending on one observation, as in the case above), but instead require some complicated summary of multiple observations. In such cases it is all too easy for a presenter to massage the data in a way that shows a theory in a favorable light. Given a choice of five different ways of showing the “observed results” (each using a different source of data, or a different way of summarizing the data), someone can choose whichever set of “observed results” is most in agreement with his theory.

Another way in which bias can be displayed is by massaging and cherry-picking the “predicted results” shown in a graph such as the one below. Theories often have multiple flavors, which vary because of a choice of parameters that can be used within the theory. In other words, the “predicted results” from a particular theory are often very fuzzy, rather like an electron probability cloud. A presenter can pick particular values within that fuzzy cloud that most closely match the “observed results,” and plot such values as the “predicted values” on a line graph. The result will show the theory in the most favorable light, but may be misleading. For a recent specific example of this type of cherry-picking, see this blog post.

Another way in which bias can be shown in matching observed results with predicted results is simply by choosing the start point and the end point of the data being graphed. For example, if I have a theory that bonds tend to out-perform stocks, I may use a start point of January 1, 2000 and an end point of Dec 1, 2008. That will show a huge advantage for investing in bonds as compared to investing in stocks. But a different start point and end point would tell a very different story. A similar technique can be used to try to show the likelihood of a particular scientific theory. A supporter can choose to graph whatever start point and end point shows the closest match between the theory and observations, even though different start points and end points on the line graph would show a much smaller degree of agreement.

Even if a theory is the only theory that predicts an observed phenomenon, that does not mean the theory is true, because there may be many possible theories not yet imagined that can explain the phenomenon.

One type of reasoning sometimes made is: x is the only theory that predicts the observed phenomenon y, so x must be true. But that does not follow. The human imagination is weak, and our ignorance is enormous. Almost any observed phenomenon can be explained in many different ways, but the puny human imagination may be able to think of only one or two of those ways. Back during the days of the Black Plague, the theory of “God's wrath” may have been the only theory that explained why so many people were dying, but it would have been wrong at that time to assume such a theory was correct on that basis.

With sufficient ingenuity, unbelievable theories can be contrived to make predictions that match observations.

Sometimes it is possible for a theory to make some correct predictions, even though the theory isn't plausible. One of the most famous examples is the Ptolemaic theory, a theory of the solar system. The theory held that the Earth was at the center of the solar system. To make such a theory match observations, the theory included a complex model of planetary motions, in which planets orbited in small orbits called epicycles that were part of much larger orbits. The predictions of the Ptolemaic theory seemed accurate for centuries, but the theory was quite false.

There are modern-day equivalents of the Ptolemaic theory -- theories that are very suspect because of their excessive complexity and contrivance.

 Implausible, contrived scientific theories are like this

 (Source: wikiuniversity, Howard Community College)

Scientific theories are only well-confirmed by predictions when the theories make very many predictions that have been confirmed by observations.

Thinking that a scientific theory has been confirmed because it makes a few correct predictions is like thinking that you've proven you're a great baseball player because you've pounded out a few base hits. You've only proven yourself a great baseball player if you've made hundreds or thousands of hits. Similarly, the only scientific theories that are well-confirmed by predictions are those that have made hundreds, thousands or millions of predictions that have been confirmed.

We have a great example of such a theory: the theory of gravitation. The theory is based on a simple exact formula that you can use to compute the degree to which massive bodies attract each other. Scientists and engineers (and the computers on spacecraft) have used this theory thousands or millions of times, and the predictions made by the theory have always proven true. A robot spacecraft could never reach Mars and land on Mars unless the predictions of the theory of gravitation proved true thousands of times, nor could the Apollo astronauts have landed on the moon and returned.

Another such theory is the theory of electromagnetism. The theory is based on a simple exact formula you can use to compute the attraction between two electrical charges. Scientists and engineers have used the formula thousands or millions of times, and it always gives the right answer.

Compared to these theories, any theory that claims scientific validation because it seems to make a few correct predictions is like some kid who claims to be a professional actor because he acted in a few high-school plays.

According to the standard I mention here, we might have a reason for regarding many well-known scientific theories as being on rather shaky ground, as they do not make a huge number of predictions that have been confirmed. For example, we might regard as a very shaky theory the theory that life first arose on planet Earth merely because of a lucky chance combination of chemicals. Such a theory does not make a huge number of predictions that have been confirmed, and in fact, does not seem to make any prediction that has been confirmed.