The Test for Credibility

A recent survey of school children reveals that many of them are skeptical about the veracity of information they find on the web (especially when the source gets the sports scores wrong). And well they should be. Presumably most adults have similar feelings. There is so much information out there. What are we to believe?

First, there is common sense. Releases for the Institute for Defense of Greenery, funded by a company building golf courses, may not be an entirely unbiased source for environmental information, for example. Likewise, comments emanating from the Fan Club of Bishop Ussher (who put the Earth's age at about 6000 years ago) might be treated with a lot more skepticism than, say, statements by the U.S. Geologic Survey or the Smithsonian.

However, as we have seen in the case of Percival Lowell (professor of astronomy at MIT), trustworthiness is not invariably tied to position of authority. So, one would like a kind of "instant test", a kind of warning device that indicates the possibility that garbage is coming our way.

Unfortunately, there is no such test. However, we are not entirely helpless in the face of the oncoming stream of (mis-)information. A checklist might look as follows:

Is the statement such that - in principle - it could be proven wrong (falsifiable)? If not, it is not about science. Ignore it.
Is the statement to the advantage of the one making it? This could be in regard to money, status, or cultural comfort. If yes, be suspicious.
Is the statement one reflecting wishful thinking? (Wouldn't it be nice if ...) If yes: suspect junk science.
Are there a lot of natural laws being magically overridden? If yes: suspect crank science.
Are scientists heatedly disagreeing about it? If yes, be aware that this is a matter under discussion. This is normal in science, but wear a hard hat.
Is the statement made by one who absolutely, positively, 100% knows the answer? If yes, it may be right or wrong, but the declarer is not a scientist, for sure.

A scientist can never be absolutely sure about anything, because a new observation or insight might force him to abandon previously cherished ideas. Every scientific theory implies that something must not happen. If it does happen, that theory is dead or must be modified. If, for example, a couple of scientists build a machine which delivers heat without any input of mass or energy, then our ideas about the conservation of mass and energy would have to be revised. Scientists are extremely confident that no such machine will ever be built. If it should be built, we would decide that there is something that is present in space and time, cryptically, that can emerge as energy or mass. We would give it a name, and proceed from there.

When testing hypotheses (such as about the typical life span of stars, say) scientists use mathematics. They make a large number of observations (how often do we see stars explode in galaxies) and compare this with expectations (given the number of stars in the galaxies, how many should be exploding every year). Such comparisons are a matter of "statistics". There are in fact two kinds of different statistics: descriptive statistics, and statistical tests. The first is useful in describing populations: on average there are so many stars per galaxy, with x percent having a factor of two less, and y percent having a factor of two more (x and y tend to be similar). The second explores the probability that a statement is correct (or false). Take the statement: "More than half of the stars are double or triple stars". Correct or false? You need to count the proportion of singles in a sample. Then you need to establish whether a count of, say, 48% singles establishes the truth of the statement, given that you only have evidence for a sample and not the entire population. The answer depends on the sample size. If you have a large enough sample, you can be fairly sure the statement is correct. The confidence in the statement is reflected by its likelihood of being wrong. P (the probability of occurrence) < 0.01 means that chances are less than one in a hundred that the statement is wrong. Most scientists will accept this as a "true" statement.