How to Find Good Science

A recent survey of school children reveals that many of them are skeptical about the veracity of information they find on the web (especially when the source gets the sports scores wrong). And well they should be. Presumably most adults have similar feelings. There is so much information out there. What are we to believe?

First, there is common sense. Releases for the Institute for Defense of Greenery, funded by a company building golf courses, may not be an entirely unbiased source for environmental information, for example. Likewise, comments emanating from the Fan Club of Bishop Ussher (who put the Earth's age at about 6000 years ago) might be treated with a lot more skepticism than, say, statements by the U.S. Geologic Survey or the Smithsonian.

However, trustworthiness is not invariably tied to position of authority. So, one would like a kind of "instant test", a kind of warning device that indicates the possibility that garbage is coming our way.

Unfortunately, there is no such test. However, we are not entirely helpless in the face of the oncoming stream of (mis-)information. A checklist might look as follows: A scientist can never be absolutely sure about anything, because a new observation or insight might force him to abandon previously cherished ideas. Every scientific theory implies that something must not happen. If it does happen, that theory is dead or must be modified. If, for example, a couple of scientists build a machine which delivers heat without any input of mass or energy, then our ideas about the conservation of mass and energy would have to be revised. Scientists are extremely confident that no such machine will ever be built. If it should be built, we would decide that there is something that is present in space and time, cryptically, that can emerge as energy or mass. We would give it a name, and proceed from there.

When testing hypotheses (such as about the typical life span of stars, say) scientists use mathematics. They make a large number of observations (how often do we see stars explode in galaxies) and compare this with expectations (given the number of stars in the galaxies, how many should be exploding every year). Such comparisons are a matter of "statistics". There are in fact two kinds of different statistics: descriptive statistics, and statistical tests. The first is useful in describing populations: on average there are so many stars per galaxy, with x percent having a factor of two less, and y percent having a factor of two more (x and y tend to be similar). The second explores the probability that a statement is correct (or false). Take the statement: "More than half of the stars are double or triple stars." Correct or false? You need to count the proportion of singles in a sample. Then you need to establish whether a count of, say, 48% singles establishes the truth of the statement, given that you only have evidence for a sample and not the entire population. The answer depends on the sample size. If you have a large enough sample, you can be fairly sure the statement is correct. The confidence in the statement is reflected by its likelihood of being wrong. P (the probability of occurrence) < 0.01 means that chances are less than one in a hundred that the statement is wrong. Most scientists will accept this as a "true" statement.

What do scientists do as far as getting the "truth"? Of course, scientists are not so different from ordinary people: they don't know much about anything in great depth, except in their own specialty. And they do have a habit of trusting their approved sources. Their highest category of believability, called "primary references", consists of refereed journals. These are journals where nothing is published that has not undergone rigorous review by experts (so-called "peer review"). Examples are the journals Science and Nature, as well as thousands of specialty journals, such as The Astrophysics Journal and The Journal for Geophysical Research. You will not, as a student without special background or special scientific ambitions, normally need to consult any such references.

For broadening one's horizons, there are "secondary references", with articles that build on the primary literature. This is good stuff and includes books and magazine articles by reputable authors (those who have lots of refereed articles that stood the test of time). It may also include web sites by reputable organizations, especially those with the suffix ".edu" and ".gov" (because there are internal checks on publishing junk). Many science writers of books and magazine articles produce usable materials, especially if they took care to have their writings reviewed by experts. There are also at least one newspaper writer who qualifies as a source of reliable secondary material: John Noble Wilford of the N.Y. Times. We would like to think that our course also falls into this category, since we have a reasonable working knowledge of much of what we talk about here, or else have used reliable sources.

Tertiary references (which build on secondary literature) are somewhat less trustworthy, of course, although things of interest may appear there, as well. In addition, there is much trash. This is mainly of interest in documenting the range of misconceptions that exist regarding any one topic of scientific interest.

Good luck with separating the grain from the chaff. It is not easy.