General Practice, Solo & Small Firm DivisionMagazine
Natural Resources, Energy, and Environmental Law
What is Good Science?
By Dr. Samuel J. McNaughton
In Daubert v. Merrell Dow Pharmaceuticals, the United States Supreme Court charged trial judges with acting as independent evaluators of scientific testimony, using the criteria of empirical testability of an assertion, peer review and publication of results, error inherent in a methodology, and degree of acceptance of that methodology by the scientific community. This article suggests four closely related criteria that courts can use to determine whether testimony is dependable scientific evidence: procedure, performance, repeatability, and peer review.
Judging whether information is good science depends on affirmative answers to four questions that encompass these criteria. Does the process follow the scientific method? Was the process performed in an objective manner? Are the results repeatable? Have the results been published in a peer-reviewed publication?
Scientific Method. The collection of the data on which a conclusion is based must conform to a scientific method. A scientist’s observations (data) about a phenomenon prompt the scientist to pose a question about the phenomenon. Next, the scientist reformulates the question as a hypothesis. Hypotheses then make predictions and data are collected and analyzed to test the prediction. A determination is then made about the likelihood that the result was due to chance and whether the result is scientifically important.
Measuring a difference in data that is gathered does not necessarily mean that a hypothesis is correct or incorrect. The scientist must first determine whether a difference in results is due to chance. Once statistical significance has been established, the scientist must decide whether the results are of scientific significance. The answer lies in the proportion of the total variation (statistically, variance) explained by the phenomenon. A phenomenon that has devastating effects on individuals or society may be important, even if it is unlikely.
Objectivity in execution is the second standard of good science. The procedure of science must be executed in a way that does not influence the results. Methods such as double blind, randomization, variable matching, and analysis of variance, are aimed at eliminating bias and are particularly important when dealing with phenomena that directly impinge upon human beings.
The third standard of good science requires that the result be repeatable. For a scientific conclusion to be acceptable, other scientists must be able to repeat it in other locations at other times employing the same methods. The fourth standard of good science requires that the results be published in a scientific journal or other publication that is peer reviewed. Passing the hurdle of peer-reviewed publication is an assurance that quality control has been exercised in communicating the results to other scientists and meets a type of reliability norm on which decision-makers can rely.
Any "science" that does not meet all four of the standards (procedure, performance, duplication, and peer scrutiny) is not good science.
Scientific Practice. Scientists have a general consensus about what constitutes good experimental design. It is that consensus about good experimental design that inherently tests a null hypothesis. The single most important aspect of good experimental design that tests a null hypothesis is the requirement that the experimenter’s hypothesis be capable of being disproved by the experiment. To allow a hypothesis of causation to be disproved, the scientist must allow an equal opportunity for the truly opposite, or null, hypothesis to be proven.
Rejecting Null Hypotheses. Science progresses by rejecting hypotheses of no effect or no relationship: If I do this, that will not happen. "If I put my hand over a burning candle, I will not get burned." That statement is an experimental null hypothesis of no effect. It involves your intervention as a scientist, by placing your hand above the candle. "There is no relationship between the number of cigarettes a person smokes and the likelihood that person will develop lung cancer." That statement is an associational null hypothesis of no relationship between two variables, number of cigarettes smoked and probability of contracting lung cancer. Generally, experimental results are more reliable, more trustworthy, and more repeatable than associational results. In addition, associational results cannot unequivocally demonstrate cause and effect. For example, the number of cigarettes smoked and probability of suffering lung cancer might both be caused by other, unidentified variables, such as stress, or poor office or home air quality.
Possibly, the most difficult facet of good science for nonpractitioners to master is that good science proceeds by disproving ideas. But because we disprove null hypotheses, every prevailing idea is based on tentative acceptance. As facts pile up and theory organizes those facts, that acceptance becomes less and less tentative, but it can never be absolute because new ideas, technologies, and perspectives may modify the available facts and theories.
Society and Science. Society views science in three ways. First, science is traditionally considered, and taught, as a body of "facts"; that is, observations about existence that are certainly true. That is the way that science commonly enters the courts, leading to the phenomenon of "dueling experts," one asserting one set of declarations and another asserting an opposite set of contentions.
Second, science is viewed as a "theory," which is more insubstantial than facts but is somehow associated with facts. Scientific facts and theory are related in the following fashion: facts confirm theory, and theory organizes facts. Facts are confirmed, repeatable, and therefore shared observations that need to be explained, and theory organizes the facts into a coherent, explanatory concept.
Third, science in our society has come to have a quality of infallibility attached to it. However, scientists vary in the proficiency with which they practice their craft. Those who do it best combine adherence to the four standards with imagination, leading to new insight into how objective reality is organized. Such ideas are theories only if they organize a spectrum of facts into a coherent whole. Otherwise, they are the step-by-step hypotheses that scientists test through disproving null hypotheses. A theory is an organized totality of hypotheses about some set of natural phenomena.
The essence of good science is repeatability. Different scientists, in different places, at different times, can repeat good science if they follow the same methods and protocols. The core of good science is thoughtful design of the test of a hypothesis, careful execution of that design, and rigorous analysis of the data to test the null hypothesis.
The standards that the courts should bring to evaluating testimony by scientific experts are the standards of good science. Did procedures follow the scientific method? Does testimony indicate that execution of tests was unbiased? Has the result been repeated by independent scientists? Have the results underlying the testimony been published in a scientific journal and met the standards of peer review?
We should be aware, however, that most "expert" testimony in litigation is not by scientists. A 1991 study found that only 10 percent of expert testimony in federal civil cases was from scientists, and 40 percent was from practitioners in medical and mental health fields. The U. S. Supreme Court decided this term, by ruling on Kumho Tires Co. v. Carmichael, that experts other than scientists should be held to the same standards as those promulgated in Daubert.
Dr. Samuel J. McNaughton is a William Rand Kenan Jr. professor of science in the biology department at Syracuse University.
- This article is an abridged and edited version of one that originally appeared on page 513 in Natural Resources & Environment, Spring 1999 (13:4).