What do significance tests really tell us about the environment?

Routine application of significance tests does not extract the maximum information from environmental data and can lead to misleading conclusions. Reasons leading to this are: a significant result can often be reached merely by collecting enough samples; a statistically significant result is not necessarily practically significant; and reports of the presence or absence of statistically significant differences for multiple tests are not comparable unless identical sample sizes are used. These problems are demonstrated by application to pH data for grazed and retired fields, and by discussion of significance tests used in recent US regulations for groundwater quality. The advantages of equivalence tests, where the tester must state the difference of practical difference, are discussed and applied to the field pH problem. We recommend that environmental managers and scientists pay more attention to statistical power and decide on what is a practical difference. Confidence intervals for the size of the differences, accompanied where necessary by equivalence tests, are the preferred means of addressing the question: “is there a difference of practical significance?”