Using biological data from field studies with multiple reference sites as a basis for environmental management: the risks for false positives and false negatives

Field surveys of biological responses can provide valuable information about environmental status and anthropogenic stress. However, it is quite usual for biological variables to differ between sites or change between two periods of time also in the absence of an impact. This means that there is an obvious risk that natural variation will be interpreted as environmental impact, or that relevant effects will be missed due to insufficient statistical power. Furthermore, statistical methods tend to focus on the risks for Type-I error, i.e. false positives. For environmental management, the risk for false negatives is (at least) equally important. The aim of the present study was to investigate how the probabilities for false positives and negatives are affected by experimental set up (number of reference sites and samples per site), decision criteria (statistical method and α-level) and effect size. A model was constructed to simulate data from multiple reference sites, a negative control and a positive control. The negative control was taken from the same distribution as the reference sites and the positive control was just outside the normal range. Using the model, the probabilities to get false positives and false negatives were calculated when a conventional statistical test, based on a null hypothesis of no difference, was used along with alternative tests that were based on the normal range of natural variation. Here, it is tested if an investigated site is significantly inside (equivalence test) and significantly outside (interval test) the normal range. Furthermore, it was tested how the risks for false positives and false negatives are affected by changes in α-level and effect size. The results of the present study show that the strategy that best balances the risks between false positives and false negatives is to use the equivalence test. Besides tests with tabulated p-values, estimates generated using a bootstrap routine were included in the present study. The simulations showed that the probability for management errors was smaller for the bootstrap compared to the traditional test and the interval test.