Rethinking receiver operating characteristic analysis applications in ecological niche modeling

The area under the curve (AUC) of the receiver operating characteristic (ROC) has become a dominant tool in evaluating the accuracy of models predicting distributions of species. ROC has the advantage of being threshold-independent, and as such does not require decisions regarding thresholds of what constitutes a prediction of presence versus a prediction of absence. However, we show that, comparing two ROCs, using the AUC systematically undervalues models that do not provide predictions across the entire spectrum of proportional areas in the study area. Current ROC approaches in ecological niche modeling applications are also inappropriate because the two error components are weighted equally. We recommend a modification of ROC that remedies these problems, using partial-area ROC approaches to provide a firmer foundation for evaluation of predictions from ecological niche models. A worked example demonstrates that models that are evaluated favorably by traditional ROC AUCs are not necessarily the best when niche modeling considerations are incorporated into the design of the test.     

Ecological relevance of performance criteria for species distribution models

Species distribution models have often been developed based on ecological data. To develop reliable data-driven models, however, a sound model training and evaluation procedures are needed. A crucial step in these procedures is the assessment of the model performance, with as key component the applied performance criterion. Therefore, we reviewed seven performance criteria commonly applied in presence-absence modelling (the correctly classified instances, Kappa, sensitivity, specificity, the normalised mutual information statistic, the true skill statistic and the odds ratio) and analysed their application in both the model training and evaluation process. Although estimates of predictive performance have been used widely to assess final model quality, a systematic overview was missing because most analyses of performance criteria have been empirical and only focused on specific aspects of the performance criteria. This paper provides such an overview showing that different performance criteria evaluate a model differently and that this difference may be explained by the dependency of these criteria on the prevalence of the validation set. We showed theoretically that these prevalence effects only occur if the data are inseparable by an n-dimensional hyperplane, n being the number of input variables. Given this inseparability, different performance criteria focus on different aspects of model performance during model training, such as sensitivity, specificity or predictive accuracy. These findings have important consequences for ecological modelling because ecological data are mostly inseparable due to data noise and the complexity of the studied system. Consequently, it should be very clear which aspect of the model performance is evaluated, and models should be evaluated consistently, that is, independent of, or taking into account, species prevalence. The practical implications of these findings are clear. They provide further insight into the evaluation of ecological presence/absence models and attempt to assist modellers in their choice of suitable performance criteria.     

关于使用硫代硫酸钠溶液省略标定过程的讨论

通过实验证明使用低浓度的硫代硫酸的钠标液时，可用已配制标定好的硫代硫酸钠标准贮备液直接稀释使用，不需重新标定。此法切实可行，完全能满足样品分析的要求，既省时省力，又节约试剂，值得推广。