Semi-automatic reduction and upscaling of large models: A farm management example

Research questions at the regional, national and global scales frequently require the upscaling of existing models. At large scales, simple model aggregation may have a prohibitive computational cost and lead to over-detailed problem representation. Methods that guide model simplification and revision have the potential to support the choice of the appropriate level of detail or heterogeneity within upscaled models. Efficient upscaling will retain only the heterogeneity that contributes to accurate aggregated results. This approach to model revision is challenging, because automatic generation of alternative models is difficult and the set of possible revised models is very large. In the case where simplification alone is considered, there are at least n²−1 possible simplified models where n is the number of model variables. Even with the availability of High Performance Computing, it is not possible to evaluate every possible simplified model if the number of model variables is greater than roughly 35. To address these issues, we propose a method that extends an existing procedure for simplifying and aggregating mechanistic models based on replacing model variables with constants. The method generates simplified models by selectively aggregating existing model variables, retaining existing model structure while reducing the size of the set of possible models and ordering them into a search tree. The tree is then searched selectively. We illustrate the method using a catchment scale optimization model with c. 50,000 variables (Farm-adapt) in the context of adaptation to climatic change. The method was successful in identifying redundant model variables and an adequate model 10% smaller than the original model. We discuss how the procedure can be extended to other large models and compare the method to those proposed by others. We conclude by urging model developers to regard their models as a starting point and to consider the need for alternative models during model development.     

Using a model selection criterion to identify appropriate complexity in aquatic biogeochemical models

Aquatic biogeochemical models are widely used as tools for understanding aquatic ecosystems and predicting their response to various stimuli (e.g., nutrient loading, toxic substances, climate change). Due to the complexity of these systems, such models are often elaborate and include a large number of estimated parameters. However, correspondingly large data sets are rarely available for calibration purposes, leading to models that may be overfit and possess reduced predictive capabilities. We apply, for the first time, information-theoretic model-selection techniques to a set of spatially explicit (1D) algal dynamics models of varying parameter dimension. We demonstrate that increases in complexity tend to produce a better model fit to calibration data, but beyond a certain degree of complexity the benefits of adding parameters are diminished (the risk of overfitting becomes greater). The particular approach taken here is computationally expensive, but several suggestions are made as to how multimodel methods may practically be extended to more sophisticated models.     

Strategic directions for agent-based modeling: avoiding the YAAWN syndrome

In this short communication, we examine how agent-based modeling has become common in land change science and is increasingly used to develop case studies for particular times and places. There is a danger that the research community is missing a prime opportunity to learn broader lessons from the use of agent-based modeling (ABM), or at the very least not sharing these lessons more widely. How do we find an appropriate balance between empirically rich, realistic models and simpler theoretically grounded models? What are appropriate and effective approaches to model evaluation in light of uncertainties not only in model parameters but also in model structure? How can we best explore hybrid model structures that enable us to better understand the dynamics of the systems under study, recognizing that no single approach is best suited to this task? Under what circumstances – in terms of model complexity, model evaluation, and model structure – can ABMs be used most effectively to lead to new insight for stakeholders? We explore these questions in the hope of helping the growing community of land change scientists using models in their research to move from ‘yet another model’ to doing better science with models.     

The role of agent-based models in wildlife ecology and management

Conservation planning of critical habitats for wildlife species at risk is a priority topic that requires the knowledge of how animals select and use their habitat, and how they respond to future developmental changes in their environment. This paper explores the role of a habitat-modeling methodological approach, agent-based modeling, which we advocate as a promising approach for ecological research. Agent-based models (ABMs) are capable of simultaneously distinguishing animal densities from habitat quality, can explicitly represent the environment and its dynamism, can accommodate spatial patterns of inter- and intra-species mechanisms, and can explore feedbacks and adaptations inherent in these systems. ABMs comprise autonomous, individual entities; each with dynamic, adaptive behaviors and heterogeneous characteristics that interact with each other and with their environment. These interactions result in emergent outcomes that can be used to quantitatively examine critical habitats from the individual- to population-level. ABMs can also explore how wildlife will respond to potential changes in environmental conditions, since they can readily incorporate adaptive animal-movement ecology in a changing landscape. This paper describes the necessary elements of an ABM developed specifically for understanding wildlife habitat selection, reviews the current empirical literature on ABMs in wildlife ecology and management, and evaluates the current and future roles these ABMs can play, specifically with regards to scenario planning of designated critical habitats.     

A proposal of an indicator for quantifying model robustness based on the relationship between variability of errors and of explored conditions

The evaluation of biophysical models is usually carried out by estimating the agreement between measured and simulated data and, more rarely, by using indices for other aspects, like model complexity and overparameterization. In spite of the importance of model robustness, especially for large area applications, no proposals for its quantification are available. In this paper, we would like to open a discussion on this issue, proposing a first approach for a quantification of robustness based on the variability of model error to variability of explored conditions ratio. We used modelling efficiency (EF) for quantifying error in model predictions and a normalized agrometeorological index (SAM) based on cumulated rainfall and reference evapotranspiration to characterize the conditions of application. Population standard deviations of EF and SAM were used to quantify their variability. The indicator was tested for models estimating meteorological variables and crop state variables. The values provided by the robustness indicator (I_R) were discussed according to the models’ features and to the typology and number of processes simulated. I_R increased with the number of processes simulated and, within the same typology of model, with the degree of overparameterization. No correlation were found between I_R and two of the most used indices of model error (RRMSE, EF). This supports its inclusion in integrated systems for model evaluation.     

Predicting aquaculture-derived benthic organic enrichment: Model validation

A sediment trap validation study was conducted near the commercial sea bass and sea bream fish farm in order to assess the predictive capability of a particle tracking deposition model. The validation procedure consisted of two distinct phases. First, the deposition of particulate waste (i.e. fecal pellets and excess feed) was measured near a single net pen containing 19 tons of sea bass. Afterwards, the model quality was determined by statistical comparison of predicted and observed values.Goodness of fit analysis indicates that the model successfully accounts for more than 75% of variance in the observed deposition. At 5% significance level, predictions do not underestimate or overestimate observations and there is no bias. Mean absolute relative error of ±48.9% compares favorably to other published deposition models. Obtained results affirm the reliability of particle tracking techniques in modeling the aquaculture-derived benthic organic enrichment.     

Using forest inventory data to assess fisher resting habitat suitability in California.

The fisher (Martes pennanti) is a forest-dwelling carnivore whose current distribution and association with late-seral forest conditions make it vulnerable to stand-altering human activities or natural disturbances. Fishers select a variety of structures for daily resting bouts. These habitat elements, together with foraging and reproductive (denning) habitat, constitute the habitat requirements of fishers. We develop a model capable of predicting the suitability of fisher resting habitat using standard forest vegetation inventory data. The inventory data were derived from Forest Inventory and Analysis (FIA), a nationwide probability-based sample used to estimate forest characteristics. We developed the model by comparing vegetation and topographic data at 75 randomly selected fisher resting structures in the southern Sierra Nevada with 232 forest inventory plots. We collected vegetation data at fisher resting locations using the FIA vegetation sampling protocol and centering the 1-ha FIA plot on the resting structure. To distinguish used and available inventory plots, we used nonparametric logistic regression to evaluate a set of a priori biological models. The top model represented a dominant portion of the Akaike weights (0.87), explained 31.5% of the deviance, and included the following variables: average canopy closure, basal area of trees <51 cm diameter breast height (dbh), average hardwood dbh, maximum tree dbh, percentage slope, and the dbh of the largest conifer snag. Our use of routinely collected forest inventory data allows the assessment and monitoring of change in fisher resting habitat suitability over large regions with no additional sampling effort. Although models were constrained to include only variables available from the list of those measured using the FIA protocol, we did not find this to be a shortcoming. The model makes it possible to compare average resting habitat suitability values before and after forest management treatments, among administrative units, across regions and over time. Considering hundreds of plot estimates as a sample of habitat conditions over large spatial scales can bring a broad perspective, at high resolution, and efficiency to the assessment and monitoring of wildlife habitat.     

Calibration of the forest vegetation simulator (FVS) model for the main forest species of Ontario,Canada

The forest vegetation simulator (FVS) model was calibrated for use in Ontario, Canada, to predict the growth of forest stands. Using data from permanent sample plots originating from different regions of Ontario, new models were derived for dbh growth rate, survival rate, stem height and species group density index for large trees and height and dbh growth rate for small trees. The dataset included black spruce (Picea mariana (Mill.) B.S.P.) and jack pine (Pinus banksiana Lamb.) for the boreal region, sugar maple (Acer saccharum Marsh.), white pine (Pinus strobus L.), red pine (Pinus resinosa Ait.) and yellow birch (Betula alleghaniensis Britton) for the Great Lakes-St. Lawrence region, and balsam fir (Abies balsamea (L.) Mill.) and trembling aspen (Populus tremuloides Michx.) for both regions. These new models were validated against an independent dataset that consisted of permanent sample plots located in Quebec. The new models predicted biologically consistent growth patterns whereas some of the original models from the Lake States version of FVS occasionally did not. The new models also fitted the calibration (Ontario) data better than the original FVS models. The validation against independent data from Quebec showed that the new models generally had a lower prediction error than the original FVS models.     

Predicting tree distributions in an East African biodiversity hotspot: model selection, data bias and envelope uncertainty

The Eastern Arc Mountains (EAMs) of Tanzania and Kenya support some of the most ancient tropical rainforest on Earth. The forests are a global priority for biodiversity conservation and provide vital resources to the Tanzanian population. Here, we make a first attempt to predict the spatial distribution of 40 EAM tree species, using generalised additive models, plot data and environmental predictor maps at sub 1 km resolution. The results of three modelling experiments are presented, investigating predictions obtained by (1) two different procedures for the stepwise selection of predictors, (2) down-weighting absence data, and (3) incorporating an autocovariate term to describe fine-scale spatial aggregation. In response to recent concerns regarding the extrapolation of model predictions beyond the restricted environmental range of training data, we also demonstrate a novel graphical tool for quantifying envelope uncertainty in restricted range niche-based models (envelope uncertainty maps). We find that even for species with very few documented occurrences useful estimates of distribution can be achieved. Initiating selection with a null model is found to be useful for explanatory purposes, while beginning with a full predictor set can over-fit the data. We show that a simple multimodel average of these two best-model predictions yields a superior compromise between generality and precision (parsimony). Down-weighting absences shifts the balance of errors in favour of higher sensitivity, reducing the number of serious mistakes (i.e., falsely predicted absences); however, response functions are more complex, exacerbating uncertainty in larger models. Spatial autocovariates help describe fine-scale patterns of occurrence and significantly improve explained deviance, though if important environmental constraints are omitted then model stability and explanatory power can be compromised. We conclude that the best modelling practice is contingent both on the intentions of the analyst (explanation or prediction) and on the quality of distribution data; generalised additive models have potential to provide valuable information for conservation in the EAMs, but methods must be carefully considered, particularly if occurrence data are scarce. Full results and details of all species models are supplied in an online Appendix.     

An evaluation of environmental factors affecting species distributions

Many different models can be built to explain the distributions of species. Often there is no single model that is clearly better than the alternatives, and this leads to uncertainty over which environmental factors are limiting species’ distributions. We investigated the support for different environmental factors by determining the drop in model performance when selected predictors were excluded from the model building process. We used a paired t-test over 37 plant species so that an environmental factor was only deemed significant if it consistently improved the results for multiple species. Geology and winter minimum temperatures were found to be the environmental factors with the most support, with a significant drop in model performance when either of these factors was excluded. However, there was less support for summer maximum temperature, as other environmental factors could combine to produce similar model performance. Our method of evaluating environmental factors using multiple species will not be capable of detecting predictors that are only important for one or two species, but it is difficult to distinguish these from spurious correlations. The strength of the method is that it increases inference for factors that consistently affect the distributions of many species. We discourage the assessment of models against predefined benchmarks, such as an area under the curve (AUC) of more than 0.7, as many alternative models for the same species produce similar results. Therefore, the benchmarks do not provide any indication of how the performance of the selected model compares to alternative models, and they provide weak inference to accept any selected model.     

The importance of spatial autocorrelation,extent and resolution in predicting forest bird occurrence

Concerns about declines in forest biodiversity underscore the need for accurate estimates of the distribution and abundance of organisms at large scales and at resolutions that are fine enough to be appropriate for management. This paper addresses three major objectives: (i) to determine whether the resolution of typical air photo-derived forest inventory is sufficient for the accurate prediction of site occupancy by forest birds. We compared prediction success of habitat models using air photo variables to models with variables derived from finer resolution, ground-sampled vegetation plots. (ii) To test whether incorporating spatial autocorrelation into habitat models via autologistic regression increases prediction success. (iii) To determine whether landscape structure is an important factor in predicting bird distribution in forest-dominated landscapes. Models were tested locally (Greater Fundy Ecosystem [GFE]) using cross-validation, and regionally using an independent data set from an area located ca. 250 km to the northwest (Riley Brook [RB]). We found significant positive spatial autocorrelation in the residuals of at least one habitat model for 76% (16/21) of species examined. In these cases, the logistic regression assumption of spatially independent errors was violated. Logistic models that ignored spatial autocorrelation tended to overestimate habitat effects. Though overall prediction success was higher for autologistic models than logistic models in the GFE, the difference was only significantly improved for one species. Further, the inclusion of spatial covariates did little to improve model performance in the geographically discrete study area. For 62% (13/21) of species examined, landscape variables were significant predictors of forest bird occurrence even after statistically controlling for stand-level variability. However, broad spatial extents explained less variation than local factors. In the GFE, 76% (16/21) of air photo and 81% (17/21) of ground plot models were accurate enough to be of practical utility (AUC > 0.7). When applied to RB, both model types performed effectively for 55% (11/20) of the species examined. We did not detect an overall difference in prediction success between air photo and ground plot models in either study area. We conclude that air photo data are as effective as fine resolution vegetation data for predicting site occupancy for the majority of species in this study. These models will be of use to forest managers who are interested in mapping species distributions under various timber harvest scenarios, and to protected areas planners attempting to optimize reserve function.     

A critique of models of the American lobster fishery

A number of studies have used the American lobster fishery to raise theoretical and empirical issues in the economic application of Schaefer yield-effort models. The present research shows that both published variants of the Schaefer yield-effort model are poor predictors of landings in the lobster fishery. The analysis suggests that minimum size rules in the lobster fishery make the Beverton-Holt dynamic pool model more appropriate to the lobster fishery than the Schaefer yield-effort model.