Simulation experiments comparing alternative process formulations using a factorial design

This paper reviews methods for exploring the differences between alternative equations in complex ecosystem models. A factorial design is proposed as a method for exposing possible interactions between equation forms in their effect on model output as well as to clarify differences between the main candidate equations. A number of display methods arising from statistical analysis are used including normal Q-Q plots, linear rank plots and interaction diagrams. The methods were illustrated using a complex ecosystem model of Lake Ontario. We found the methods effective at illustrating major differences between equations although several difficulties arose due to the complexity of the models and the diffuse nature of the data supporting model validation. Questions of the method for standardization of equation forms so that the compared equations are in some way analogous, are important. These methods are probably most useful in cases where the data are of sufficient quality to indicate not only how different equation forms affect model output but also which are to be preferred.