Recommendations on statistics and benchmarks to assess photochemical model performance

Photochemical grid models are addressing an increasing variety of air quality related issues, yet procedures and metrics used to evaluate their performance remain inconsistent. This impacts the ability to place results in quantitative context relative to other models and applications, and to inform the user and affected community of model uncertainties and weaknesses. More consistent evaluations can serve to drive improvements in the modeling process as major weaknesses are identified and addressed. The large number of North American photochemical modeling studies published in the peer-reviewed literature over the past decade affords a rich data set from which to update previously established quantitative performance “benchmarks” for ozone and particulate matter (PM) concentrations. Here we exploit this information to develop new ozone and PM benchmarks (goals and criteria) for three well-established statistical metrics over spatial scales ranging from urban to regional and over temporal scales ranging from episodic to seasonal. We also recommend additional evaluation procedures, statistical metrics, and graphical methods for good practice. While we primarily address modeling and regulatory settings in the United States, these recommendations are relevant to any such applications of state-of-the-science photochemical models. Our primary objective is to promote quantitatively consistent evaluations across different applications, scales, models, model inputs, and configurations. The purpose of benchmarks is to understand how good or poor the results are relative to historical model applications of similar nature and to guide model performance improvements prior to using results for policy assessments. To that end, it also remains critical to evaluate all aspects of the model via diagnostic and dynamic methods. A second objective is to establish a means to assess model performance changes in the future. Statistical metrics and benchmarks need to be revisited periodically as model performance and the characteristics of air quality change in the future.

Implications: We address inconsistent procedures and metrics used to evaluate photochemical model performance, recommend a specific set of statistical metrics, and develop updated quantitative performance benchmarks for those metrics. We promote quantitatively consistent evaluations across different applications, scales, models, inputs, and configurations, thereby (1) improving the user’s ability to quantitatively place results in context and guide model improvements, and (2) better informing users, regulators, and stakeholders of model uncertainties and weaknesses prior to using results for policy assessments. While we primarily address U.S. modeling and regulatory settings, these recommendations are relevant to any such applications of state-of-the-science photochemical models.     

A Qualitative Approach to Evaluating the Anticipated Reliability of a Photochemical Air Quality Simulation Model for a Selected Application

ABSTRACT

Photochemical air quality simulation models are now used widely in evaluating the merits of alternative emissions control strategies on spatial scales from metropolitan to sub-continental. Greatly varying levels of resources have been available to support modeling, from relatively comprehensive databases and evaluation of performance to a paucity of aerometric data for developing model inputs. Where data are sparse, many alternative outcomes are consistent with the knowledge at hand. Where performance evaluation is inadequately supported, the probability of error may be high. In each instance, uncertainties may be large when compared with the signal of interest, and thus confidence in the reliability of the model as an estimator of future air quality may come into question.

This paper proposes a qualitative procedure for assessing whether a particular application of a modeling system is likely to be potentially unreliable, suggesting that either (1) modification and further evaluation is needed, if supportable, prior to adoption for regulatory application; or (2) the model should not be used if improvement is not supportable. The procedure is proposed for use by policy-makers, staffs of public agencies, air quality managers, environmental staffs of industrial organizations, and other interested parties. The proposed use of the procedure is (1) to assess, a priori, whether a proposed application is likely to be judged questionable or unacceptably uncertain in outcome; and (2) to provide, a posteriori, a basis for judging quickly the likely quality of model performance. The procedure is presented with tropospheric ozone as the pollutant of concern. With adjustments, however, the procedure should be applicable for particu-late matter and other pollutants of interest.     

Air Quality Modeling and Decisions for Ozone Reduction Strategies

Abstract

Despite the widespread application of photochemical air quality models (AQMs) in U.S. state implementation planning (SIP) for attainment of the ambient ozone standard, documentation for the reliability of projections has remained highly subjective. An “idealized” evaluation framework is proposed that provides a means for assessing reliability. Applied to 18 cases of regulatory modeling in the early 1990s in North America, a comparative review of these applications is reported. The intercomparisons suggest that more than two thirds of these AQM applications suffered from having inadequate air quality and meteorological databases. Emissions representations often were unreliable; uncertainties were too high. More than two thirds of the performance evaluation efforts were judged to be substandard compared with idealized goals. Meteorological conditions chosen according regulatory guidelines were limited to one or two cases and tended to be similar, thus limiting the extent to which public policy makers could be confident that the emission controls adopted would yield attainment for a broad range of adverse atmospheric conditions. More than half of the studies reviewed did not give sufficient attention to addressing the potential for compensating errors. Corroborative analyses were conducted in only one of the 18 studies reviewed. Insufficient attention was given to the estimation of model and/or input database errors, uncertainties, or variability in all of the cases examined. However, recent SIP and policy‐related regional modeling provides evidence of substantial improvements in the underlying science and available modeling systems used for regulatory decision making. Nevertheless, the availability of suitable databases to support increasingly sophisticated modeling continues to be a concern for many locations. Thus, AQM results may still be subject to significant uncertainties. The evaluative process used here provides a framework for modelers and public policy makers to assess the adequacy of contemporary and future modeling work.     

Effects of Land Use Data on Dry Deposition in a Regional Photochemical Model for Eastern Texas

ABSTRACT

Land use data are among the inputs used to determine dry deposition velocities for photochemical grid models such as the Comprehensive Air Quality Model with extensions (CAMx) that is currently used for attainment demonstrations and air quality planning by the state of Texas. The sensitivity of dry deposition and O₃ mixing ratios to land use classification was investigated by comparing predictions based on default U.S. Geological Survey (USGS) land use data to predictions based on recently compiled land use data that were collected to improve biogenic emissions estimates. Dry deposition of O₃ decreased throughout much of eastern Texas, especially in urban areas, with the new land use data. Predicted 1-hr averaged O₃ mixing ratios with the new land use data were as much as 11 ppbv greater and 6 ppbv less than predictions based on USGS land use data during the late afternoon. In addition, the area with peak O₃ mixing ratios in excess of 100 ppbv increased significantly in urban areas when deposition velocities were calculated based on the new land use data. Finally, more detailed data on land use within urban areas resulted in peak changes in O₃ mixing ratios of ~2 ppbv. These results indicate the importance of establishing accurate, internally consistent land use data for photochemical modeling in urban areas in Texas. They also indicate the need for field validation of deposition rates in areas experiencing changing land use patterns, such as during urban reforestation programs or residential and commercial development.     

Modeling the long-term frequency distribution of regional ozone concentrations

Efficient methods are developed for modeling emissions – air quality relationships that govern ozone and NO₂ concentrations over very long periods of time. A baseline model evaluation study is conducted to assess the accuracy and speed with which the relationship between pollutant emissions and the frequency distribution of O₃ concentrations throughout the year can be computed along with annual average NO₂ values using a deterministic photochemical airshed model driven by automated objective analysis of measured meteorological parameters. Methods developed are illustrated by application to the air quality situation that exists in Southern California. Model performance statistics for O₃ are similar to the results obtained in previous short-term episodic model evaluation studies that were based on hand-crafted meteorological inputs that are supplemented by expensive field measurement campaigns. Model predictions at one of the highest NO₂ concentration sites in the US indicate that measured violation of the US annual average NO₂ air quality standard at that site occurs because other species such as HNO₃ and PAN are measured as if they were NO₂ by the chemiluminescent NO_x monitors in current use.     

An Air Quality Modeling Study Comparing Two Possible Sites for the New International Airport for Mexico City

Abstract

Using an air quality model, two future urban scenarios induced by the construction of the new international airport for Mexico City are compared at a regional level. The air quality model couples the meteorology model MM5 and state-of-the-art photochemistry. The air quality comparison is made using metrics for the criterion gases selected for the study. From the two urban scenarios compared, the option for Tizayuca is moderately better than the option for Texcoco, because relative reductions in O₃ and other photochemical pollutants are achieved over highly populated areas. Regardless of the site, the air quality for the central region of Mexico in the future will deteriorate. In the region of central Mexico, SO₂ and NO₂ will become important pollutants.     

Application of a scenario-based modeling system to evaluate the air quality impacts of future growth

The structure and design of future urban development can have significant adverse effects on air pollutant emissions as well as other environmental factors. When considering the future impact of growth on mobile source emissions, we generally model the increase in vehicle kilometers traveled (VKT) as a function of population growth. However, diverse and poorly planned urban development (i.e., urban sprawl) can force higher rates of motor vehicle use and in return increase levels of pollutant emissions than alternative land-use scenarios. The objective of this study is to develop and implement an air quality assessment tool that takes into account the influence of alternative growth and development scenarios on air quality.The use of scenario-based techniques in land use planning has been around since the late 1940s and been tested in many different applications to aid in decision-making. In this study, we introduce the development of an advanced interactive scenario-based land use and atmospheric chemistry modeling system coupled with a GIS (Geographical Information System) framework. The modeling system is designed to be modular and includes land use/land cover information, transportation, meteorological, emissions, and photochemical modeling components. The methods and modularity of the developed system allow its application to both broad areas and applications.To investigate the impact of possible land use change and urbanization, we evaluated a set of alternative future patterns of land use developed for a study area in Southwest California. Four land use and two population variants (increases of 500k and 1M) were considered. Overall, a Regional Low-Density Future was seen to have the highest pollutant emissions, largest increase in VKT, and the greatest impact on air quality. On the other hand, a Three-Centers Future appeared to be the most beneficial alternative future land-use scenario in terms of air quality. For all cases, the increase in population was the main factor leading to the change on predicted pollutant levels.     

Frank A. Chambers

Abstract

This work assessed the usefulness of a current air quality model (American Meteorological Society/Environmental Protection Agency Regulatory Model [AERMOD]) for predicting air concentrations and deposition of perfluorooctanoate (PFO) near a manufacturing facility. Air quality models play an important role in providing information for verifying permitting conditions and for exposure assessment purposes. It is important to ensure traditional modeling approaches are applicable to perfluorinated compounds, which are known to have unusual properties. Measured field data were compared with modeling predictions to show that AERMOD adequately located the maximum air concentration in the study area, provided representative or conservative air concentration estimates, and demonstrated bias and scatter not significantly different than that reported for other compounds. Surface soil/grass concentrations resulting from modeled deposition flux also showed acceptable bias and scatter compared with measured concentrations of PFO in soil/grass samples. Errors in predictions of air concentrations or deposition may be best explained by meteorological input uncertainty and conservatism in the PRIME algorithm used to account for building downwash. In general, AERMOD was found to be a useful screening tool for modeling the dispersion and deposition of PFO in air near a manufacturing facility.     

Emission Reductions Needed to Meet the Standard for Ozone in Southern California: Effect of Boundary Conditions

Abstract

Combinations of total reactive organic gas (ROG) and nitrogen oxide (NO_x) emissions that do not exceed the National Ambient Air Quality Standard (NAAQS) for ozone for the meteorological conditions of the August 26-28, 1987 SCAQS episode, have been determined using the California Institute of Technology (CIT) photochemical air quality model. The sensitivity of these combinations to pollutant boundary conditions is examined.     

Conjunctive Use of Models to Design Cost-Effective Ozone Control Strategies

Abstract

The management of tropospheric ozone (O₃) is particularly difficult. The formulation of emission control strategies requires considerable information including: (1) emission inventories, (2) available control technologies, (3) meteorological data for critical design episodes, and (4) computer models that simulate atmospheric transport and chemistry. The simultaneous consideration of this information during control strategy design can be exceedingly difficult for a decision-maker. Traditional management approaches do not explicitly address cost minimization. This study presents a new approach for designing air quality management strategies; a simple air quality model is used conjunctively with a complex air quality model to obtain low-cost management strategies. A simple air quality model is used to identify potentially good solutions, and two heuristic methods are used to identify cost-effective control strategies using only a small number of simple air quality model simulations. Subsequently, the resulting strategies are verified and refined using a complex air quality model. The use of this approach may greatly reduce the number of complex air quality model runs that are required. An important component of this heuristic design framework is the use of the simple air quality model as a screening and exploratory tool. To achieve similar results with the simple and complex air quality models, it may be necessary to “tweak” or calibrate the simple model. A genetic algorithm-based optimization procedure is used to automate this tweaking process. These methods are demonstrated to be computationally practical using two realistic case studies, which are based on data from a metropolitan region in the United States.     

Application of Genetic Algorithms for the Design of Ozone Control Strategies

ABSTRACT

Designing air quality management strategies is complicated by the difficulty in simultaneously considering large amounts of relevant data, sophisticated air quality models, competing design objectives, and unquantifiable issues. For many problems, mathematical optimization can be used to simplify the design process by identifying cost-effective solutions. Optimization applications for controlling nonlinearly reactive pollutants such as tropospheric ozone, however, have been lacking because of the difficulty in representing nonlinear chemistry in mathematical programming models.

We discuss the use of genetic algorithms (GAs) as an alternative optimization approach for developing ozone control strategies. A GA formulation is described and demonstrated for an urban-scale ozone control problem in which controls are considered for thousands of pollutant sources simultaneously. A simple air quality model is integrated into the GA to represent ozone transport and chemistry. Variations of the GA formulation for multiobjective and chance-constrained optimization are also presented. The paper concludes with a discussion of the practicality of using more sophisticated, regulatory-scale air quality models with the GA. We anticipate that such an approach will be practical in the near term for supporting regulatory decision-making.     

Space-time analysis of the Air Quality Model Evaluation International Initiative (AQMEII) Phase 1 air quality simulations

This study presents an evaluation of summertime ozone concentrations over North America (NA) and Europe (EU) using the database generated from Phase 1 of the Air Quality Model Evaluation International Initiative (AQMEII). The analysis focuses on identifying temporal and spatial features that can be used to stratify operational model evaluation metrics and to test the extent to which the various modeling systems can replicate the features seen in the observations. Using a synoptic map typing approach, it is demonstrated that model performance varies with meteorological conditions associated with specific synoptic-scale flow patterns over both eastern NA and EU. For example, the root mean square error of simulated daily maximum 8-hr ozone was twice as high when cloud fractions were high compared with when cloud fractions were low over eastern NA. Furthermore, results show that over both NA and EU the regional models participating in AQMEII were able to better reproduce the observed variance in ambient ozone levels than the global model used to specify chemical boundary conditions, although the variance simulated by almost all regional models is still less that the observed variance on all spatiotemporal scales. In addition, all modeling systems showed poor correlations with observed fluctuations on the intraday time scale over both NA and EU. Furthermore, a methodology is introduced to distinguish between locally influenced and regionally representative sites for the purpose of model evaluation. Results reveal that all models have worse model performance at locally influenced sites. Overall, the analyses presented in this paper show how observed temporal and spatial information can be used to stratify operational model performance statistics and to test the modeling systems’ ability to replicate observed temporal and spatial features, especially at scales the modeling systems are designed to capture.

Implications: The analyses presented in this paper demonstrate how observed temporal and spatial information can be used to stratify operational model performance and to test the modeling systems’ ability to replicate observed temporal and spatial features. Decisions for the improvement of regional air quality models should be based on the information derived from only regionally representative sites.     

An Assessment of the Emissions Inventory Processing Systems EMS-2001 and SMOKE in Grid-Based Air Quality Models

Abstract

In the United States, emission processing models such as Emissions Modeling System-2001 (EMS-2001), Emissions Preprocessor System-Version 2.5 (EPS2.5), and the Sparse Matrix Operator Kernel Emissions (SMOKE) model are currently being used to generate gridded, hourly, speciated emission inputs for urban and regional-scale photochemical models from aggregated pollutant inventories. In this study, two models, EMS-2001 and SMOKE, were applied with their default internal data sets to process a common inventory database for a high ozone (O₃) episode over the eastern United States using the Carbon Bond IV (CB4) chemical speciation mechanism. A comparison of the emissions processed by these systems shows differences in all three of the major processing steps performed by the two models (i.e., in temporal allocation, spatial allocation, and chemical speciation). Results from a simulation with a photochemical model using these two sets of emissions indicate differences on the order of ±20 ppb in the predicted 1-hr daily maximum O₃ concentrations. It is therefore critical to develop and implement more common and synchronized temporal, spatial, and speciation cross-reference systems such that the processes within each emissions model converge toward reasonably similar results. This would also help to increase confidence in the validity of photochemical grid model results by reducing one aspect of modeling uncertainty.     

A Comparison of Measured and Simulated Ozone Concentrations in the Rural Areas of the Eastern United States during Summer 1995

ABSTRACT

The recent regulatory actions toward a longer-term (i.e., 8-hr) average ozone standard have brought forth the potential for many rural areas in the eastern United States to be in noncompliance. However, since a majority of these rural areas have generally few sources of anthropogenic emissions, the measured ozone levels primarily reflect the effects of the transport of ozone and its precursor pollutants and natural emissions. While photochemical grid models have been applied to urban areas to develop ozone mitigation measures, these efforts have been limited to high ozone episode events only and do not adequately cover rural regions. In this study, we applied a photochemical modeling system, RAMS/UAM-V, to the eastern United States from June 1-August 31, 1995. The purpose of the study is to examine the predictive ability of the modeling system at rural monitoring stations that are part of the Clean Air Status Trends Network (CASTNet) and the Gaseous Pollutant Monitoring Program (GPMP).

The results show that the measured daily 1-hr ozone maxima and the seasonal average of the daily 1-hr ozone maxima are in better agreement with the predictions of the modeling system than those for the daily 8-hr ozone maxima. Also, the response of the modeling system in reproducing the measured range of ozone levels over the diurnal cycle is poor, suggesting the need for improvement in the treatment of the physical and chemical processes of the modeling system during the nighttime and morning hours if it is to be used to address the 8-hr ozone standard.     

Control of Ozone Precursors in a Complex Industrial Terrain by Using Multiscale-Nested Air Quality Models with Fine Spatial Resolution (1 km2)

Abstract

The location of the northeastern Iberian Peninsula (NEIP) in the northwestern Mediterranean basin, the presence of the Pyrenees mountain range (with altitudes >3000 m), and the influence of the Mediterranean Sea and the large valley canalization of Ebro river induce an extremely complicated structure for the dispersion of photochemical pollutants. Air pollution studies in very complex terrains such as the NEIP require high-resolution modeling for resolving the very complex dynamics of flows. To deal with the influence of larger-scale transport, however, high-resolution models have to be nested in larger models to generate appropriate initial and boundary conditions for the finer resolution domains. This article shows the results obtained through the utilization of the MM5-EMICAT2000-CMAQ multiscale-nested air quality model relating the sensitivity regimes for ozone (O₃)-nitrogen oxides (NO_x)-volatile organic compounds (VOCs) in an area of high geographical complexity, like the industrial area of Tarragona, located in the NEIP. The model was applied with fine temporal (one-hour) and spatial resolution (cells of 24 km, 2 km, and 1 km) to represent the chemistry and transport of tropospheric O₃ and other photochemical species with respect to different hypothetical scenarios of emission controls and to quantify the influence of different emission sources in the area. Results indicate that O₃ chemistry in the industrial domain of Tarragona is strongly sensitive to VOCs; the higher percentages of reduction for ground-level O₃ are achieved when reducing by 25% the emissions of industrial VOCs. On the contrary, reductions in the industrial emissions of NO_x contribute to a strong increase in hourly peak levels of O₃. At the same time, the contribution of on-road traffic and biogenic emissions to ground-level O₃ concentrations in the area is negligible with respect to the pervasive weight of industrial sources. This analysis provides an assessment of the effectiveness of different policies for the control of emission of precursors by comparing the modeled results for different scenarios.     

Improving Urban Air Quality in China: Beijing Case Study

Abstract

China is undergoing rapid urbanization because of unprecedented economic growth. As a result, many cities suffer from air pollution. Two-thirds of China’s cities have not attained the ambient air quality standards applicable to urban residential areas (Grade II). Particulate matter (PM), rather than sulfur dioxide (SO₂), is the major pollutant reflecting the shift from coal burning to mixed source pollution. In 2002, 63.2 and 22.4% of the monitored cities have PM and SO₂ concentrations exceeding the Grade II standard, respectively. Nitrogen oxides (NO_x) concentration kept a relatively stable level near the Grade II standard in the last decade and had an increasing potential in recent years because of the rapid motorization. In general, the air pollutants emission did not increase as quickly as the economic growth and energy consumption, and air quality in Chinese cities has improved to some extent. Beijing, a typical representative of rapidly developing cities, is an example to illustrate the possible options for urban air pollution control. Beijing’s case provides hope that the challenges associated with improving air quality can be met during a period of explosive development and motorization.     

Ozone Predictions in Atlanta,Georgia: Analysis of the 1999 Ozone Season

ABSTRACT

Ozone prediction has become an important activity in many U.S. ozone nonattainment areas. In this study, we describe the ozone prediction program in the Atlanta metropolitan area and analyze the performance of this program during the 1999 ozone-forecasting season. From May to September, a team of 10 air quality regulators, meteorologists, and atmospheric scientists made a daily prediction of the next-day maximum 8-hr average ozone concentration. The daily forecast was made aided by two linear regression models, a 3-dimensional air quality model, and the no-skill ozone persistence model. The team's performance is compared with the numerical models using several numerical indicators. Our analysis indicated that (1) the team correctly predicted next-day peak ozone concentrations 84% of the time, (2) the two linear regression models had a better performance than a 3-dimensional air quality model, (3) persistence was a strong predictor of ozone concentrations with a performance of 78%, and (4) about half of the team's wrong predictions could be prevented with improved meteorological predictions.     

Coupling air quality and land use modeling within an optimization framework

Land use regulations and air quality standards can be effective tools to control air pollution. Atmospheric transport/chemistry simulation models could be used to develop suitable regulations and standards; however, these models are not as efficient as air quality management models developed from embedding governing equations for atmospheric transport/chemistry into an optimization framework. Formulations of two steady-state air quality management models are presented to facilitate the development or evaluation of land use strategies to protect regional air quality from pollution generated from distributed point or nonpoint sources. Both models are linear programs constructed with equations that describe steady-state atmospheric pollutant fate and transport. The first model determines feasible pollutant loading patterns for multiple land use activities to accommodate the greatest regional population. The second model ascertains patterns of expanded land use which have a minimum impact on air quality. The primary goal of this paper is to explain how air pollution and land use modeling may be coupled to create an effective management tool to aid scientists and engineers with decisions affecting air quality and land use. The secondary goal is to show the types of air quality and regulatory information which could be obtained from these models. This latter goal is attained with general conclusions as consequence of applying ‘duality theory.’     

Incinerator Conference Covers Emissions Control Aspects

The success of the application of computer modeling to decision-making will depend on the degree to which the scientifically valid “cause-and-effect” features of the air pollution system are represented. For this reason, dynamic simulation models are to be preferred to statistical and empirical models. A digital simulation model based on a stoichiometrically logical chemical mechanism and trajectory estimating routines was constructed, using Los Angeles source, meteorological and geographic input. The basic physical concept underlying the simulation model is the process of evolution of photochemical pollution in a parcel of air as it moves in a dynamic urban emission/meteorological environment along a given urban wind trajectory. Both the photochemical evolution and the trajectory are numerically integrated by a standard linear multistep predictor-corrector method. Concentrations of photochemical reactants and products (i.e., primary and secondary contaminants) are determined by this numerical integration, which also includes appropriate terms for relevant effects. In five preliminary validation runs, simulated NO₂, NO, and O₃ values were within 20% or 0.05 ppm of those observed at air monitoring stations located near the termini of the runs. The trajectories were plotted on the basis of hourly meteorological data for 22 stations. Six control strategy exercises were conducted to illustrate the application of the model to problem-solving situations.