Guidance for the Performance Evaluation of Three-Dimensional Air Quality Modeling Systems for Particulate Matter and Visibility

ABSTRACT

Guidance for the performance evaluation of three-dimensional air quality modeling systems for particulate matter and visibility is presented. Four levels are considered: operational, diagnostic, mechanistic, and probabilistic evaluations. First, a comprehensive model evaluation should be conducted in at least two distinct geographical locations and for several meteorological episodes. Next, streamlined evaluations can be conducted for other similar applications if the comprehensive evaluation is deemed satisfactory. In all cases, the operational evaluation alone is insufficient, and some diagnostic evaluation must always be carried out. Recommendations are provided for designing field measurement programs that can provide the data needed for such model performance evaluations.     

Rethinking the assessment of photochemical modelin systems in air quality planning applications

The U.S. Environmental Protection Agency provides guidelines for demonstrating that future 8-hr ozone (O3) design values will be at or below the National Ambient Air Quality Standards on the basis of the application of photochemical modeling systems to simulate the effect of emission reductions. These guidelines also require assessment of the model simulation against observations. In this study, we examined the link between the simulated relative responses to emission reductions and model performance as measured by operational evaluation metrics, a part of the model evaluation required by the guidance, which often is the cornerstone of model evaluation in many practical applications. To this end, summertime O3 concentrations were simulated with two modeling systems for both 2002 and 2009 emission conditions. One of these two modeling systems was applied with two different parameterizations for vertical mixing. Comparison of the simulated base-case 8-hr daily maximum O3 concentrations showed marked model-to-model differences of up to 20 ppb, resulting in significant differences in operational model performance measures. In contrast, only relatively minor differences were detected in the relative response of O3 concentrations to emission reductions, resulting in differences of a few ppb or less in estimated future year design values. These findings imply that operational model evaluation metrics provide little insight into the reliability of the actual model application in the regulatory setting (i.e., the estimation of relative changes). In agreement with the guidance, it is argued that more emphasis should be placed on the diagnostic evaluation of O3-precursor relationships and on the development and application of dynamic and retrospective evaluation approaches in which the response of the model to changes in meteorology and emissions is compared with observed changes. As an example, simulated relative O3 changes between 1995 and 2007 are compared against observed changes. It is suggested that such retrospective studies can serve as the starting point for targeted diagnostic studies in which individual aspects of the modeling system are evaluated and refined to improve the characterization of observed changes.     

High Volume Sampling:Evaluation of an Inverted Sampler for Ambient TSP Measurements

Recent advances in the development of receptor-oriented source apportionment techniques (models) have provided a new approach to evaluating the performance of particulate dispersion models. Rather than limiting performance evaluations to comparisons of particulate mass, receptor model estimates of source impacts can be used to open new opportunities for in-depth analysis of dispersion model performance. Recent experiences in the joint application of receptor and dispersion models have proven valuable in developing increased confidence in source impact projections used for control strategy development. Airshed studies that have followed this approach have identified major errors in emission inventory data bases and provided technical support for modeling assumptions.

This paper focuses on the joint application of dispersion and receptor models to particulate source impact analysis and dispersion model performance and evaluation. The limitations and advantages of each form of modeling are reviewed and case studies are examined. The paper is offered to provide several new perspectives into the model evaluation process in the hope that they may prove useful to those that manage our nation’s air resources.     

Air Quality Model Evaluation and Uncertainty

The many advances made in air quality model evaluation procedures during the past ten years are discussed and some components of model uncertainty presented. Simplified statistical procedures for operational model evaluation are suggested. The fundamental model performance measures are the mean bias, the mean square error, and the correlation. The bootstrap resampling technique is used to estimate confidence limits on the performance measures, In order to determine if a model agrees satisfactorily with data or if one model is significantly different from another model. Applications to two tracer experiments are described.

It is emphasized that review and evaluation of the scientific components of models are often of greater Importance than the strictly statistical evaluation. A necessary condition for acceptance Of a model should be that it is scientifically correct. It Is shown that even in research-grade tracer experiments, data Input errors can cause errors In hourly-average model predictions of point concentrations almost as large as the predictions themselves. The turbulent or stochastic component of model uncertainty has a similar magnitude. These components of the uncertainty decrease as averaging time increases.     

Space-time analysis of the Air Quality Model Evaluation International Initiative (AQMEII) Phase 1 air quality simulations

This study presents an evaluation of summertime ozone concentrations over North America (NA) and Europe (EU) using the database generated from Phase 1 of the Air Quality Model Evaluation International Initiative (AQMEII). The analysis focuses on identifying temporal and spatial features that can be used to stratify operational model evaluation metrics and to test the extent to which the various modeling systems can replicate the features seen in the observations. Using a synoptic map typing approach, it is demonstrated that model performance varies with meteorological conditions associated with specific synoptic-scale flow patterns over both eastern NA and EU. For example, the root mean square error of simulated daily maximum 8-hr ozone was twice as high when cloud fractions were high compared with when cloud fractions were low over eastern NA. Furthermore, results show that over both NA and EU the regional models participating in AQMEII were able to better reproduce the observed variance in ambient ozone levels than the global model used to specify chemical boundary conditions, although the variance simulated by almost all regional models is still less that the observed variance on all spatiotemporal scales. In addition, all modeling systems showed poor correlations with observed fluctuations on the intraday time scale over both NA and EU. Furthermore, a methodology is introduced to distinguish between locally influenced and regionally representative sites for the purpose of model evaluation. Results reveal that all models have worse model performance at locally influenced sites. Overall, the analyses presented in this paper show how observed temporal and spatial information can be used to stratify operational model performance statistics and to test the modeling systems’ ability to replicate observed temporal and spatial features, especially at scales the modeling systems are designed to capture.

Implications: The analyses presented in this paper demonstrate how observed temporal and spatial information can be used to stratify operational model performance and to test the modeling systems’ ability to replicate observed temporal and spatial features. Decisions for the improvement of regional air quality models should be based on the information derived from only regionally representative sites.     

Evaluation of a CMAQ simulation at high resolution over the UK for the calendar year 2003

A comprehensive ‘operational’ evaluation of the performance of the Community Multiscale Air Quality (CMAQ) modelling system version 4.6 was conducted in support of pollution assessment in the UK for the calendar year 2003. The model was run on multiple grids using one-way nests down to a horizontal resolution as fine as 5 km over the whole of the UK. The model performance was evaluated for pollutants with standards and limit values (e.g. O₃, PM₁₀) and species contributing to acidic and nitrogenous deposition (e.g. NH₃, SO₄^2–, NO₃^–, NH₄⁺) against data from operational national monitoring networks. The key performance characteristics of the modelling system were found to be variable according to acceptance criteria and to depend on the type (e.g. urban, rural) and location of the sites, as well as on the time of the year. As regards the techniques that were used for ‘operational’ evaluation, performance generally complied with expected levels and ranged from good (e.g. O₃, SO₄^2–) to moderate (e.g. PM₁₀, NO₃^–). At a few sites low correlations and large standard deviations for some species (e.g. SO₂) suggest that these sites are subject to local factors (e.g. topography, emission sources) that are not well described in the model. Overall, the model tends to over predict O₃ and under predict aerosol species (except SO₄^2–). Discrepancies between predicted and observed concentrations may be due to a variety of intertwined factors, which include inaccuracies in meteorological predictions, chemical boundary conditions, temporal variability in emissions, and uncertainties in the treatment of gas and aerosol chemistry. Further work is thus required to investigate the respective contributions of such factors on the predicted concentrations.     

Recommendations on statistics and benchmarks to assess photochemical model performance

Photochemical grid models are addressing an increasing variety of air quality related issues, yet procedures and metrics used to evaluate their performance remain inconsistent. This impacts the ability to place results in quantitative context relative to other models and applications, and to inform the user and affected community of model uncertainties and weaknesses. More consistent evaluations can serve to drive improvements in the modeling process as major weaknesses are identified and addressed. The large number of North American photochemical modeling studies published in the peer-reviewed literature over the past decade affords a rich data set from which to update previously established quantitative performance “benchmarks” for ozone and particulate matter (PM) concentrations. Here we exploit this information to develop new ozone and PM benchmarks (goals and criteria) for three well-established statistical metrics over spatial scales ranging from urban to regional and over temporal scales ranging from episodic to seasonal. We also recommend additional evaluation procedures, statistical metrics, and graphical methods for good practice. While we primarily address modeling and regulatory settings in the United States, these recommendations are relevant to any such applications of state-of-the-science photochemical models. Our primary objective is to promote quantitatively consistent evaluations across different applications, scales, models, model inputs, and configurations. The purpose of benchmarks is to understand how good or poor the results are relative to historical model applications of similar nature and to guide model performance improvements prior to using results for policy assessments. To that end, it also remains critical to evaluate all aspects of the model via diagnostic and dynamic methods. A second objective is to establish a means to assess model performance changes in the future. Statistical metrics and benchmarks need to be revisited periodically as model performance and the characteristics of air quality change in the future.

Implications: We address inconsistent procedures and metrics used to evaluate photochemical model performance, recommend a specific set of statistical metrics, and develop updated quantitative performance benchmarks for those metrics. We promote quantitatively consistent evaluations across different applications, scales, models, inputs, and configurations, thereby (1) improving the user’s ability to quantitatively place results in context and guide model improvements, and (2) better informing users, regulators, and stakeholders of model uncertainties and weaknesses prior to using results for policy assessments. While we primarily address U.S. modeling and regulatory settings, these recommendations are relevant to any such applications of state-of-the-science photochemical models.     

Sensitivity analysis and evaluation of microFacCO: a microscale motor vehicle emission factor model for CO emissions

This paper presents a sensitivity analysis of a microscale emission factor model (MicroFacCO) for predicting real-time site-specific motor vehicle CO emissions to input variables, as well as a limited field study evaluation of the model. The sensitivity analysis has shown that MicroFacCO emission estimates are very sensitive to vehicle fleet composition, speed, and ambient temperature. For the present U.S. traffic fleet, the CO emission rate (g/mi) is increased by more than 500% at 5 mph in comparison with a speed greater than 40 mph and by approximately 67% at ambient temperatures of 45 degrees F and > or = 95 degrees F in comparison with an ambient temperature of 75 degrees F. The input variable "emission failure standard rate" is more sensitive to estimating emission rates in the 1990s than in the 2000s. The estimation of emission rates is not very sensitive to relative humidity. MicroFacCO can also be applied to examine the contribution of emission rates per vehicle class and model year. The model evaluation is presented for tunnel studies at five locations. In general, this evaluation study found good agreement between the measured and the modeled emissions. These analyses and evaluations have identified the need for additional studies to update the high-speed (>35 mph) air conditioning (A/C) correction factor and to add effects due to road grades. MicroFacCO emission estimates are very sensitive to the emission standard failure rate. Therefore, the model performance can be greatly improved by using a local emission standard failure rate.     

A suggested method for dispersion model evaluation

Too often operational atmospheric dispersion models are evaluated in their ability to replicate short-term concentration maxima, when in fact a valid model evaluation procedure would evaluate a model's ability to replicate ensemble-average patterns in hourly concentration values. A valid model evaluation includes two basic tasks: In Step 1 we should analyze the observations to provide average patterns for comparison with modeled patterns, and in Step 2 we should account for the uncertainties inherent in Step 1 so we can tell whether differences seen in a comparison of performance of several models are statistically significant. Using comparisons of model simulation results from AERMOD and ISCST3 with tracer concentration values collected during the EPRI Kincaid experiment, a candidate model evaluation procedure is demonstrated that assesses whether a model has the correct total mass at the receptor level (crosswind integrated concentration values) and whether a model is correctly spreading the mass laterally (lateral dispersion), and assesses the uncertainty in characterizing the transport. The use of the BOOT software (preferably using the ASTM D 6589 resampling procedure) is suggested to provide an objective assessment of whether differences in model performance between models are significant.

Implications:

Regulatory agencies can choose to treat modeling results as “pseudo-monitors,” but air quality models actually only predict what they are constructed to predict, which certainly does not include the stochastic variations that result in observed short-term maxima (e.g., arc-maxima). Models predict the average concentration pattern of a collection of hours having very similar dispersive conditions. An easy-to-implement evaluation procedure is presented that challenges a model to properly estimate ensemble average concentration values, reveals where to look in a model to remove bias, and provides statistical tests to assess the significance of skill differences seen between competing models.     

Sensitivity analysis and evaluation of MicroFacPM: a microscale motor vehicle emission factor model for particulate matter emissions

A microscale emission factor model (MicroFacPM) for predicting real-time site-specific motor vehicle particulate matter emissions was presented in the companion paper titled "Development of a Microscale Emission Factor Model for Particulate Matter (MicroFacPM) for Predicting Real-Time Motor Vehicle Emissions". The emission rates discussed are in mass per unit distance with the model providing estimates of fine particulate matter (PM2.5) and coarse particulate matter. This paper complements the companion paper by presenting a sensitivity analysis of the model to input variables and evaluation model outputs using data from limited field studies. The sensitivity analysis has shown that MicroFacPM emission estimates are very sensitive to vehicle fleet composition, speed, and the percentage of high-emitting vehicles. The vehicle fleet composition can affect fleet emission rates from 8 mg/mi to 1215 mg/mi; an increase of 5% in the smoking (high-emitting) current average U.S. light-duty vehicle fleet (compared with 0%) increased PM2.5 emission rates by -272% for 2000; and for the current U.S. fleet, PM2.5 emission rates are reduced by a factor of -0.64 for speeds >50 miles per hour (mph) relative to a speed of 10 mph. MicroFacPM can also be applied to examine the contribution of emission rates per vehicle class, model year, and sources of PM. The model evaluation is presented for the Tuscarora Mountain Tunnel, Pennsylvania Turnpike, PA, and some limited evaluations at two locations: Sepulveda Tunnel, Los Angeles, CA, and Van Nuys Tunnel, Van Nuys, CA. In general, the performance of MicroFacPM has shown very encouraging results.     

Performance Characteristics of Instrumental Methods for Monitoring Sulfur Dioxide

The performance characteristics of commercially available continuous sulfur dioxide monitors were determined. Conductimetric, colorimetric, and coulometric analyzers were investigated. The study was conducted to develop information on such instrument characteristics as stability, sensitivity, response time, collection efficiency, and response to interfering substances. The methodology and apparatus used to determine each operational parameter are described. The instrument performance data developed in this study should be useful to those involved in selection of instrumental methods for monitoring atmospheric sulfur dioxide.     

A Qualitative Approach to Evaluating the Anticipated Reliability of a Photochemical Air Quality Simulation Model for a Selected Application

ABSTRACT

Photochemical air quality simulation models are now used widely in evaluating the merits of alternative emissions control strategies on spatial scales from metropolitan to sub-continental. Greatly varying levels of resources have been available to support modeling, from relatively comprehensive databases and evaluation of performance to a paucity of aerometric data for developing model inputs. Where data are sparse, many alternative outcomes are consistent with the knowledge at hand. Where performance evaluation is inadequately supported, the probability of error may be high. In each instance, uncertainties may be large when compared with the signal of interest, and thus confidence in the reliability of the model as an estimator of future air quality may come into question.

This paper proposes a qualitative procedure for assessing whether a particular application of a modeling system is likely to be potentially unreliable, suggesting that either (1) modification and further evaluation is needed, if supportable, prior to adoption for regulatory application; or (2) the model should not be used if improvement is not supportable. The procedure is proposed for use by policy-makers, staffs of public agencies, air quality managers, environmental staffs of industrial organizations, and other interested parties. The proposed use of the procedure is (1) to assess, a priori, whether a proposed application is likely to be judged questionable or unacceptably uncertain in outcome; and (2) to provide, a posteriori, a basis for judging quickly the likely quality of model performance. The procedure is presented with tropospheric ozone as the pollutant of concern. With adjustments, however, the procedure should be applicable for particu-late matter and other pollutants of interest.     

Validation of the lagrangian particle dispersion model FLEXPART against large-scale tracer experiment data

A comprehensive validation of FLEXPART, a recently developed Lagrangian particle dispersion model based on meteorological data from the European Centre for Medium-Range Weather Forecasts, is described in this paper. Measurement data from three large-scale tracer experiments, the Cross-Appalachian Tracer Experiment (CAPTEX), the Across North America Tracer Experiment (ANATEX) and the European Tracer Experiment (ETEX) are used for this purpose. The evaluation is based entirely on comparisons of model results and measurements paired in space and time. It is found that some of the statistical parameters often used for model validation are extremely sensitive to small measurement errors and should not be used in future studies. 40 cases of tracer dispersion are studied, allowing a validation of the model performance under a variety of different meteorological conditions. The model usually performs very well under undisturbed meteorological conditions, but it is less skilful in the presence of fronts. The two ETEX cases reveal the full range of the model’s skill, with the first one being among the best cases studied, and the second one being, by far, the worst. The model performance in terms of the statistical parameters used stays rather constant with time over the periods (up to 117 h) studied here. It is shown that the method used to estimate the concentrations at the receptor locations has a significant effect on the evaluation results. The vertical wind component sometimes has a large influence on the model results, but on the average only a slight improvement over simulations which neglect the vertical wind can be demonstrated. Subgrid variability of mixing heights is important and must be accounted for.     

基于二级模糊综合评价的白洋淀淀中村生活污水处理技术决策方法

本研究旨在为白洋淀淀中村生活污水处理技术选择提供一个优化决策方法。在对国内外农村生活污水处理技术深入调研的基础上,采用二级模糊综合评价法,对14项预选处理技术进行综合评估。首先从处理效果、技术性能以及经济性能三方面对各处理技术进行初级量化评估,再根据其评估数据对各项技术进行二级评估,最后结合淀中村自身特点筛选出较好的生活污水处理技术,探讨最适合白洋淀淀中村生活污水处理的优化方案。纯水村人口相对集中地区可采用人工湿地和水解酸化+人工湿地技术,人口相对分散地区可采用沼气净化技术;半水村人口相对集中地区可采用厌氧滤池+土壤渗滤和人工湿地+氧化塘技术,人口相对分散地区可采用沼气净化技术和土壤渗滤技术。     

Assimilation of conventional and satellite wind observations in a mesoscale atmospheric model for studying atmospheric dispersion

A mesoscale atmospheric model PSU/NCAR MM5 is used to provide operational weather forecasts for a nuclear emergency response decision support system on the southeast coast of India. In this study the performance of the MM5 model with assimilation of conventional surface and upper-air observations along with satellite derived 2-d surface wind data from QuickSCAT sources is examined. Two numerical experiments with MM5 are conducted: one with static initialization using NCEP FNL data and second with dynamic initialization by assimilation of observations using four dimensional data assimilation (FDDA) analysis nudging for a pre-forecast period of 12 h. Dispersion simulations are conducted for a hypothetical source at Kalpakkam location with the HYSPLIT Lagrangian particle model using simulated wind field from the above experiments. The present paper brings out the differences in the atmospheric model predictions and the differences in dispersion model results from control and assimilation runs. An improvement is noted in the atmospheric fields from the assimilation experiment which has led to significant alteration in the trajectory positions, plume orientation and its distribution pattern. Sensitivity tests using different PBL and surface parameterizations indicated the simple first order closure schemes (Blackadar, MRF) coupled with the simple soil model have given better results for various atmospheric fields. The study illustrates the impact of the assimilation of the scatterometer wind and automated weather stations (AWS) observations on the meteorological model predictions and the dispersion results.     

A modeling study of fouling development in membrane bioreactors for wastewater treatment.

Membrane fouling is a primary concern in membrane bioreactors (MBRs) in wastewater treatment because it strongly affects both system stability and economic feasibility. A mathematical model was developed in this study for membrane fouling in submerged MBR systems for wastewater treatment, in which both reversible and irreversible fouling were quantified. While mixed liquor suspended solids are the major components of the reversible fouling layer, dissolved organic matter is thought to be the key foulant, in particular, responsible for the long-term irreversible fouling of the filtration unit. The model was calibrated (parameter identification) with a set of operational data from a pilot MBR and then verified with other independent operational data from the MBR. The good agreement between theoretical predictions and operational data demonstrates that the outlined modeling concept can be successfully applied to describe membrane fouling in submerged MBR systems.     

Determination of Baghouse Performance from Coal and Ash Properties: Part II

Baghouse performance at utility coal-fired power plants is determined by baghouse design, operating procedures, and the characteristics of the ash that is collected as a dustcake on the fabric filter. The Electric Power Research Institute has conducted laboratory research to identify the fundamental variables that influence baghouse performance. A database was assembled including measured characteristics of coal and dustcake ash, and data describing operating parameters and performance of full-scale and pilotscale baghouses. Predictions of performance can be based on physical characteristics of the ash to be filtered (discussed in Part I of this article), as well as chemical characterizations of the ash, or empirical correlations with the alkali content of the source coal The effects of design and operational variables can be included in these predictions. Baghouse performance can be optimized by exercising proper operating practices and by selecting a filtering fabric and cleaning method matched to the cohesivity of the ash to be collected.     

An updated state of the science EQC model for evaluating chemical fate in the environment: application to D5 (decamethylcyclopentasiloxane)

The EQuilibrium Criterion (EQC) model developed and published in 1996 has been widely used for screening level evaluations of the multimedia, fugacity-based environmental fate of organic chemicals for educational, industrial, and regulatory purposes. Advances in the science of chemical partitioning and reactivity and the need for more rigorous regulatory evaluations have resulted in a need to update the model. The New EQC model is described which includes an improved treatment of input partitioning and reactivity data, temperature dependence and an easier sensitivity and uncertainty analysis but uses the same multi-level approach, equations and environmental parameters as in the original version. A narrative output is also produced. The New EQC model, which uses a Microsoft Excel platform, is described and applied in detail to decamethylcyclopentasiloxane (D5; CAS No. 541-02-6). The implications of these results for the more detailed exposure and risk assessment of D5 are discussed. The need for rigorous evaluation and documentation of the input parameters is outlined.     

Recent Advances in Liquid Scrubbing for Fine Particulate Collection

Lone Star Steel has developed a family of high performance, long life, low capital, and low operating cost Hydro-Sonic* cleaners. A choice of driving mediums is available; either a compressible fluid ejector or fan may be used. The ejector models have no moving parts in the gas stream. Where available, waste heat may be used to power the cleaners. Cleaning levels are adjustable by the user by the selective input of energy. The Coalescer model employs a unique pressure recovery device for separation thereby reducing energy requirements for the cleaner. The Hydro-Sonic cleaners are operational In the steel, paper, and other industries.     

Global circulation of atmospheric mercury: a modelling study

This paper presents a comprehensive atmospheric global and regional mercury model and its capability in describing the atmospheric cycling of mercury. This is an on-line model (integrated within the Canadian operational environmental forecasting and data assimilation system) which can be used to understand the role of meteorology in mercury cycling (atmospheric pathways), the inter-annual variability of mercury and can be evaluated against observations on global scales. This is due to the fact that the model uses a combination of actual observed and predicted meteorological state of the atmosphere at high resolution to integrate the model as opposed to the climatological approach used in existing global mercury models. The model was integrated and evaluated on global scale using only anthropogenic emissions. North to south gradients in mercury concentrations, seasonal variability, dry and wet deposition and vertical structure are well simulated by the model. The model was used to explain the observed seasonal variations in atmospheric mercury circulation. The results from this study include a global animation of surface air concentrations of total gaseous mercury for 1997.