Performance evaluation of NOAA-EPA developmental aerosol forecasts

To aid air quality model development and assess air quality forecasts, the Meteorological Development Laboratory (MDL) provided categorical verification metrics for developmental aerosol predictions. The National Air Quality Forecasting Capability (NAQFC) generated 48 h (of) gridded hourly developmental predictions for the lower 48 states (CONUS) domain in 12 km horizontal spacing. The NAQFC uses the North American Mesoscale (NAM) model with EPA’s Community Multiscale Air Quality (CMAQ) model to produce predictions of ground level aerosol concentrations. We used bilinear interpolation to calculate predicted daily maximum values at the location of the observation sites. We compared these interpolated predicted values to the observed daily maximum to produce 2 × 2 contingency tables, with a threshold of 40 μg/m³ during the months of March–August, 2007. The model showed some degree of skill in predicting aerosol exceedances. These results are preliminary as the NAQFC model for aerosol prediction is in the developmental stage. A more comprehensive performance evaluation will be accomplished in 2008, when more data become available. Our verification metrics included categorical analyses for Fraction Correct (FC) or percent correct (FC × 100), Threat Score (TS) or Critical Success Index (CSI), Probability of Detection (POD), and the False Alarm Rate (FAR), Mean Absolute Error (MAE) and mean algebraic error or bias, where bias is forecast minus observation. Graphic products included weekly statistics for the CONUS displayed in the form of bar charts, scatterplots, and graphs. In addition, we split the CONUS into six geographic regions and provided regional statistics on a monthly basis. MDL produced spatial maps of daily 1-h maximum predicted aerosol values overlaid with the corresponding point observations. MDL also provided spatial maps of the daily maximum of the 24-h running average. We derived the 24-h running average from the 1-h average predicted aerosol values and observations.