Evaluating the Generalizability of Foundation Models for Extreme Environmental Events: Case Study of California Wildfire PM2.5
Yongcan Huang, Li Jiang, Ze Yu Liu
Why It Matters
What makes this one worth your time
Understanding the limitations of foundation models in extreme environmental conditions is crucial for public health forecasting and resource allocation during wildfire events.
Traditional models outperform foundation models in predicting hazardous PM2.5 levels from wildfires.
Summary
The paper systematically benchmarks six time series foundation models against traditional models for forecasting PM2.5 concentrations during California wildfires, revealing that traditional models outperform foundation models in this context.
Key contributions
- A comprehensive benchmark of six TSFM configurations against traditional forecasting models.
- Evaluation of generalization capabilities using a leave-one-incident-out protocol.
- Identification of performance metrics that highlight the superiority of traditional models in extreme conditions.
Notable insights
- Zero-shot TSFMs show only modest improvements over naive persistence, indicating their limitations in extreme forecasting scenarios.
- LoRA fine-tuning significantly enhances model stability, suggesting that adaptation techniques are essential for practical deployment.
Possible limitations
- The abstract does not address potential variations in model performance across different environmental conditions or the scalability of the findings beyond California.
- Not stated in the abstract.
Abstract
arXiv:2607.07951v1 Announce Type: new Abstract: Wildfire smoke events produce extreme PM$_{2.5}$ concentrations that pose severe public health risks, yet forecasting rare, hazardous-level spikes remains a fundamental challenge. Time series foundation models (TSFMs), pretrained models offering zero-shot inference and efficient adaptation, perform strongly on general benchmarks, but their behavior under extreme out-of-distribution conditions is poorly understood. We present the first systematic benchmark comparing six TSFM configurations (zero-shot TimesFM, Chronos-2, Moirai-2, and Time-MoE, plus LoRA fine-tuned Chronos-2 and Time-MoE) against fully-trained baselines (LSTM, BiLSTM, Transformer) and naive persistence on a 12-year (2013--2025) hourly PM$_{2.5}$ dataset covering 1,375 wildfire incidents across 79 California monitoring sites. A leave-one-incident-out (LOIO) protocol evaluates generalization to unseen fires, using MAE, RMSE, and exceedance F1 at EPA AQI thresholds across 6-, 12-, and 24-hour horizons. Results reveal a consistent hierarchy. The BiLSTM achieves the lowest MAE ($5.16\,\mu g/m^3$) and the highest exceedance F1 at every threshold, including the Hazardous band ($>225.5\,\mu g/m^3$), reaching 0.63 versus at most 0.54 for any foundation model. Zero-shot TSFMs improve on persistence only modestly, and zero-shot Chronos-2 exhibits severe RMSE tail instability ($23.4\,\mu g/m^3$, negative $R^2$) from sporadic large errors. LoRA fine-tuning substantially improves both adapted families and largely repairs this instability, yet no foundation model surpasses the trained recurrent baselines on any metric. These findings challenge the assumption that larger pretrained models universally dominate environmental forecasting and provide actionable deployment guidance for wildfire air quality prediction.