An adapted vector autoregressive expectation maximization imputation algorithm for climate data networks - Washington, Benjamin J.; Seymour, Lynne (Wiley interdisciplinary reviews. Computational statistics, 2020)
Missingness in historical climate data networks is a pervasive phenomenon due to the conditions under which these measurements are made. Accurate estimation of these data is a critical issue as projections of future climate depend on a reliable historical climate record. After all, how can the impact of climate change be reliably forecasted when a large proportion of historical climate records are permeated with missing data? We propose an iterative multivariate infilling algorithm and explore its effectiveness on three United States temperature networks of varying densities (number of stations per unit area). Although other multivariate data are not explored here, the application of this infilling procedure is not restricted to climate networks exclusively. We also explore its utility as a function of the type of missing data (i.e., whether or not there is a mechanism or pattern behind the missing observations themselves) and the proportion of missing data within the network. As one may expect, we observe a slightly smaller root mean square error (RMSE) for temperature networks with more stations and less missingness. Somewhat surprisingly, the RMSE tends to be lower for data which is missing at random (there is some mechanism behind the missing data) rather than for data that is missing completely at random (no mechanism behind the missing data). Not surprisingly, the RMSE is largest for data which is missing not at random (missing data are directly related to the values of the observations themselves). Perhaps the most surprising result observed in these networks is that the inclusion of lagged temperature data does not necessarily improve the accuracy of data imputation. The typical RMSE for monthly minimum temperature networks varies from around 1.5 to 2.5°C. This article is categorized under: Data: Types and Structure > Massive Data Data: Types and Structure > Image and Spatial Data Data: Types and Structure > Time Series, Stochastic Processes, and Functional Data Applications of Computational Statistics > Computational Climate Change and Numerical Weather Forecasting The RMSE of an iterative adapted vector autoregressive expectation–maximization algorithm as a function of Iteration (left) and autoregressive order (right) for three types of missing data MCAR (top), MAR (middle) and MNAR (bottom).