Detecting and correcting sensor drifts in long-term weather data

Georg von Arx, Matthias Dobbertin, Martine Rebetez

Quality control of long-term monitoring data of thousands and millions of individual records as present in meteorological data is cumbersome. In such data series, sensor drifts, stalled values, and scale shifts may occur and potentially result in flawed conclusions if not noticed and handled properly. However, there is no established standard procedure to perform quality control of high-frequency meteorological data. In this paper, we outline a procedure to remove sensor drift in high-frequency data series using the example of 15-year-long sets of hourly relative humidity (RH) data from 28 stations subdivided into 202 individual sensor operation periods. The procedure involves basic quality control, relative homogeneity testing, and drift removal. Significant sensor drifts were observed in 40.6 % of all sensor operation periods. The drifts varied between data series and depended in a complex, usually inconsistent way on absolute RH values; within single series for instance, a drift could be negative in the lower RH range and positive in the upper RH range. Detrending changed RH values by, on average, 1.96 %. For one fifth of the detrended data, adjustments were 2.75 % and more of the measured value, and in one tenth 4.75 % and more. Overall, drifts were strongest for RH values close to 100 %. The detrending procedure proved to effectively remove sensor drifts. The principles of the procedure also apply to other meteorological parameters and more generally to any time series of data for which comparable reference data are available.