Data journey

The journey begins

At 09.00 UTC (10.00am British Summer Time) on 24 June 2014 our weather datum’s “Secret Life” began when a temperature of 18.5°C was observed at Sheffield Weston Park weather station by two different sets of equipment.

One datum was generated using equipment owned by the Weston Park Museum. The other datum was produced by equipment owned by the Met Office (the UK’s meteorological service). The Met Office has been automatically transmitting weather observations from this site to its data centre every 10 minutes since 2008.

Despite the addition of the automated Met Office equipment to the weather station compound, the museum’s observation data still play an important role in climate science and meteorology. We’ll therefore begin by focusing on the production and distribution of the museum data, and its intersection with Met Office generated data during a monthly quality control process conducted at Weston Park.

Producing the Museum dataset

The Weston Park Museum climate station has been observing the weather since 1882.

Observations for climate data are made according to a strict routine. Daily data are collected at 9am GMT during winter months and 10am BST during summer months. Historically these observations have been recorded in a Met Office observers’ Blue Book – Metform 3100. They are recorded according to the strict conventions and measurements detailed in The Observer’s Handbook. Completed records are retained in both the Weston Park Museum and Met Office archives.

These data are generated using the museum’s weather station equipment, which has been automated since around 2000. Since the installation of this automated equipment, hourly readings are taken from which key climate data can be calculated automatically.

All of these weather observation data – including our temperature reading of 18.5°C on 24 June – are automatically written to CSV files and uploaded daily into an Access database by curator Alistair McLean. This database is specifically designed for the purpose of maintaining the station’s weather records and disseminating data and reports to others who are interested in the data.

Ensuring quality

As they are uploaded to the museum’s Access database, data are converted into standardised units. This ensures consistency with Met Office conventions and with observations made in previous years.

Different instruments made by different manufacturers can vary in the type of units they measure (for example – metric or imperial measurements). Changes in instruments over the years can therefore mean changes in unit measurements. This standardisation process means that all data are recorded in the database using the same unit of measurement.

Once the temperature datum has been standardised, the dataset it belongs to is automatically checked for errors. There are acceptable tolerance levels for each type of weather observation. A computer looks at whether our temperature reading is much higher or lower than expected compared with seasonal expectations, manual observations, or other recent observations.

If an error is spotted it is investigated by Alistair and corrected if necessary. The data set is annotated with a record of any changes Alistair needs to make.

Errors in the data can be caused by various things including equipment malfunction and vandalism. If an equipment problem is found then Alistair will ensure it is dealt with in order to maintain quality.

Once a month the museum data come into contact with the Met Office data during a quality control process. Every month, the Met Office calculates its own summaries of climate data from its data and shares these with Alistair who compares the Met Office summary with the museum generated data. Any significant differences are reported back to the Met Office, along with any additional data from the museum’s equipment such as the daily weather diary. This checking process aims to spot any problems with the data being generated by either of the two sets of equipment.

Sharing the data

Even though the Met Office now collects its own data automatically from Weston Park, Alistair continues to supply them with monthly climate data generated by the museum equipment.

In addition to supplying these climate data to the Met Office, Weston Park museum shares its weather data in a number of ways. Every day, Alistair creates a report of weather observations for the last 24 hours and a forecast for the next 24 hours. He sends this out to local organisations such as Sheffield Town Hall, local libraries, local media and the universities.

His report also gets sent to a number of other paying subscribers, and is posted to Twitter for use by the general public.

Every week Alistair also sends a weekly summary to a firm called WeatherNet for a fee. WeatherNet aggregate weather observation data from across the UK and supply it to commercial clients in the legal, insurance and construction sectors.

Monthly and annual reports based on data sent to the Met Office are also compiled by Alistair and supplied to email subscribers and social media followers.

In addition to these regular reports, Alistair also shares data with people on an ad hoc basis. Requests for data come from a diverse range of people including members of the public trying to corroborate insurance claims, university students and researchers.