Data journey

Our temperature datum arrives at the Met Office via the World Meteorological Organisation’s global telecommunication system (GTS).

Weather and climate observation data from automated weather stations such as Weston Park in Sheffield are all transmitted via the international GTS. The UK Met Office and other national meteorological organisations then download these weather station observation data from GTS and ingest them into their own systems, alongside data from radar, weather balloons, aircraft, satellite and buoys.

Data for forecasting are the first priority. Near real-time observations from synoptic weather stations are downloaded from GTS, decoded, formatted and incorporated as quickly as possible into the numerical weather prediction (NWP) system used by weather forecasters.

Climate data from synoptic stations also arrive straightaway via GTS. However, climate data from the network of co-operating climate observers, such as Weston Park Museum, arrives at a slower pace – typically daily or monthly.

This short Met Office video discusses how the organisation processes the weather observations that arrive at its data centre from stations such as Weston Park.

Quality control

Once observation data have been received by the Met Office, they go through a rigorous quality control process. This aims to ensure consistency and accuracy, and to remove errors.

Three types of automated consistency checks are performed:

• Whether the observation is compatible with other weather variables – eg is the temperature compatible with the pressure and rainfall readings?

• Whether the observation falls within expected limits for the climate.

• Whether the observation is consistent with nearby stations.

“Looking for errors, consistency. So there’s a set of automated checks they’ll do for, you know simple things, range values, numbers that are clearly in error. They’ll take information from the automated quality control that the forecasts provide, and use that as well. So there can be all sorts of reasons why the observations will be in error, there may be a rain gauge has got stuck. Then if every other rain gauge in the vicinity has 20 millimetres and that one has zero – then you can be fairly confident that it’s probably an error.”

Any data that are flagged as inconsistent by these automated checks are then examined by trained meteorologists in a dedicated Met Office quality assurance team based in Edinburgh.

Climate observation data are typically checked on a monthly basis, and then re-assessed a few months later.

Whilst quality checks are carried out, it is important that errors are avoided as far as possible through management of the weather station site, annual site inspections and equipment maintenance.

Nearby trees, roads and buildings, and site elevation, can all impact on observation data. These characteristics are therefore recorded in the station metadata. Any movement in the location of the station is also logged and evaluated for the possible impact on the observations.

Archiving the data

Following quality control, all real time observation data are stored in the MetDB synoptic database. Climate observation data from automated stations are then integrated from MetDB into the MIDAS database (Met Office Integrated Data Archive System) twice daily. Non-automated climate records are manually added to the MIDAS database at a later date.

MIDAS is specifically designed to support the needs of various climate applications. It covers 1853 to the present and incorporates UK land and marine observations, and more recently, global climate observations data.

MetDB is the backbone of all weather forecasting undertaken by the Met Office. Similarly, MIDAS supports all Met Office climate-related activity. MIDAS and a subset of MetDB are also made available for UK-based academic research funded by the Natural Environment Research Council (NERC), via the British Atmospheric Data Centre (BADC).

National Climate data processing

The National Climate Information Centre (NCIC) based in the Met Office Hadley Centre is responsible for producing UK gridded climate datasets and a range of summaries and analyses of climate data.

“Our focus and our remit is very much UK climate monitoring, rather than global or international. So the wider team that we sit in build global datasets. So some of the well known global temperature series and so on. But our remit is very much on the UK monitoring aspect. So on a day-to-day basis we’re sort of maintaining and developing these datasets.”

Observation data from weather stations such as Weston Park in Sheffield feed into these datasets:

“So we take the data out of this MIDAS database. So all the observations for a particular day, or a particular month, and we create the climate statistics for individual stations. So for any observing stations, particularly the long running stations, we’ll produce climate statistics. So that might be the average temperature for the month, and it’s the long term averages for different seasons of the month.

“So we provide the sort of base line reference dataset for individual stations. But actually for a lot of what a lot of users are interested in is sort of integrated statistics for the county – was the UK particularly warm or cold? So we need to average up all of the observations we’ve got for the country.”

The team at NCIC work to make sure all the grid squares across the UK have a data point:

“Our approach to handling this process is that we take all of the observations and we interpolate them onto a uniformed grid. So we’re essentially making estimates of what the observation would have been at ten and a half thousand locations across the country, based on the observations we’ve got.”

They also ensure that the data set is consistent over time. The team complete a process called data homogenisation to make sure that any changes in weather stations over the years – for example, location and instruments – do not impact upon the accuracy of the observation data in the climate record:

“Now, for a climate record, obviously, one of our key considerations is the homogeneity of the record at the time. And one of the biggest sources of inhomogeneities in that, are the fact that the number and location of observing stations changes all the time.”

NCIC’s work is used by policymakers, public sector and commercial organisations interested in planning for and mitigating against climate change. For example, the national gridded climate datasets support the UKCP09 Climate Projections initiative, which aims to support researchers and decision makers tackling climate change impacts and adaptation.

Data sharing

Much of the data produced for the purpose of climate science is shared openly amongst academic researchers.

UK monthly and annual climate summaries and averages, UKCP09 gridded datasets and monthly climate records for historical weather stations such as Weston Park, are all made available for the public by the NCIC on the Met Office website.

Central repositories, such as the BADC in the UK and the National Climatic Data Center (NCDC) operated by NOAA in the USA, facilitate sharing within the scientific community.

BADC provides access to several weather and climate datasets produced by the Met Office and the Climatic Research Unit, although a licence and proof of academic research status is required to gain full access.

Selected sets of UK observations, forecast, climate and forecast data are also available as open data from the Met Office DataPoint API. They are licenced under the Open Government Licence, which means they can be downloaded and used by anyone. However, there are limitations on the volume of data that can be downloaded from DataPoint. High-volume users, such as the weather market data supplier on our map (the green line), still need to arrange a wholesale data contract with the Met Office. These contracts have conditions and fees attached.