Policy

Data policy

Data that are produced and owned by the Met Office, including those generated by the Met Office equipment at Sheffield Weston Park, are subject to a number of different policy and legislative developments that impact upon their processing and re-use as public sector data.

These include:

• EU Directive 2007/2/EC on Infrastructure for Spatial Information in the European Community (INSPIRE).

• EU Directive 2003/98/EC on the Re-use of Public Sector Information (its replacement Directive 2013/37/EU is currently being transposed into UK law).

• UK Open Government Data policy.

• Various related frameworks and schemes at the UK government level including Exceptions to Marginal Cost pricing, the UK Government Licensing Framework and the Information Fair Trader Scheme.

These data policies and legislation also intersect with broader policy areas including competition law, and the Trading Fund model of public data production which is governed by the Managing Public Money framework.

The complexity of the data that are produced and processed by the Met Office mean that compliance with the developing legislative frameworks has been a difficult task, and Met Office policies and practices are still in development in relation to some of their data holdings.

INSPIRE (Infrastructure for Spatial Information in the European Community)

EU Directive 2007/2/EC on Infrastructure for Spatial Information in the European Community, better known as INSPIRE, is a Directive that aims to develop the infrastructure for spatial information in the European Union. The types of data covered by INSPIRE include a wide range of environmental data, including some types of meteorological data.

The Directive should improve access to such data and make it easier to share and link with other data. The ultimate objective is to create a data infrastructure that can contribute to better informed environmental policy. There are a range of milestones for INSPIRE compliance, including set dates for when metadata, data and networked access to data will be available.

In order to move towards compliance with the INSPIRE Directive, the Met Office are working to overcome a number of challenges. These include developing a means for people to access their data and contributing to the development of INSPIRE standards.

“Meteorological forecast data is different in nature to a lot of other data. You issue a forecast, but your forecast then has various different time elements to it. It’s the date that it was issued, the date that it’s valid for, and then you have forecast time steps going out, you know, five days or whatever.

“So it makes it quite difficult when everyone else is just going this is the data and this is the time that it was created, and therefore that’s when it’s valid. We need more than one time thing. So we’re working with the relevant bodies that are trying to develop the standards for INSPIRE.”

Despite the challenges of INSPIRE compliance, the Met Office is confident about its ability to comply and enthusiastic about the benefits for interoperability with other organisations and easing re-use by third parties. It has therefore decided that the organisation will apply the INSPIRE principles to a wider range of data than those required under the regulations:

“But we have just started a programme of work. For making sure that by 2020 we are compliant with INSPIRE, as we should be with the various milestones as we go forward.”

“We’ve decided that yes okay, INSPIRE’s only supposed to be about your public task, but actually well, it makes sense. To actually do it across everything, as long as it’s appropriate to do it.”

Re-use of Public Sector Information

Directive 2013/37/EU on the Re-use of Public Sector Information amended the 2003 Directive (2003/98/EC) of the same name. The 2013 Directive is currently being transposed into UK law to replace the Re-use of Public Sector Information Regulations 2005. At present it is regulated in the UK by The National Archives.

The Directive covers data and information produced by public bodies (other than public service broadcasters, educational or research establishments) as part of their public task.

Since 2003, the Directive has stipulated that such data must be made available for re-use by others non-exclusively, without discrimination and in most cases at marginal cost. After some debate, the 2013 Directive still allows for above marginal cost charges for trading funds such as the Met Office, but stipulates that these charges must be calculated according to “objective, transparent and verifiable criteria”.

The Directive only applies to data and information that is produced by a public body as part of its public task. In relation to the Met Office, its public task is defined in relation to the agreed outputs of the Public Weather Service part of the organisation. These include things such as national severe weather warnings, public weather services, civil contingency services, and some international services.

Definitions of public sector organisations’ public tasks have proved to be an area of tension in recent years. Some have argued that public sector bodies do not have a well defined public task. This has meant it is sometimes unclear which data the Re-use of Public Sector Information regulations apply to. There has therefore been a push from The National Archives, as regulators of the Re-use of Public Sector Information (PSI), to make public sector bodies define their public task. This process has had both advantages and disadvantages from the Met Office’s perspective:

“Sometimes though it’s the way it’s interpreted, so it’s your public task. Okay well, how do we define our public task? Well, you define your public task. Okay. So now we’re defining our own public task that will then be used to say whether or not we’re complying with legislation or not. So in some ways, yes it is useful to have the legislation there, but then there are times when, actually, we almost have a little bit too much scope. Which makes it more difficult to kind of go, no this is clearly what we need to be doing.”

Providing access to the high volume of data underlying the Public Weather Service is a difficult task, which can make compliance with the Re-use of Public Sector Information regulations technically challenging:

“We have our raw model data... and obviously, the observations that we have. And that underpins our public weather service outputs. So for us that forms our requirement for what we need to make available. That’s big datasets – they’re being updated four times a day or more, and that’s significant volumes of data.

“We are struggling a bit at the moment because our models have got so big and so detailed. We’re now trying to work out how are we actually going to make all of this data available appropriately, now and in the future. So we’re going through a piece of work on that.”

Whilst most re-usable PSI in the UK has been charged for at marginal cost since the Cross-Cutting Review of the Knowledge Economy (2000) this has not been the case for trading funds such as the Met Office which are exempt from this policy.

The rules around charging to re-use information held by UK trading funds are stated in a policy called Managing Public Money. Maximum charges are legislated under the re-use of PSI regulations which allow for above marginal cost pricing in the case of trading funds.

Trading funds such as the Met Office must also meet the standards of the Information Fair Trading Scheme, including meeting the charging criteria outlined in the Exceptions to Marginal Cost pricing framework.

Following these various regulations and policies, the Met Office currently charges third parties for re-use of its data, although they frame these charges as service-related rather than charges for the data:

“But all of that’s charged for. They don’t really pay for the data as such, they pay for the service that goes with it. It gets to a point in the processing chain and we go right, anything from there on in is particular to them getting that data. So they’ll pay for those steps in the processing chain. Creating it in the format that is appropriate for them to get it, the message switching to get it to them, so the network charges and those sorts of things.

“We have an account manager that manages them getting it, and we have a kind of service charge. To make sure that if they have a problem with it they can call in, and those sorts of things. So the charges are around that – Rather than the actual creating data.”

The ability of the Met Office to charge above marginal cost for re-use of public sector information has been challenged from a number of directions including the open data movement and the PSI lobby which represents commercial re-users of public sector information including the weather derivatives markets. Despite this lobbying, the UK government was successful in negotiating at the EU for an option for above marginal cost charging to remain in the 2013 amendments to the Directive.

Open data

Open data policy development in the UK began under the Labour government, and continued under the Conservative-Liberal Democrat coalition government that came to power in 2010.

For data to be ‘open’ it means that anybody can re-use the data, for any purpose, at marginal cost – which for digital data is usually free of charge. Open data challenges exemptions to marginal cost pricing under the Re-use of Public Sector Information regulations.

The development of open data policies at the Met Office has been complex and contested.

One former senior policy maker we spoke to described how meteorological data were “certainly one of the early priorities” of the Open Data policy initiative under the new Labour government. This continued under the coalition government, with the Chancellor of the Exchequer announcing a significant release of open data from the Met Office in his autumn statement of 2011:

“The Met Office will, from today and for the first time, release under the Open Government Licence (OGL) as machine-readable and machine-processable for unrestricted use, the following Public Weather Service weather forecast and real-time observation datasets, which together represent the largest volume of high quality weather data and information made available by a national meteorological organisation anywhere in the world.” – Cabinet Office

In response to this push from government, the Met Office agreed that it would make open the data that was already available on its website:

“So our open data is the data that’s on our website. So that is our weather forecast data that’s paid for by the Public Weather Service... which also includes some climate data as well, so we have things like the anomalies and averages, and those sorts of things, they are also considered open data. So that was agreed with the Cabinet Office probably about two, two and a half years ago because we needed to somehow scope and we said well, if it’s on our website and people are using it then that would seem like a reasonable kind of boundary of what is open data.”

The Met Office has found adoption of the new open data policies challenging, and some policy makers have complained about the Met Office being “resistant” on open data.

Significant barriers for Met Office open data policy adoption include the trading fund business model it operates under and the complexity of the data that it generates, manages and processes. As one Met Office participant described, the organisation felt somewhat threatened by the new open data landscape:

“So we wouldn’t have chosen to have made data openly available specifically had it not been for the government’s drive for open data and the way that that was going.”

“And PWS (the Public Weather Service) were being pressed to release all of their – what was on the website that needs to be open data, you need to make all your data open. And we realised that as a trading fund that causes us issues because if we provide the data for free then it’s very difficult to... not only sell the data, but also it means that you’re opening up a huge amount of competition, they’re getting the data for free.

“Because of the way the organisation is set up we’re not as agile as we would like to be in some cases, and certainly not on the commercial side. And so they were struggling that if you just gave it all away well, we’ve got no chance.”

One critical area of contention with some policy makers has been around the opening of the Met Office’s historic data:

“I don’t think they’ve implemented in spirit or probably in letter what ministers agreed and announced they would do. And particularly I think their historical stuff, which they’ve definitely digitised.”

Currently, historic data is only available for commercial re-use through the Met Office’s commercial team, rather than as a wholesale product available under the Re-use of Public Sector Information regulations or as open data. Although, subsets of historical data are available free of charge through the Met Office library for personal and research use:

“We don’t have any historical data through our wholesale manager. There was a time when we did and it caused confusion because we have a commercial team that were also providing historical data, and the two pricing models were different. And we went this is ridiculous because actually, wholesale is not about that it’s about the provision of the stuff kind of as it happens. So we have a team that make available that commercial – that data under commercial licence fees.

“We have the library and archive though, so if it’s a member of the public or someone trying to do some research then they can go to the library, and the library will do that, a reasonable amount of extraction for them... and there’s no charges for that. But it’s done under this licence for private and research use only.”

This barrier to open historic data has been challenged by some policy makers. These policy makers have argued that open historic data is of high economic value, looking to the weather risk markets and firms such as Climate Corp that take advantage of open weather data in the USA.

“What’s less clear to me is whether the historic observation data has been opened, which was definitely a Public Weather Service product. Historical data – they’ve dragged on – but they’ve definitely digitised – it has proved in other countries to be enormous value. So, for instance Climate Corp in the US.”

You can read more about policy makers’ views on the relationship between open weather data and weather risk markets in the Weather market data supplier – Policy section. However, the fact that different policy makers and politicians see weather risk market innovation as a driver for opening data suggest that pressure is being put on the Met Office to expand its open data policy in this direction.

The government’s open data agenda has been a driving force behind Met Office developments to improve and standardise its data policy across the organisation:

“So we have a data policy steering group. The group was set up, and [a policy manager job], probably about two years or so ago. Up until that point there had been no kind of formal policy decision joined up making. People did what they thought was appropriate.”

Q: So what triggered that? Was it a change in the landscape?

“It was basically the drive for open data.”

These developments have included a number of changes around the management of open data to enable decisions to be made quickly and consistently about what data should be opened or otherwise made available for third parties to access and re-use. Developments include eight criteria for open Met Office data and a system for classifying the different types of data the Met Office handles.

“So we’ve divided our data into open, managed, research, and internal. Managed data doesn’t necessarily mean paid for, but it means that we will control it in some way. The terms and conditions might be different, it might be because it contains somebody else’s data, and they’ve put restrictions on us. So our national severe weather warnings are managed data. To make sure that we have a consistent coherent message so if there is a warning out everyone gets the same message.”

New systems have also been developed to provide appropriate access to the data that is being opened for re-use. For example, the Met Office developed the DataPoint API to allow access to near real time open data. This API was funded by the Public Weather Service Customer Group, and was developed in response to the perceived shortcomings of the government’s data.gov.uk portal which cannot handle the real time data needed for commercial applications:

“One of the things that we weren’t happy with – data.gov.uk – you can download a CSV. And we said that’s really not – certainly with the forecast data if we’re making this data open, because it’s about supporting economic growth you need an API. You need the Application Programming Interface (API) because you need people to be able to have it in real time and just automatically get hold of the data they need. So we put in place something called DataPoint.”

Whilst this DataPoint API currently only provides access to dynamic open data, plans are in place to make “everything available through the API” in the future, as this will contribute to Met Office compliance with the INSPIRE regulations.

Adapting to the new open data landscape has been challenging for the Met Office. Yet, some initial hurdles have been overcome and the organisation is beginning to take a more proactive approach to open data policy development. In late 2014, it announced it had become a partner of the Open Data Institute’s membership programme.

At the time of writing, the Sheffield Weston Park data generated by the Met Office’s own equipment are not available via the DataPoint service, or via other open data portals, however it is possible that this may change over time as the Met Office’s open data programme becomes more embedded.

Public sector governance

Trading funds

The Met Office has been a Trading Fund since 1996. Trading Funds are public bodies that must gain a significant proportion of their revenue from the commercial exploitation of the goods and services they produce. They became popular in the UK during the 1990s. The Met Office’s revenue comes primarily from the services it is contracted to provide to the UK government and public sector. It also provides some commercial services to the private sector.

Government commissioned reports such as the Cambridge Study (2008) and the Shakespeare Review (2013) have critiqued the commercial exploitation of public sector information by trading funds.

Over recent years the Trading Fund model has been questioned by some policy makers:

“Trading funds… Essentially self-financing agencies by the sale of information and that I think is now recognised as a flawed model. I mean if you read the stuff in Shakespeare’s report. It’s a crazy way to finance what are essential bits of the national infrastructure.”

Public Data User Group

In response to some of these concerns, the Met Office and other trading funds have been subject to a number of governance changes since the coalition government came to power in 2010.

These began with the 2011 proposal for a Public Data Corporation that would bring together key trading funds including the Met Office into one single organisation. Then the Met Office, Ordnance Survey and HM Land Registry were moved into the Department for Business, Innovation and Skills (BIS) in July 2011.

Whilst the Public Data Corporation plans were abandoned, in late 2011 the Chancellor of the Exchequer announced in the autumn statement that three new bodies would be formed:

• A Public Data Group made up of the Met Office, Ordnance Survey, Companies House and the Land Registry trading funds.

• A Data Strategy Board.

• An Open Data User Group.

These three bodies are at the heart of new governance processes around the opening of trading fund data. The new decision making process is based on a commissioning model. Advisers based in the Data Strategy Board (and its sub-committee the Open Data User Group) advise ministers to commission data from the Public Data Group, which can then be opened up for free re-use. A similar process is also utilised for commissioning data for public sector re-use.

The Public Data Group was formed based upon its members being trading funds holding economically valuable data that might be opened. However, these public bodies face different challenges in relation to open data, and at times seem like an odd grouping:

“The PDG is more about us working with the other three organisations. To sort of share lessons and learn from each other about how do we do it. That’s one of the issues, when you have Met Office, Land Registry, Companies House, and Ordnance Survey, their data, Ordnance Survey there are similarities with, but Companies House and Land Registry their data is so different from ours.

“I think one of the problems that we have with the Met Office is that we collect data, which is our observations, but then we do stuff to it, and our reason for being is not collecting data. Our reason for being is making forecasts. So although there are some things we’ve learned from each other. We have far more alignment with people like the Environment Agency, and BGS, and CEH where they’re working on environmental data. So yeah, our PDG is a bit of a weird conglomeration of people.”

Privatisation and outsourcing

Despite these developments in the governance structure around the Met Office, some policy makers are still unhappy. One former policy maker we spoke to complained that trading funds such as the Met Office:

“... Haven’t been subject to some of the disciplines that even the rest of the public services have been subject to. So essentially have been monopoly suppliers of information they can charge whatever they–, they can charge cost recovery but they’re the monopoly supplier so eventually they’ll just be able to pass on their costs to their information.”

Whilst rumours have circulated for some time about the potential privatisation of the Met Office and other trading funds, this policy maker preferred the option of outsourcing significant parts of the Met Office to a private sector management company:

“I think the experience elsewhere when national information assets have been privatised has not been good. Frankly, the value of the business is in the information that it holds. So people will only pay that value for the business if they think they can exploit that information. Which ultimately means either charging for it or denying access to it... and so privatisation is probably a poor model.

”A better model is the sort of thing that was done with the National Savings and Investments (NSI) which is where it’s still a government owned asset but it’s sort of hollowed out to a private sector management company to run it in the most efficient way possible. That is you know, something that is begging out to be done in the case of the Met Office.”

The governance of the Met Office as a public sector organisation has gone through many changes in recent years. One of the key reasons underlying these changes is related to the flow of data through the organisation and efforts to ‘open’ parts of the UK’s weather data infrastructure. As data policy developments continue to unfold, the future governance of the organisation remains uncertain.