Post: 10 October 2015
Updated: 9 November 2015 (see end)
The current data request process on Data.gov.uk was launched in late September 2012 with a blog post from Heather Savory, then chair of the Open Data User Group.
To see where we are today take a look at the landing page for the data request process, the list of requests, and the form for submitting requests.
The request process remains active. Requests to “unlock” public datasets for open re-use are still being forwarded to data holding departments within government, and each request has a status (such as “new”, “escalated to data holder” or “closed”) to show progression.
But there are all sorts of problems.
The data request process was originally set up as a primary mechanism through which the
Open Data User Group
(ODUG) would interact with the community. The idea was that developers, businesses and third sector organisations who wanted to re-use public data would submit their needs and arguments via the request form.
ODUG would then sift and prioritise the data requests, and selectively build benefits cases for submission to the Data Strategy Board. The Data Strategy Board itself had a pot of government money that it could use to help remove practical barriers to open release of public data.
Three years later the institutional picture has changed. The Data Strategy Board shut down in June 2013. ODUG refocused on providing advice to the Public Sector Transparency Board on how it should best fund release of data. While that arrangement did help support some useful open data projects, most of those projects were driven by data producers rather than by demand from potential users. And ODUG itself was disbanded in June 2015.
Where does that leave the data request process?
James Cattell at Cabinet Office has recently been giving some thought to improving and better explaining the process, but at the moment the process seems to be operating at a subsistence level while Cabinet Office decides how it wants to move forward on its open data policy (if indeed it still has one).
As an attempt to move things on a bit, following are ten ideas for improving the Data.gov.uk (DGU) data request process:
1. Update the contextual material
Now that ODUG is no longer part of the process the contextual material on DGU, which mainly talks about ODUG and what it planned to do with the data requests, is now well out of date. In particular the link to ODUG’s dashboard for tracking requests no longer works.
This is likely to confuse or mislead users, most of whom probably do not follow the development of open data policy and may be submitting data requests based on bad information or with false expectations.
2. Explain the process better
According to a recent note from James Cattell this is how the process currently works when a new request for data is received via Data.gov.uk:
This is a reasonable approach; a bit mechanistic, and based on what I’ve seen on handling of individual requests probably more the ideal than the practical reality, but it at least provides a framework for understanding what is supposed to happen after the user clicks the button to submit a data request.
So why not just put that on the website? At the moment the typical DGU user has no understanding of these roles and relationships.
The contextual material should also explain more clearly the type of data requests Cabinet Office will consider through the request process. Anecdotally, based on responses to individual requests, it looks as if Cabinet Office has relationships with the full range of central government departments and their executive agencies, but has little interest in handling requests for data held by local government or by public services that have been devolved to Wales, Scotland or Northern Ireland.
3. Manage expectations
Although it looks as if Cabinet Office has made an attempt this summer to address the backlog of outstanding data requests, many users have been disappointed by the slow or (in many cases) non-existent response to their data requests.
A better explanation of the process will help manage expectations, but Cabinet Office should also recognise that the communications it receives via the data request process are highly varied. Some are from users asking for data that is readily available but that they cannot find themselves. Others are from users who have data-related problems or queries that don’t really fit the intended purposes of the request process. Many of these could be dealt with swiftly by DGU administrators if they had a better general knowledge of public data. A one-size-fits-all process is not in the best interests of users.
4. Give users alternatives
DGU should also give users some other options they can try before or instead of the data request process, according to their needs.
I think it’s clear at this point that Data.gov.uk will never realise its early ambition to become a comprehensive and authoritative catalogue of UK public data. Many government departments manage their own subject-specific data repositories, and many local government bodies now also publish their own data. DGU should signpost the best of those resources, so that users have more opportunity to confirm for themselves
before submitting a data request that the data they need is not available.
DGU should also introduce users to the basics of making a data request via the Freedom of Information Act, the Environmental Information Regulations, INSPIRE Regulations and the Re-use of Public Sector Information Regulations. Those channels will not necessarily lead to open data release but, unlike the DGU request process, they are underpinned by statutory rights and may provide a better alternative when a request is time-sensitive. And of course there’s no reason why a user cannot request data via the DGU process in addition to one of those other channels.
5. Clarify the FOI status of DGU data requests
I strongly suspect that within the body of DGU data requests are several hundred examples of Cabinet Office abjectly failing to comply with the Freedom of Information Act.
As near as I can tell a straightforward request for a specific dataset via DGU’s data request form meets all the necessary requirements for communication of a FOI (or EIR) information request, in which case the requester is normally legally entitled to a response within 20 working days. There is no recognition of that in the existing DGU process.
Of course in most such cases the DGU user will not have intended to make a FOI request. But public bodies are obliged to treat valid information requests under FOI even if the requester does not mention the FOI Act. Most requests via DGU are for data that Cabinet Office itself does not hold, but that does not obviate the requirement for a formal response.
Cabinet Office should address this anomaly and regularise its practice, either by providing a compliant response to any such request within 20 working days or by re-framing the request process as a request for assistance with unlocking specific data rather than a request for the data itself.
6. Make the request form easier to find
The data request process was originally promoted on the Data.gov.uk homepage:
As we can see from the graph below, the number of requests dropped off after the link was removed from the homepage during the DGU site redesign in October 2013. The data request form is now a couple of additional clicks away from the homepage, which makes it harder for users to discover this facility.
If Cabinet Office wants to encourage more data requests it should give the process a higher profile by restoring the link from the DGU home page (alongside better explanation of the process, in order to improve the relevance and quality of the requests).
7. Improve feedback and commenting
One of the on-going difficulties with the data request process is DGU’s system for commenting and feedback to users. Although under the current process there is an assurance that users who submit requests will get some kind of final response from Cabinet Office or the data holder, this has not always been the case.
Generally the only way users can keep track of progress on their requests is to bookmark the request on DGU and check back periodically. There is no automated system for notifying users when the status of their request changes or when another DGU user adds a comment. (I sometimes add comments to data requests myself, particularly when I can see the user is looking for data that is already public, but am rarely sure the comment gets back to the user.)
8. Fix the reporting
The landing page currently reports an overview of all data requests, in the form of four pie charts. Unfortunately the numbers underlying the charts do not reconcile to a consistent total. This limits their usefulness.
Personally I’d find it helpful to have a simple csv of key information about all of the (non-confidential) requests submitted via DGU. While it’s debatable whether the data request process has had much overall impact on release of new open data, the requests themselves are useful as a source of insight into the types of data users want and the difficulties they encounter in getting it.
9. Consider reverting to the old unlocking service format
Given that the Open Data User Group is defunct and the original rationale underlying the current data request process is unlikely to be revived, there may be an argument for simply reverting to some version of the simpler (and arguably more popular) public sector information unlocking service originally launched by OPSI in 2008. This service was transferred to Data.gov.uk and the request form in use then may be closer to the needs of the current (non-ODUG) data request process.
10. Review the fundamentals
I also think there’s a case for taking a step back and reviewing more fundamentally whether there should be a centralised data request process on DGU at all. Perhaps there is something else Cabinet Office could provide (or sponsor) to better address user needs for discovery and unlocking of public data.
The premise of the request process launched in 2012 was that demand for open data would emerge from the community in sufficient volume that Cabinet Office, with the assistance of ODUG, would be able to identify datasets with the greatest economic or social potential for re-use if released as open data.
That never really happened. In retrospect it’s clear Cabinet Office had unrealistic expectations about the extent to which re-users are willing to engage in a policy process well in advance of the actual availability of new open data. Three years on we also have a better appreciation of how difficult it can be to predict how open data will be re-used, and the impact it will have, before it is actually released.
As far as I know Cabinet Office does not currently control any funds earmarked for release of open data. So does Cabinet Office still have a role or interest in facilitating demand-led release? If each government department now has a relationship manager to facilitate requests for release of open data, it might make more sense for DGU to simply publish a directory of contacts so that users can get in touch with departments directly.
Scrolling through past data requests it strikes me how few actually relate to business plans or projects that the users believe could be realised if a particular public dataset was available. More commonly the requests are about difficulties the user is having identifying a relevant dataset or the area of government that might hold that data or whether the data exists at all. And many requests are from users who are simply trying to answer a specific question (or a research problem) with a data-related angle.
Removing barriers to discovery of existing open data is in many ways just as important as unlocking that data in the first place. Perhaps what the public needs instead of (or alongside) the data request process is a simple facility to submit queries and receive expert advice about public datasets. Something on the Ask Jack model, on a newspaper website, could fulfill this function; but there’s no reason Cabinet Office should not organise or host something similar as a public service on Data.gov.uk.
Update: 9 November 2015
Subsequent to my post above DGU has removed references to ODUG from the data request process. Following is the new preamble on the data request form:
This is now very much a bare-bones approach. Cabinet Office is undertaking only to forward requests to the relevant department. There is no specific mention of open data.
Image credits: all images cropped or based on OGL data from Data.gov.uk.