Post: 30 November 2012
Update Dec 4: Please see the end of the post. I have corrected errors affecting a couple of hundred rows in the 2011-2012 data and released a revised csv. Additionally I’ve released a csv containing the bulk data for the 2010-2011 financial year.
On Tuesday the Scottish Parliament released the latest figures for MSPs’ allowances and expenses. Totals by category were included in a press release, and newspapers quickly published the list of overall spend for each MSP in 2011.
Further detail of individual expenses is available via a new search tool on the Scottish Parliament website:
The tool is rather clunky and looks very much like the functionality on the website of IPSA, which publishes expenses for MPs in England and Wales.
However unlike IPSA the Scottish Parliament has declined to provide access to the bulk data. The search tool allows users to drill down to a Results Details screen for individual claims …
… but there’s no way to download all of that detailed information in a csv file or similar machine-readable format.
STV journalist Michael MacLeod highlighted this problem on the Data.gov.uk website:
I spoke to the Scottish Parliament’s media officer Eric MacLeod on the telephone to ask whether the information was available in a machine readable format. I suggested that a CSV spreadsheet would be of use to journalists like myself wishing to help the public find out how much their MSPs cost them. I was told: “We don’t think that would be meaningful in any way.” I was also told that an FOI request for the information in a machine readable format was likely to be rejected.
The open data movement hasn’t really taken off in Scotland to the same extent as in England. However it’s difficult to believe that the Scottish Parliament is not well aware that there might be some public interest in further analysis and scrutiny of MSPs’ allowances and expenses.
So, for purposes of transparency and to re-enforce the principle that open data should really be released in a re-usable format, I have scraped all of the detailed Scottish Parliament MSP Allowances information for the 2011-2012 financial year and put it into a single csv file.
Anyone who wants the data can download it from my Google folder here:
https://docs.google.com/open?id=0B0AX7sP1au8Yc3FrQm10Ml9QZ2M
https://docs.google.com/open?id=0B0AX7sP1au8YNk1TdHI3Umt5Skk [ new link ]
(The folder is public but you may need a Google account to access it. If you have any difficulties downloading the data please contact me.)
The SPMSPA_2011-2012.csv file contains 33,260 rows of data from 17,736 Results Details screens. I’ve included a ReadMe.txt file with some notes on the structure of the data.
Update Dec 4
Good news: I’ve now also scraped the bulk data for the 2010-2011 financial year and released that in a separate csv file, SPMSPA_2010-2011.csv.
Bad news: I’ve deprecated the SPMSPA_2011-2012.csv file released on 30/11/2012 and released a replacement, SPMSPA_2011-2012_V2.csv.
Due to errors I made during the scraping process, a small number of transactions were duplicated and included in the 2011-2012 data in place of an equal number of omitted transactions. A total of 186 rows of data were duplicates, and a total of 294 rows of data were omitted. I have corrected those errors in the new file.
I’ve also discovered that where a transaction has been itemised in a table containing more than ten rows, only the first ten item rows were captured by the scraping process. That means that some travel-related allowances are not fully itemised. This issue affects slightly less than 800 transactions in the 2011-2012 data and slightly more than 800 transactions in the 2010-2011 data. The transaction totals are unaffected, i.e. they are correct. I have not fixed this issue; sorry.
There’s a new link to the data; see above.