Contents
1 Executive summary and key findings
Once the preserve of academics and statisticians, data has become a development cause embraced by everyone from grassroots activists to the UN Secretary-General. There’s now a clear understanding that we need robust data to drive democracy and development — and a lot of it.
Last year, the world agreed the Sustainable Development Goals (SDGs) — seventeen global commitments that set an ambitious agenda to end poverty, fight inequality and tackle climate change by 2030. Recognising that good data is essential to the success of the SDGs, the Global Partnership for Sustainable Development Data and the International Open Data Charter were launched as the SDGs were unveiled. These alliances mean the “data revolution” now has over 100 champions willing to fight for it. Meanwhile, Africa adopted the African Data Consensus — a roadmap to improving data standards and availability in a region that has notoriously struggled to capture even basic information such as birth registration.
But while much has been made of the need for bigger and better data to power the SDGs, this year’s Barometer follows the lead set by the International Open Data Charter by focusing on how much of this data will be openly available to the public.
Open data is essential to building accountable and effective institutions, and to ensuring public access to information — both goals of SDG 16. It is also essential for meaningful monitoring of progress on all 169 SDG targets. Yet the promise and possibilities offered by opening up data to journalists, human rights defenders, parliamentarians, and citizens at large go far beyond even these.
What is open data & why is it important?
In a well-functioning democratic society, citizens need to be informed and have access to information on government policies and progress. Open data — data which is freely available and shareable online, without charge — dramatically reduces the time and money citizens need to invest to understand what government is doing and to hold it to account. At the same time, because open data is made available in bulk and in formats that simple computer programmes can analyse, comparing and combining data from different sources becomes faster and easier, even across national boundaries. This greatly enhances the ability of policymakers, scientists and entrepreneurs to find solutions to complex development problems.
According to the open definition, to be truly open, data should be:
- Available online so as to accommodate the widest practical range of users and uses.
- Open-licensed so that anyone has permission to use and reuse the data.
- Machine-readable so that large datasets can be analysed efficiently.
- Available in bulk so that it can be downloaded as one dataset and easily analysed by a machine.
- Free of charge so that anyone can access it no matter their budget.
When made freely available to everyone via the Web, without charge, in formats that are easy to share, combine and cross-reference, open data is more than a tool to hold governments accountable. It is also a driver of innovation that can improve education and healthcare, create new businesses, and stimulate scientific progress. As the World Bank has noted, “sharing Open Data and the methods for using it will accelerate progress and help to make the SDGs possible.”
Yet our research shows that much more remains to be done to unlock open data as an SDG accelerator. Only a small portion of countries provide open and free online access to data critical to the SDGs, such as public spending, health, education, maps or census data. Implementation and impact of open data commitments is stalling, and open data availability and capacity remains heavily concentrated in the rich world.
Covering 92 countries in the present edition, the Barometer ranks nations on:
- Readiness: How prepared are governments for open data initiatives? What policies are in place?
- Implementation: Are governments putting their commitments into practice?
- Impact: Is open government data being used in ways that bring practical benefit?
At a glance, here are this year’s key findings on the state of open data around the world:
-
- 1Open data is entering the mainstream. The majority of the countries in the survey (55%) now have an open data initiative in place and a national data catalogue providing access to datasets available for re-use. Moreover, new open data initiatives are getting underway or are promised for the near future in a number of countries, including Ecuador, Jamaica, St. Lucia, Nepal, Thailand, Botswana, Ethiopia, Nigeria, Rwanda and Uganda. Demand is high: civil society and the tech community are using government data in 93% of countries surveyed, even in countries where that data is not yet fully open.
- 2Despite this, there’s been little to no progress on the number of truly open datasets around the world. Even with the rapid spread of open government data plans and policies, too much critical data remains locked in government filing cabinets. For example, only two countries publish acceptable detailed open public spending data. Of all 1,380 government datasets surveyed, almost 90% are still closed — roughly the same as in the last edition of the Open Data Barometer (when only 130 out of 1,290 datasets, or 10%, were open). What is more, much of the approximately 10% of data that meets the open definition is of poor quality, making it difficult for potential data users to access, process and work with it effectively.
Figure 1: Evolution of the availability of online data and open data.
- 3“Open-washing” is jeopardising progress. Many governments have advertised their open data policies as a way to burnish their democratic and transparent credentials. But open data, while extremely important, is just one component of a responsive and accountable government. Open data initiatives cannot be effective if not supported by a culture of openness where citizens are encouraged to ask questions and engage, and supported by a legal framework. Disturbingly, in this edition we saw a backslide on freedom of information, transparency, accountability, and privacy indicators in some countries. Until all these factors are in place, open data cannot be a true SDG accelerator.
- 4Implementation and resourcing are the weakest links. Progress on the Barometer’s implementation and impact indicators has stalled or even gone into reverse in some cases. Open data can result in net savings for the public purse, but getting individual ministries to allocate the budget and staff needed to publish their data is often an uphill battle, and investment in building user capacity (both inside and outside of government) is scarce. Open data is not yet entrenched in law or policy, and the legal frameworks supporting most open data initiatives are weak. This is a symptom of the tendency of governments to view open data as a fad or experiment with little to no long-term strategy behind its implementation. This results in haphazard implementation, weak demand and limited impact.
- 5The gap between data haves and have-nots needs urgent attention. Twenty-six of the top 30 countries in the ranking are high-income countries. Half of open datasets in our study are found in just the top 10 OECD countries, while almost none are in African countries. As the UN pointed out last year, such gaps could create “a whole new inequality frontier” if allowed to persist. Open data champions in several developing countries have launched fledgling initiatives, but too often those good open data intentions are not adequately resourced, resulting in weak momentum and limited success.
Position | Country | Score | Income | HDI Rank | OECD | G20 |
---|---|---|---|---|---|---|
1 | UK | 100 | High Income | Very High | ||
2 | USA | 81.89 | High Income | Very High | ||
2 | France | 81.65 | High Income | Very High | ||
4 | Canada | 80.35 | High Income | Very High | ||
5 | Denmark | 76.62 | High Income | Very High | ||
6 | New Zealand | 76.35 | High Income | Very High | ||
7 | Netherlands | 75.13 | High Income | Very High | ||
8 | Korea | 71.19 | High Income | Very High | ||
9 | Sweden | 69.26 | High Income | Very High | ||
10 | Australia | 67.99 | High Income | Very High |
Table 1: Economic and development classifications of the top ten countries in this year’s Open Data Barometer.
- 6Governments at the top of the Barometer are being challenged by a new generation of open data adopters Traditional open data stalwarts such as the USA and UK have seen their rate of progress on open data slow, signalling that new political will and momentum may be needed as more difficult elements of open data are tackled. Fortunately, a new generation of open data adopters, including France, Canada, Mexico, Uruguay, South Korea and the Philippines, are starting to challenge the ranking leaders and are adopting a leadership attitude in their respective regions. The International Open Data Charter could be an important vehicle to sustain and increase momentum in challenger countries, while also stimulating renewed energy in traditional open data leaders.
Country | Global rank | Regional rank | Rank change | |
---|---|---|---|---|
Traditional leaders | UK | 1 | 1 | no change |
USA | 2 | 1 | no change | |
New challengers | France | 2 | 2 | +1 |
Canada | 4 | 2 | +3 | |
Korea | 8 | 2 | +9 | |
Mexico | 16 | 1 | +18 | |
Uruguay | 19 | 3 | +6 | |
Philippines | 36 | 6 | +17 |
Table 2: New generation of open data adopters challenging the usual global and regional ranking leaders.
These findings reveal that the open data movement is at a turning point. With the SDGs still high on the political agenda, recognition of data’s importance to development is at an all-time high. The international community can seize the moment by giving initiatives such as the International Open Data Charter and the Open Data for Development Network (OD4D) the backing and resources they need to translate growing open data policy commitments into successful implementation and impact. If we allow this moment to slip away, however, open data could fade into a ghost town of abandoned pilots, outdated data portals, and unused apps.
2 Readiness
Effective open data initiatives require collaboration between the state, private sector, and civil society. A balance is needed between governments with the capacity to create, manage and publish data, and third parties with the technical skills, freedoms and resources to use data as a tool for change. Are governments ready to take full advantage of open data as a tool for development?
In this section of the report, we analyse:
- Policies and data management approaches: Do governments have adequate policies and protocols in place for ensuring open data can be made available over the long term?
- Government action at the national and subnational level: Is the groundwork being laid for the benefits of open data to be used at all levels of government?
- Civil rights and the role of citizens: Are citizens and civil society empowered to participate in government decision-making using open data?
- Business and entrepreneurship: Are businesses and entrepreneurs able to take advantage of the economic opportunity offered by open data?
2. 1. Policies and data management approaches
To guarantee open data will be available over the long-term and be used to deliver impact, it must be rooted in a clear policy framework and benefit from a consistent global data management approach. We set out to examine whether or not this is in place across the countries studied.
Many governments develop open data programmes as part of a more general open government and transparency agenda or occasionally as a component of a more general information management programme. In such cases, a clear open data definition doesn’t always exist, and as open data is not the sole focus of these initiatives, the concept often remains diluted and lacks strong principles to guide it. For example, only six countries of the 92 studied had an explicit policy commitment to make government data open by default, guaranteeing a general right to reuse.
But even among this group, we see substantial differences — some countries, such as the USA, Italy and Moldova, have complete open-by-default policies, while others, such as the UK, Canada and Austria, have non-binding policies. Some countries, including France, Greece and Switzerland, have stated their intention to include an explicit statement in the law in the near future and others, like Finland, Macedonia and Japan, have stated their intention to actively release high-value public data, but have not explicitly guaranteed it will be truly open data. Actual implementation of these policies is irregular. Long-term strategies (defined as lasting a period of at least two years) are also rare.
Existing policies usually promote and encourage some open data principles, including machine readability or the adoption of data standards, but are weak on the specifics of implementation guidelines and standards for data publication, such as the specific datasets to be published and the metadata and formats required. The publication of data is not frequently considered part of a government’s key performance indicators, and the strategies rarely make more than a passing mention to which institution or agency is responsible for implementation. The absence of clear processes, timelines, resources and delegated responsibilities mean there is little pressure on anyone to deliver.
Figure 2: Strength of open data policies and data management approaches.
Furthermore, we have found almost no evidence of quality control processes for the data before publication. Only a few countries (e.g., the UK, South Korea, Norway) appear to have some measures in place; a few other countries (e.g., France, Austria) have organised working groups to address quality issues. We have not found convincing evidence that governments are adopting and implementing specific processes for the release and update of the data or the technical standards to be used. Around nine in 10 countries with an open data initiative in place make at least some minimal description of the datasets available through basic metadata. However, the quality and extent of such metadata varies widely, and is often implemented inconsistently.
One way to tackle these challenges is to engage with potential data users before setting or adjusting policies. Public consultations on the data needs and preferences of users are becoming more common, often administered via online participation systems, social media networks or feedback-gathering workshops. However, these are not yet conducted regularly or promoted actively. Some good examples on how to foster closer collaboration between the government and other stakeholders are Cooperation OGD Austria, the Network of Experts from the Etalab mission in France or the recently discontinued Open Data User Group in the UK.
2. 2. Government action at the national and subnational level
At the national level, both the number and quality of official open data initiatives and catalogues grew in 2015, albeit more slowly than in previous editions of the Barometer. Several of these initiatives are still small-scale and most of them are not yet properly resourced, usually lacking the minimum dedicated staff and budgets required to be successful in the medium and long term.
Figure 3: Evolution of Open Government data initiatives at national and subnational level.
The majority of countries in the survey (55%) now have an open data initiative in place and a national data catalogue providing access to datasets available for reuse. Moreover, there are also a number of new commitments to drive very incipient or new open data initiatives in countries around the world, including Ecuador, Jamaica, St. Lucia, Nepal, Thailand, Botswana, Ethiopia, Nigeria, Rwanda and Uganda.
In a few exceptional cases, existing initiatives have stalled or been discontinued. In some cases, such as Ghana, there are plans to take stock and then revamp the initiative completely, as early as this year. But in others — such as Costa Rica, Kazakhstan and Qatar — the initiatives have simply not demonstrated any progress for some time.
Figure 4: Evolution of the average scores for the Government Action indicators.
Open data has not yet become priority for governments at the subnational level. The local and regional initiatives that do exist are primarily concentrated in capital cities in Europe and North America.
This is unfortunate, as local data is an area with huge potential. Web Foundation research has shown that local government data can have tangible impacts on people’s everyday lives — from financing community schools in the Philippines to verifying whether public funds have been properly spent to build public sanitation facilities — and can contribute to achieving the inclusive, resilient and sustainable cities called for in SDG 11.
2. 3. Civil rights and the role of citizens
SDG 16 challenges societies to be inclusive and to build strong, accountable institutions. Open data puts information — and therefore power — in the hands of citizens and, if used effectively, it can increase the accountability of government institutions.
In order to make the most of open data for citizen empowerment, certain foundations need to be in place. At a minimum these include (but are not limited to): strong privacy laws, freedom of information legislation and the right to access data. If these factors aren’t in place, open data initiatives risk simply being window-dressing, or “open washing” – when data is called “open” data upon release but it does not truly meet the full open criteria. What is more, these factors contribute to a vibrant, democratic culture; in turn, this increases the likelihood of media, civil society, and ordinary citizens to seek out information and data about what their government is doing, and get involved in helping government shape policy and solve problems.
Figure 5: Correlation between the score of the civil society engagement indicator and final impact.
In over two-thirds of the countries studied, a regulatory or legal framework to protect the privacy of people’s personal data exists, with European countries leading the way. Such safeguards, together with transparent checks and balances on state data retention and surveillance powers, are critical to maintain trust and mitigate possible harm from wider sharing and reuse of data. However, implementation of data protection laws is inconsistent, and in the past year there have been several scandals over the misuse of personal data collected by government agencies. In South Africa for example, a private company contracted to distribute social grants to the poor is alleged to have misused recipients’ data to sell them airtime and loans, while in the UK, there was public outcry when it emerged that the medical records of nearly a million NHS patients may have been sold to insurance companies against their will. Government-held data is increasingly being plugged into algorithms that governments and companies use to make decisions affecting us, on everything from prison sentencing to termination of medical benefits to predictive policing. Yet to date, transparency, accountability, and fairness in the use of such algorithms is largely unregulated.
The situation is worrying with regard to freedom of information (FOI) frameworks as well — only about half of the countries studied have reasonably strong laws to guarantee citizen access to information. In those countries that have FOI laws, practical implementation is patchy, hampered by slow response times and poor quality of the information provided. It is still too rare that citizens receive acceptable responses to requests for government information within the legally stipulated time, with an effective and independent redress mechanism in place.
Figure 6: Evolution of data protection and right to information indicators.
Generally speaking, a legal “right to data” as a component of FOI laws or frameworks is still rare. Such legislation would require the government to disclose data proactively, with clear guidelines on formats, appropriate privacy safeguards and a general right to reuse.
Promisingly, in all but six of the countries studied, civil society and information technology professionals were active in using government data, even if this data was not open. In approximately two-thirds of the countries studied, governments are engaging with civil society on opening data to some extent. However, these cases are usually isolated one-offs on specific issues, rather than comprehensive programmes of government-civil society dialogue.
2. 4. Business and entrepreneurship
The UN’s ambitious development agenda naturally includes economic goals. SDG 8 sets out provisions for employment and economic growth, Goal 9 emphasises innovation, and Goal 10 aims to reduce inequality. All of these go hand-in-hand, and governments can use open data to promote innovation, support data-driven businesses and create jobs.
Our study considered how the economic benefits of open data could be unlocked — through training and the promotion of innovation.
In terms of training, governments remain very focused on general data-related issues, such as statistics, data science, geographic information systems (GIS), visualisation, and big data. Academic training on these more general data topics is increasing at universities in particular. Access to high-quality specialist training for individuals or businesses that want to increase their technical skills or develop data-driven businesses is on the rise, thanks to the efforts of groups like the Open Data Institute’s global network and training programmes, and Open Knowledge’s School of Data network. But this is still not enough. It is difficult to find countries where a full range of advanced and specialised training on data analytics and open data issues is available, and finding courses on more specific themes with a sectoral approach — such as natural resource transparency, health data management, or improved instruction through open education data — is equally challenging.
Only one in five countries studied have an advanced and sustainable programme of support for innovation that is designed to take advantage of open data in the medium to long term. Many of these programmes use open data through a series of connected planned events or funding schemes in partnership with government agencies or departments towards a final common objective. These are frequently focussed not only on open data issues, but also on more general data-related topics, such as mobile apps, big data, or subject-specific challenges.
Support for innovation using open data specifically remains particularly limited in Africa and the Middle East, and to a lesser extent in South Asia, and Latin America and the Caribbean. In these regions, official interventions to support a culture of innovation are usually limited to one-off challenges, hackathons or co-creation sessions that are mainly driven by civil society and involve limited government support.
Figure 7: Readiness for business and entrepreneurship.
The Open Data for Development (OD4D) programme has been increasing efforts to address these issues through the creation and suport of multi-stakeholder initiatives, such as the Iniciativa Latinoamericana por los Datos Abiertos (ILDA), the Caribbean Open Institute (COI) , the Open Data Labs in Jakarta, and the Open Data in Europe and Central Asia initiative (ODECA)bringing them together to support innovation in the use of open data.
3Implementation
Effective open government data initiatives should provide access to a wide range of data, but all too often governments are still publishing only selected data sets. The implementation component of the Barometer looks at the extent to which government data — both open and otherwise — is published. We then assess how much of this data is open, accessible, and timely. The 15 kinds of data included in our survey reflect a wide range of functions of government.
Complete, high-quality government data and metadata is still difficult to find. This is compounded by a low level of detail in the data published. We studied some of the most sought-after datasets to establish how widely available they are.
3. 1. Open government data availability
All governments are publishing at least some data. Most of the datasets in our survey (76%) are available online in some form. However, of the 1,380 datasets we studied, barely 10% are fully open. Those that can be considered fully open datasets are concentrated in the top-ranking countries, with 46% of all open datasets belonging to the top ten countries in the Barometer. Only around one-third of the countries studied have at least one open data set, while the remaining two-thirds of countries studied have no open data whatsoever. Considering the top ranking countries are all OECD members, this illustrates just how far we have to go to realise the full potential of open data for development. We have hardly begun to make progress outside a very limited number of countries.
In a number of cases, countries have attempted to release open data, but have not followed the open definition properly. The most common pitfalls are the lack of open licences, lack of bulk download options and the use of non-machine readable formats. These mistakes are often due to a lack of expertise, resources and support for open data initiatives in the readiness phase. Hopefully initiatives like the International Open Data Charter can create a community of best practice and support to help governments embarking on new open data initiatives to improve their performance.
3. 1. 1. Openness by dataset type
Across the countries we studied, government budget data is most likely to be open, with open data available in 17 of the countries studied, just under one in five (18%).
But there are many critical areas where open datasets are unlikely to be found, for example:
- Company Registers: only available in Australia
- Government Spending: only available in Brazil and the United Kingdom
- Legislation: only available in Brazil, France, Korea and the United Kingdom
- Land Ownership: only available in Australia, Canada, Estonia, the United Kingdom and Uruguay
Figure 8: Evolution of the average scores for each of the datasets groups in the survey.
In the table below, we’ve outlined which data sets are most commonly available in open formats and which are most commonly available online in any format. Data is counted as available if it can be found online. This does not mean the data is of good quality and easily usable or downloadable for free, only that it exists online. Please see the table below for details on data quality by type. Additionally, we’ve highlighted which Sustainable Development Goals would benefit from having each of these specific datasets made open.
Data type | % data type available as fully open data | % data type available online | Highlights of current data quality |
---|---|---|---|
Budget | 18% | 97% | Comparatively one of the better datasets in terms of availability and openness. In 95% of countries where it was available it was regularly updated. In several instances it is even required by law to be updated and available, although not necessarily open. |
SDGs this data could contribute to advancing if open |
|||
National Statistics | 15% | 99% | Although this is the type of data governments are most likely to make available, it is not often released in an open format. |
SDGs this data could contribute to advancing if open |
|||
Public Transport | 13% | 65% | In spite of being one of the most in-demand datasets with great potential for social and economic impact, any kind of data (open or otherwise) is totally absent in 24 of the 92 countries in the study, 15 of which are African – unsurprising given the informal nature of the sector across most of the continent. |
SDGs this data could contribute to advancing if open |
|||
Health | 13% | 88% | The data is not straightforward to find as it is scarce and distributed through different agencies and departments rather than stored and released all in one place. And the health data that is available is usually of limited quantity and quality, out of date, and there are no plans in place to make sure it is regularly maintained and updated. Moreover, privacy concerns may also affect availability of data in such a sensitive sector. |
SDGs this data could contribute to advancing if open |
|||
Environment | 13% | 71% | There is a significant lack of environmental data, with 27 of the countries studied not publishing it in any form, open or otherwise. Finding environmental data is especially problematic in African countries (17 of the 27 countries who are not publishing any environmental data). |
SDGs this data could contribute to advancing if open |
|||
Maps | 12% | 72% | The data available online is not updated regularly enough to be useful in 45% of countries studied. |
SDGs this data could contribute to advancing if open |
|||
International trade | 12% | 97% | Frequently available online (97%) and usually for free (98%). However the data is often in an aggregated form that reduces its usefulness and is rarely published in an open format. |
SDGs this data could contribute to advancing if open |
|||
Crime | 12% | 80% | Crime data is most often available in the form of thematic reports. Each country categorises crime data differently, making it very hard to compare across countries. Sometimes the reports are quite detailed but the data contained in most of them is not in open or machine readable formats. |
SDGs this data could contribute to advancing if open |
|||
Elections | 12% | 88% | Election data is widely available online – reported in 88% of countries studied. However, it is not always to the desired level of detail. |
SDGs this data could contribute to advancing if open |
|||
Education | 11% | 87% | Most education data is available in the form of one-off reports that are not open or machine readable. The type of data available varies by country, and is rarely published on a regular schedule using the same categories, making it hard to compare across time as well. |
SDGs this data could contribute to advancing if open |
|||
Contracting | 8% | 82% | Only 28% of the data available online is in machine-readable formats reducing practical accountability as this makes analysing the high volume of historical data very difficult. |
SDGs this data could contribute to advancing if open |
|||
Land ownership | 5% | 46% | Rarely available online, difficult to find when available and quite frequently behind paywalls. |
SDGs this data could contribute to advancing if open |
|||
Legislation | 4% | 98% | Hardly available in any machine readable or open format, which is problematic considering the huge volumes of data managed. |
SDGs this data could contribute to advancing if open |
|||
Spending | 2% | 4% | Weakest dataset in the study. Even in the limited cases when it is available online, the data is usually not published at the transactional level. Only four countries – the USA, the UK, Japan and Brazil – publish spending data at the transactional level and from those only two, the UK and Brazil, release this information as open data. This makes it nearly impossible for government, citizens and civil society to tackle corruption |
SDGs this data could contribute to advancing if open |
|||
Companies | 1% | 72% | Least open dataset in the world with just Australia publishing it as open data and only for very top level data for free. Looking at all datasets available, it is only accessible online in a machine readable format and for free in just a dozen of countries. The absence of adequate open data on companies makes tracking beneficial ownership challenging and hampers efforts to tackle corruption. |
SDGs this data could contribute to advancing if open |
Table 3: Summary of data availability, openness and quality.
3. 2. Open government data quality
Most governments still struggle to ensure their data – open or otherwise – is of good quality. In many cases, so-called “open” data actually fails to be truly open due to a range of different quality issues. In the section above, we gave a brief description of our overall observations on quality for each type of data in the research. Below, we break this down even further and assess how many of each dataset have the characteristics of usable open data according to the key open data properties.
Dataset | Machine readable | Bulk | Free | Open license | Updated | Sustainable | Discoverable | Linked data |
---|---|---|---|---|---|---|---|---|
Maps | 67% | 36% | 64% | 23% | 55% | 65% | 68% | 3% |
Land | 36% | 14% | 52% | 19% | 64% | 71% | 64% | 5% |
Statistics | 69% | 42% | 93% | 25% | 82% | 77% | 96% | 5% |
Budgets | 47% | 33% | 99% | 20% | 96% | 89% | 87% | 2% |
Spending | 100% | 100% | 100% | 50% | 100% | 100% | 100% | 0% |
Companies | 29% | 14% | 61% | 8% | 64% | 70% | 67% | 2% |
Legislation | 18% | 8% | 93% | 16% | 81% | 86% | 79% | 3% |
Transport | 43% | 28% | 95% | 28% | 80% | 73% | 82% | 2% |
Trade | 70% | 35% | 99% | 20% | 75% | 81% | 80% | 1% |
Health | 65% | 27% | 95% | 31% | 47% | 51% | 65% | 1% |
Education | 63% | 34% | 96% | 23% | 59% | 64% | 68% | 0% |
Crime | 59% | 26% | 97% | 24% | 69% | 68% | 65% | 1% |
Environment | 75% | 34% | 98% | 32% | 60% | 69% | 71% | 2% |
Elections | 54% | 32% | 99% | 21% | 94% | 78% | 80% | 1% |
Contracts | 28% | 21% | 95% | 19% | 81% | 72% | 61% | 0% |
Table 4: Summary of data quality checklist results.
Appropriately licensed: Government data must have an appropriate open licence for reuse in order to enable people to take advantage of it. However, less than 18% of the government data studied has an open licence, if any licensing information is provided at all. Generally speaking, governments do not have consistent policies on how to license data, resulting in a patchwork of licences within the same country or even within the same department or institution. In some instances, such policies or guidelines indeed exist, but they are not applied consistently across all different agencies and departments, or they are simply ignored.
Free: Although the majority (90%) of government data we studied was available for free, it remains common for fees to be charged to access certain datasets or to unlock a deeper level of detail. For example only 52% of land ownership data available online is free, while only 61% of online data on companies is available for free. Some governments are still reluctant to give up valuable datasets up as a revenue stream, not considering the potential social and economic added value they could generate.
Properly formatted: Only about half of the government data studied is available in a machine-readable format. And of that machine-readable data, only half is available for download in bulk. This makes data reuse complicated and in some cases impossible for information intermediaries like researchers, academics, civil society and the media. This is particularly problematic in areas where machine readable data is rarely available (e.g. legislation, companies and contracts) or where the historic volume of data is very high but bulk downloads are not available (e.g. maps, contracts, land, census, companies or legislation).
The most popular machine readable formats (in order of popularity) continue to be xls(x), csv/tsv, xml, json and raw dbf/mdb database dumps. There are also still a significant number of datasets that are published in other non-reusable formats such as pdf, plain html, ods or plain txt and plain jpeg/png images.
More elaborated APIs that facilitate access to data are still very rare among government data. A number of standard formats are also frequently used in some specific cases, such as gml/kml/wms for Maps or pc-axis/sdmx/spss for census and statistics in general or, to a lesser extent, gtfs for transport data.
Up-to-date: 73% of government data studied were updated to a regular timetable at the time of the study, although some of these timetables are quite infrequent (e.g. every five years). However, there is a large discrepancy between the most up-to-date data (budgets – 95% and elections – 94% up-to-date) and those which are the most outdated (health – 47% and maps – 55% up-to-date). The absence of up-to-date data on topics like health and mapping could cost lives in the event of an epidemic or natural disaster when this information is particularly critical.
The publication of data series tends to be irregular and managed inconsistently and in most cases it is very difficult to determine how and when any given data will be available or updated in the future given the total lack of information to this respect.
Easy to find: When available online, government data tends to be easy to find at the individual dataset level (75% of all data studied), but complete data on a topic is often difficult to obtain without spending a significant amount of time searching, as different related and complementary datasets tend to be split among several official sources and/or government agencies.
Sustainable: For one in four of the government datasets we studied, there is no guarantee of its future availability on a regular basis. In a few countries there is at least a general open-by-default provisions, but not a timetable or process for regular updates. This makes the future use of government data, open or otherwise, very uncertain and subject to political changes.
Linked: Linked data remains niche and scarce with only 21 datasets from a total of 1,380 in the survey (1.5%) being officially available as linked data. Half of these cases are concentrated in two of the leading countries in our ranking: USA and UK. We can find also a number of other extra-official examples usually driven by academic institutions, occasionally with the collaboration of governments. These are mostly reduced to pilot projects that very rarely replace or supplement the original government data source after the pilot phase. This lack of connectivity between different related datasets limits the potential benefits of opening government data by making it more difficult to discover the existing relationships between different datasets.
4Impact
The difficulties of measuring the impact of open data initiatives properly is a serious challenge. We are increasingly finding very good isolated examples of open data being used to improve our daily lives, but the measurement of concrete impact is not often undertaken in great detail – if attempted at all – given its complexity. This results in vague or subjective impressions and unclear evidence of impact, even though real benefits are very likely to be found. More structured research and analysis, detailed use cases and quantitative evidence are necessary to go beyond anecdotes and demonstrate the full value of these initiatives.
As a result, evidence of the impact of open data still remains limited. We assessed countries for impact at the political, social and economic levels, including:
- Transparency & accountability, and improved government efficiency and effectiveness.
- Environmental impacts, and contributions to greater inclusion for marginalised groups in society.
- General contribution to the economy, support to startups, entrepreneurs and existing businesses.
Political impact has declined, particularly on transparency and accountability which saw a 22% decrease. This trend is a warning that if governments do not dedicate enough staff and resources to implementation across all aspects of open data – economic, social and political – open data policies could fail to fulfill their expectations of real social change, and amount to little more than “open washing”.
We have seen the least evidence of impact on social issues. Although environmental sustainability was a bright spot – with considerable improvement (14% increase) in impact, there was a decline of 4% on the impact of open data on social inclusion.
Evidence of the economic impact of open data continues to increase, particularly on entrepreneurship which saw a 15% increase. Entrepreneurship had the highest average impact values across the survey. But the impact on government efficiency still declined by 9% over the last edition.
Figure 9: Number of countries with significant impact on key areas.
In addition to the decline in political impact and implementation, the lack of global progress since the previous edition of the Barometer can in part be attributed to the characteristics of the new countries we have included in our survey, all of which were below the average on their impact scores. Some of the change can also be attributed to improvements made to the accuracy of impact measurements in our study.
Figure 10: Evolution of impact indicators.
4. 1. Impact examples
As a proxy approach to impact measurement, the Open Data Barometer team identifies case studies in the media or academic literature from the last twelve months. The following are some of the more interesting examples of those case studies:
Health
DATA Uruguay and the Health Ministry in Uruguay launched “A tu servicio”, an application allowing citizens to compare different health providers, helping the Ministry to increase efficiency and effectiveness and respond to citizen feedback as part of their mission to improve health centres. This platform not only enabled better informed decisions to be made by citizens, but also revealed some quality issues in the data provided by the health suppliers.
Crime and policing
The White House will be releasing new data on police officers under the umbrella of its Policy Data Initiative that intended to increase government accountability on the issue of racial profiling, a very timely issue in American society in light of high-profile police violence against African Americans in the past year. The new datasets that are released to the public can have a real impact as they include information on police stops, use of force and officer-involved shootings.
Sustainable production and consumption
After the 2011 earthquake and tsunami off the Pacific coast of Tōhoku in Japan, electricity supply and demand data were released as open data through a joint effort of the Ministry of Economy, Trade and Industry (METI) and Electric Power Companies. These open data have triggered various application developments on electricity demand and supply, raising citizen awareness of their electricity consumption and allowing them to help the government avoid power failures during periods when demand spiked and minimise environmental impact.
Jobs and economy
Dataconnexions in France – an Etalab mission project – was able to identify new open data entrepreneurship initiatives with potential to tackle different social issues and contribute to the country economy. Through the first five editions of the programme they have already identified more than 200 startups or projects leveraging open data and have also provided support for their growth and consolidation.
Impact and the SDGs
The availability of open data is vital, especially if we are to meet the SDG targets. And its importance goes beyond just monitoring progress and spending on the goals. A key feature of open data is that is published in formats that make it easier to share, combine and analyse vast quantities of information, across different parts of government and even across national boundaries.
Many SDG challenges – such as ending malnutrition, tackling climate change, preventing epidemics and stopping illicit financial flows – demand multi-dimensional and sometimes multi-country solutions, which require sharing and analysing data from many different sources. What’s more, by opening up these data sources to anyone who is interested, governments increase the chances that someone may spot a problem’s hidden cause, uncover inaccuracies or falsifications in the information about the problem, or develop a better or more efficient solution.
Below are two examples of how open data could help to tackle the challenges highlighted by the SDGs.
Stopping child malnutrition
Preventing stunting and wasting in children (one of the SDG 2 targets) is not just about nutrition. Diarrhea and gastrointestinal disease is a major cause of child malnutrition and death, so better water and sanitation in affected communities is a priority. Women’s status, especially maternal health and literacy, is also a critical determinant, so governments must improve the education and health of women, whilst also taking action to stop child marriage, which is strongly correlated with poor health outcomes for teenage mothers and their babies alike. Social protection schemes targeting women can enhance child health by raising the incomes and status of their mothers. Education and health authorities also need to work together on interventions such as deworming and school feeding.
In rural areas, promoting climate-resilient agriculture practices and increasing women farmers’ access to land, credit and extension advice may be necessary to improve the food security of mothers and children alike. Anticipating and managing crises, such as droughts and conflicts, is also critical to avoid localised hunger. All of this shows why fully open data – putting information on health, education, income, agriculture, environment and population at the fingertips of policymakers and experts in formats that can easily be analysed by computers – is vital to design well-targeted interventions to end child malnutrition.
Reducing illicit financial flows
Illicit financial flows cost developing countries $1.26 trillion per year, according to the UN, and SDG 16 includes a commitment to significantly reduce such losses by 2030. In recent weeks, this problem has been headline news thanks to the Panama Papers, a massive leak of documents from Mossack Fonseca – a law firm that specialises in setting up offshore companies in tax haven jurisdictions. Such companies are often used for money laundering and for legal or illegal tax avoidance.
Notably, the discovery of the Panama Papers abuses was accidental (it was only made possible by a leak from inside the law firm), partial (Mossack Fonseca is only one of many firms facilitating offshore companies), and laborious (because none of the information leaked was in bulk machine-readable formats, it took a global team of investigators over a year to even begin to unravel it). As the Financial Times observed, it still leaves “plenty of secrecy to go around”.
If governments and financial services providers actually want to stop such abuses, they will need access to comprehensive cross-border data that makes it easy to routinely monitor and compare company registrations, tax payments, government contracts, import and export flows, and politicians’ assets, among other data points. Under the current system, the Washington Post found that even within the OECD it can take police in one country six months to get information from another country on a single bank transfer. By contrast, “perpetrators can move money at will and at great speed”. Without open data, it may be almost impossible for anti-corruption agencies – particularly less well resourced ones in the developing world – to keep up with the international complexity and lightning pace of illicit financial flows.
The OECD has released a new standard for automatic exchange of tax information and backed open data as a critical weapon against corruption. Four countries are already publishing information on government procurement through the Open Contracting Data Standard and 14 more are at different stages of doing so. The International Open Data Charter has begun work on an anti-corruption package that will support the creation and release of open, interoperable anti-corruption data holdings, and efforts are also underway to establish a new public register that would end the use of “beneficial ownership” to shield shell companies from scrutiny. Such efforts to open up critical financial information must be dramatically accelerated to give governments and citizen watchdogs a fighting chance against the scale and complexity of the rot revealed by the Panama Papers.
5Rankings
In spite of stalling progress, particularly on impact, long-standing leaders such as the UK and the USA remain top of the rankings. But a new generation of open data adopters is challenging their global leadership.
Figure 11: Regional distribution of open data remains irregular.
Notable challengers can be found in most regions: France and Canada in Europe and North America respectively, Korea, Japan, Singapore and the Philippines in Asia and Mexico and Uruguay in Latin America. The exception remains Africa, where no country has truly demonstrated clear leadership. Rwanda remains the most stable throughout the three editions of the Barometer, with former regional pioneers, like Kenya or Ghana, in a holding pattern as they try to revamp their initiatives. Other countries, such as Burkina Faso, have become active lately but we have yet to see results.
“The full global rankings of the Open Data Barometer are available at the data explorer. Scaled country scores are rounded to the nearest whole number before ranks are assigned, meaning a number of countries receive tied rankings.
It is important to note that a high ranking in the Open Data Barometer is not an indication of perfection, but rather an indication of how well a country is doing against its peers in getting the fundamentals of open data readiness, implementation and impact right. Therefore, the top-ranked country has a scaled score of 100, and other countries’ score values are relative to it. (For full details please see the Methodology section)
As this edition of the Barometer covers six new countries and two new readiness questions, a change in rank position may result both from the new countries entering the assessment, as well as from substantial changes to that country’s performance.”
5. 1. Top ten
Countries at the top of the ranking are characterised by strong readiness. Implementation is also strong – 46% of all open datasets we found are in these top 10 countries. Some significant impact are also typically found in these countries.
Position | Country | Score | Readiness | Implementation | Impact |
---|---|---|---|---|---|
1 | UK | 100 | 100 | 100 | 100 |
2 | USA | 81.89 | 97 | 76 | 76 |
2 | France | 81.65 | 97 | 76 | 74 |
4 | Canada | 80.35 | 89 | 84 | 67 |
5 | Denmark | 76.62 | 77 | 77 | 78 |
6 | New Zealand | 76.35 | 87 | 62 | 87 |
7 | Netherlands | 75.13 | 90 | 69 | 70 |
8 | Korea | 71.19 | 95 | 64 | 58 |
9 | Sweden | 69.26 | 88 | 60 | 64 |
10 | Australia | 67.99 | 84 | 77 | 39 |
Average top 10 | 78.04 | 90.04 | 74.50 | 71.30 |
Table 5: Top ten countries in the Open Data Barometer 3rd edition ranking.
5. 2. Bottom ten
Countries at the bottom of the table are characterised by very weak general readiness, few or no open datasets available and a lack of demonstrable impact. As it can be seen in the table, sub-Saharan Africa dominates the bottom of the table, clearly indicating that strong leadership and more efforts are needed in the region.
Position | Country | Score | Readiness | Implementation | Impact |
---|---|---|---|---|---|
78 | Cameroon | 6.57 | 12 | 8 | 3 |
78 | Botswana | 6.51 | 18 | 5 | 0 |
85 | Pakistan | 6.22 | 19 | 3 | 0 |
86 | Sierra Leone | 5.44 | 19 | 2 | 0 |
86 | Zambia | 4.91 | 16 | 3 | 0 |
88 | Mali | 3.97 | 13 | 3 | 0 |
88 | Myanmar | 3.57 | 0 | 11 | 0 |
90 | Zimbabwe | 3.38 | 11 | 3 | 0 |
91 | Yemen | 1.43 | 3 | 4 | 0 |
92 | Haiti | 0 | 4 | 0 | 0 |
Average | 4.20 | 11.50 | 4.20 | 0.30 |
Table 6: Bottom ten countries in the Open Data Barometer 3rd edition ranking.
Access the data explorer for full rankings and additional data
6Conclusion and recommendations
The open data movement is at a turning point. The UN Sustainable Development Goals have created momentum for a data revolution, and open data policy commitments and initiatives have been spreading fast. Yet, implementation and impact are lagging behind, creating a risk that the open data movement could fade into a ghost town of abandoned portals and forgotten apps.
Below are six recommendations to increase the readiness, implementation and ultimately impact of open data for development:
- 1Get behind the International Open Data Charter. The international community should give the International Open Charter systematic backing and resources to mount a large-scale drive to translate growing open data policy commitments into successful implementation and impact. The Charter will lock in political momentum around data needs by establishing clear and solid guidance on policy, and helping governments to develop long-term plans that set out timelines, resources and responsibilities for implementation.
- 2Expand and deepen open data practice. It is important to encourage an increasing number of governments to adopt open data policies. But a real open data initiative goes well beyond just creating a portal or publishing a national data catalogue. To see real impact, governments need to embrace open data as a long-term cultural shift in governance with sufficient resources and staff, ensuring that infrastructure, laws and policies are strong enough for long-term open data implementation and results. Open data portals that centralise open data on dedicated websites through specific catalogs should lead to the establishment of real open data infrastructures supporting open by default policies, explicit publication objectives, requirements and timetables and performance indicators across the whole of government.
- 3Finish the job – make sure the government data published is truly open. More and more government data is becoming publicly available, but not always in an open format given frequent issues with licensing, formats, bulk downloads or free availability. Unfortunately, due to knowledge gaps, many governments with good intentions think they are making data open but overlook key requirements for true ease and power of data reuse. For example, if all countries in our survey clearly indicated an explicit open licence for data they have already placed online, the number of fully open datasets would double overnight.
- 4Harmonise open data, privacy and freedom of information efforts. While advocating for more transparent and participatory forms of governance, the open data movement has paid insufficient attention to date to the wider political and institutional enablers for such a shift. However, widespread concerns over shrinking civic space and large-scale state intrusions on privacy mean this is no longer possible. Open data advocates should work more closely with transparency, privacy and right to information activists to achieve better mutual understanding and coordination of efforts. The International Open Data Charter should educate stakeholders that open data cannot be fully effective in the absence of basic foundations such as an effective freedom of information regime and robust privacy safeguards.
- 5Consult data users and prioritise the data citizens and data users want. Governments and civil society should work together to identify the most pressing societal priorities and the data needs linked to these. Instead of sporadic hackathons or the one-time release of a few datasets, they should systematically invest in user capacity to harness open data to solve these challenges, as part of a clear long-term open data strategy.
- 6Provide funding, training and support for developing countries to close the data gap. Donors should make it a priority to close the gap between developed and developing countries on open data availability and use, not only providing initial support and assistance to get the ball rolling but also helping developing countries tackle and overcome long-standing barriers of low connectivity, poor data management infrastructure, weak legal foundations and scarce skills that limit open data achieving scale in the developing world.