Making Scottish public data easily discoverable

Data, data everywhere – but not easy to find?

From population surveys to vaccination statistics to electric vehicle sales over the past decade, the world is awash with data. Yet when it comes to data discoverability, a crucial element is often overlooked: metadata, meaning information such as file names, titles, descriptions, publisher details and licence types.

Regardless of how data is published, metadata availability can vary widely, from non-existent (fewer than 5 fields) to complete (more than 20 fields). Our audit of over 900 data portals worldwide reveals that only a third of them offer complete metadata. Imagine trying to find a data file labelled simply as "A1204.csv" with no accompanying metadata. It becomes a challenge even with search engines like Google or AI chatbots.

Metadata standards are also critical in ensuring data is easily detected by search engines. In Scotland, our research shows that fewer than a third of data portals comply with such standards, making their data less likely to appear in search results. This issue is even worse for websites that publish data in non-standard formats.

Data users such as analysts, researchers and policymakers may spend up to 50% of their time searching for data, which incurs significant costs for their organisations. But it is not just data users who suffer; data provider organisations that neglect metadata make their data less accessible and lose insight into how their data is being used or what impact it is having.

A breakthrough for Scotland: find.data.gov.scot

Through the CivTech programme, the Scottish Government's Digital Directorate and NatureScot sought a solution to improve the discoverability of public data in Scotland. They partnered with Dtechtive to create find.data.gov.scot, the largest search engine for Scottish public data.

find.data.gov.scot provides data teams with access to a vast range of Scottish public data. Enhanced metadata gives more context to the data, while intelligent search algorithms help data teams find relevant data through simple search terms. Because the platform adheres to metadata standards, its data is indexed by Google, making it accessible to a broader audience. The platform sets a new standard for how Scottish public data is found and used.

This innovative solution not only offers enhanced metadata and intelligent search algorithms but also provides valuable insights into user behaviour. The CivTech process enabled a successful partnership with Dtechtive, resulting in a cutting-edge tool for accessing a vast array of Scottish public data.

Shona Nicol, Head of Technical Data Policy, Digital Directorate, The Scottish Government

The wider impact of better data discoverability

By improving metadata and adopting standards, find.data.gov.scot increases the visibility and usability of Scottish public data. This has wide-reaching benefits for various stakeholders.

• Efficiency and cost savings. Data users such as analysts, researchers and businesses save time and costs by accessing well-organised, standardised public data, allowing them to focus on insights rather than retrieval.
• Data providers' insights. Providers gain valuable feedback on data usage, helping them tailor future releases to user needs.
• Boosting innovation. Easier access to public data encourages innovation across sectors such as healthcare, technology and energy, enabling new products and services that address societal challenges.
• Collaboration. Discoverable data fosters partnerships between government, academia and the private sector, leading to shared solutions to complex problems.
• Civic empowerment. Communities and local groups can use public data to hold institutions accountable, advocate for change and engage in civic decision-making.
• Better policy development. Policymakers benefit from data-driven, evidence-based decisions, improving public services and addressing societal issues more effectively.
• Improving data literacy. Increased data accessibility supports public understanding and use, promoting a more informed, data-literate society.

Overall, better access to Scottish public data leads to economic growth, improved governance and greater civic engagement.

Conclusion

Data is only as valuable as it is accessible. Improving metadata and adhering to standards are critical steps in ensuring data can be easily found and used. find.data.gov.scot is a powerful example of how better data discoverability benefits both data users and providers, driving data-driven innovation and enhancing the impact of publicly available information.