Building Scotland's national search engine for public data

The challenge

Scotland publishes a great deal of public data, but it was scattered across dozens of separate portals, each with its own structure, vocabulary and level of description. For anyone trying to answer a question with public data, the first problem was not analysis. It was finding out whether the data existed at all.

Dtechtive's audit of more than 900 data portals worldwide found that only around a third offered complete metadata. In Scotland specifically, fewer than a third of data portals complied with recognised metadata standards, which meant their datasets were unlikely to surface in search results at all.

The cost of that falls in two directions. Data users such as analysts, researchers and policymakers can spend up to half their time simply searching for data. Data providers, meanwhile, lose any visibility of how their published data is used, or whether it has any impact.

Our approach

Working through the CivTech accelerator with the Digital Directorate and NatureScot, we treated discoverability as a metadata problem rather than a search-box problem.

• Harvest, don't migrate. Rather than asking 70 publishers to move their data, we harvested metadata from where it already lived, leaving existing publishing workflows untouched.
• Standardise on the open web's vocabulary. Metadata was mapped to Schema.org and DCAT so that datasets became legible not just to the platform, but to mainstream search engines.
• Score and deduplicate. Quality scoring exposed gaps in the source metadata, and deduplication resolved the same dataset appearing under different names across portals.
• Design for non-specialists. The interface was built for analysts, policymakers, journalists and the public alike, with ranked results and plain-language filtering.

The solution

find.data.gov.scot is the largest search engine for Scottish public data. It brings together more than 25,000 datasets from around 70 data portals into a single catalogue, with enriched metadata giving each dataset the context data teams need to judge whether it is fit for their purpose.

Because the platform adheres to metadata standards, its datasets are indexed by Google, so they are discoverable to people who never visit the platform at all. That is the point that is easy to miss: standardised metadata does not just improve search on the site, it makes the underlying data visible across the open web.

The platform sets a new benchmark for how Scottish public data is found and used, and the benefits extend well beyond the search box.

• Efficiency and cost savings. Analysts, researchers and businesses spend less time on retrieval and more on insight.
• Provider insight. Publishers gain feedback on how their data is actually used, helping them prioritise future releases.
• Innovation and collaboration. Easier access to public data supports partnerships across government, academia and the private sector.
• Civic empowerment and better policy. Communities and policymakers alike can work from evidence that is genuinely accessible.

Standards and stack