Missing Data Is A Known Unknown

March 16, 2021
Procurement

There is a famous quote about the fragility of knowledge by Donald Rumsfeld, the hawkish US Secretary of Defence during the Iraq war:

“as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns—the ones we don’t know we don’t know.”

Following the legal judgement against the UK Government for delaying publication of their contract notices, the question of how many contracting records are yet to be published is effectively a known unknown for the UK.

In relative terms, the UK publishes more data on procurement than almost any other country, but our inability to know how much contracting data is missing indicates how far the UK and other countries have to travel if we’re to achieve true transparency.

In an attempt to answer this question we’ve taken the last four years of contracting data for central government in the UK and compared it to the spending data for the same period. If we count how many suppliers have been awarded contracts each year between 2017 and 2020 we find an average of 6,401 suppliers have been listed each year. However, when we look at the spend data, the average number of suppliers being paid by central government entities during that period is over 31,852.

So an 80% difference.

When we reconcile this data to supplier records, something that takes significant amounts of work, including manual and automatic matching, we still end up with a 78% difference.

Does this mean that around 80% of award notices are missing? Not quite. This analysis can only indicate that there’s an awful lot of data missing, but it can’t say precisely how much data is missing.

The sheer scale of the differential indicates that there is a problem here. For instance a search on Contracts Finder, England’s procurement transparency database, for a prominent London Borough returns just two contract award notices. It simply isn’t credible that a council in London has let only two contracts in four years. The only reasonable explanation is that they have failed to publish the data required.

We know that the team at the Cabinet Office has been working hard to improve this data, but the responsibility for compliance lies with the publishers. It is not unreasonable to suggest that the Cabinet Office has better things to do, the publishers knows what they should be doing, they have the tools to do so, why can’t they publish this information?

Rumsfeld was roundly ridiculed for his quote, somewhat unfairly to my mind, trying find new ways to describe our collective ignorance seems like a useful endeavour, despite what else you might be getting up to.

Covid-19 has put a spotlight on all of our governments, their integrity is very much at stake and transparency is a chance to turn known unknowns into known knowns. Governments should think earnestly about the benefit of choosing collective enlightenment over collective ignorance.

If you’d like to discuss our government data or research capabilities, please get in touch.

May 12, 2022

MoD £2 Billion in Nuclear Contracts.

Britain's Ministry of Defence announced this week that over £2 billion worth of defence contracts have been awarded to start the third...
May 10, 2022

New Support For Canadian SME Traders.

The Government of Canada recently released its 2022 Federal Budget. Within it was an announcement of an intention to establish a Trade...
April 29, 2022

Progressive Procurement Yields Results

Last year, we shared the results of the landmark Aboriginal Procurement policy (APP), first introduced by Western Australia’s Premier, Mark MacGowan in...
April 29, 2022

Cutting Carbon Is A Procurement Issue

Just a few months ago, we shared an article on findings by the Boston Consulting Group and the World Economic Forum that showed procurement...
April 26, 2022

OECD: Promoting Gender Equality Through Procurement

COVID-19 showed that women have been disproportionately affected by the economic and social fallout from the pandemic(OECD, 2020[1]), as the pandemic exacerbated...
April 21, 2022

Progress In The Open Government National Action Plan

We’ve been working closely with The Cabinet Office to develop and refine a robust and repeatable methodology for the identification and matching...
April 19, 2022

Analysis Is A Service.

Understanding a market or a phenomenon has to account for when something happened. If you build an analysis that says 20% of...
April 14, 2022

Consultants Still Winning Huge Government Contracts.

Consultancies have won contracts worth more than £700m from UK government in Covid related contracts since the start of the pandemic in...
April 12, 2022

Two Solutions To A Hard Problem

In our last post, I suggested that public procurement has a big problem: an inability to account for, or record failure. Today...
April 7, 2022

Public Procurement Has A Big Problem

The world has changed. We live in a world where commerce, industry, and work are powered by data. Performance is measured and...
April 5, 2022

Ukrainian Invasion & UK Procurement

Last week, the UK Government released a Procurement Policy Note (PPN) that sets out how contracting authorities can further cut ties with...
March 29, 2022

Aviation Procurement Inquiry

The Defence Committee, made up of a number of MP’s appointed by the House of Commons to examine the expenditure, administration, and...
March 22, 2022

Solving Search

Search is difficult. By that we mean building a service that delivers the right information to people who are searching for it....
March 15, 2022

EU Medical Procurement Recommendations

The European Federation of Pharmaceutical Industries and Associations (EFPIA) has released a paper that examines procurement practices across the EU in their...
March 10, 2022

You Can Shove Our Data……

Right into the heart of your business. That’s right, we want you to put our data in your own database. We don’t...

Newsletter

Compelling research, insights and data directly into your inbox.

Recent media stories

Search

Scroll to Top