Missing Data Is A Known Unknown

March 16, 2021
Procurement

There is a famous quote about the fragility of knowledge by Donald Rumsfeld, the hawkish US Secretary of Defence during the Iraq war:

“as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns—the ones we don’t know we don’t know.”

Following the legal judgement against the UK Government for delaying publication of their contract notices, the question of how many contracting records are yet to be published is effectively a known unknown for the UK.

In relative terms, the UK publishes more data on procurement than almost any other country, but our inability to know how much contracting data is missing indicates how far the UK and other countries have to travel if we’re to achieve true transparency.

In an attempt to answer this question we’ve taken the last four years of contracting data for central government in the UK and compared it to the spending data for the same period. If we count how many suppliers have been awarded contracts each year between 2017 and 2020 we find an average of 6,401 suppliers have been listed each year. However, when we look at the spend data, the average number of suppliers being paid by central government entities during that period is over 31,852.

So an 80% difference.

When we reconcile this data to supplier records, something that takes significant amounts of work, including manual and automatic matching, we still end up with a 78% difference.

Does this mean that around 80% of award notices are missing? Not quite. This analysis can only indicate that there’s an awful lot of data missing, but it can’t say precisely how much data is missing.

The sheer scale of the differential indicates that there is a problem here. For instance a search on Contracts Finder, England’s procurement transparency database, for a prominent London Borough returns just two contract award notices. It simply isn’t credible that a council in London has let only two contracts in four years. The only reasonable explanation is that they have failed to publish the data required.

We know that the team at the Cabinet Office has been working hard to improve this data, but the responsibility for compliance lies with the publishers. It is not unreasonable to suggest that the Cabinet Office has better things to do, the publishers knows what they should be doing, they have the tools to do so, why can’t they publish this information?

Rumsfeld was roundly ridiculed for his quote, somewhat unfairly to my mind, trying find new ways to describe our collective ignorance seems like a useful endeavour, despite what else you might be getting up to.

Covid-19 has put a spotlight on all of our governments, their integrity is very much at stake and transparency is a chance to turn known unknowns into known knowns. Governments should think earnestly about the benefit of choosing collective enlightenment over collective ignorance.

If you’d like to discuss our government data or research capabilities, please get in touch.

December 2, 2021

We Don’t Want Your Trust.

Not because we're shifty. We're not. You don't have to trust us. Specifically, you don't have to trust our data. If you...
November 30, 2021

Intelligent Procurement Fraud Is Coming.

Are you ready for artificial intelligence fraud? We've already written about the almost magical capabilities of artificial intelligence here. We're using it...
November 18, 2021

What Do You Want To Predict?

Machine learning is changing the world. Spend Network is already using machine learning to categorize all of our documents. The results are...
November 17, 2021

New Government IT Procurement Opportunities For Smaller Businesses.

The autumn budget included an announcement that HM Revenue and Customs will receive approximately £750m over the next three years to upgrade...
November 11, 2021

What does good data look like?

If you're lucky, poor data will only make you look inefficient, but if you're unlucky, it will make a real mess of...
November 9, 2021

Greek Defence Spends Billions On Cyber, Navy and Aerospace.

In early September, Greek Prime Minister Kyriakos Mitsotakis outlined major defence procurement plans as a way to overhaul its loss-making Greek defence...
November 4, 2021

Pressures Mounts On UK Procurement Processes

The UK Government has come under renewed pressure after being ordered to reveal the names of companies that were directly awarded significant...
November 2, 2021

South Korea Targets SME Growth Through Health Procurement

When selling to governments, SME's often have challenges with matching the buying power and supply chain strength of larger organisations. For the...
October 26, 2021

List Of UK Tender Sources.

Our list of tender sources in the UK extends to over 180 individual publishers. If you use Spend Network you can access...
October 20, 2021

Canada Levels Up Procurement For Young Entrepreneurs.

Canadian not-for-profit organisation Futurpreneur,  has partnered with the Canadian Aboriginal and Minority Supplier Council (CAMSC), on a project to diversify the Canadian...
October 19, 2021

Delivery Guaranteed.

Our customers are the special ones. We make the bricks, they build the houses. They're the ones that make it easy to...
October 14, 2021

Contracting Emissions

We are launching a new analysis; the carbon emitted by the public sector's supply chain (scope three emissions). We're calling it Contracting...
October 12, 2021

The Services Sector Opens Up

The WTO have announced a new agreement on the regulation of the services sector in over sixty countries. The deal ensures more...
October 7, 2021

Greece Spends €3 Billion in French Defence Deal.

A few weeks back, when the US, UK and Australian governments announced their surprise military pact, France wore the brunt of the...
October 5, 2021

UK Government £650 Billion Pipeline.

The UK Government has released plans for the investment of £650 billion of public and private cash into infrastructure projects across the...

Newsletter

Compelling research, insights and data directly into your inbox.

Recent media stories

Search

Scroll to Top