Not All Data Is The Same: Rules For Data Integrity.

February 28, 2021
Procurement

-Ian Mackill

Not all data is the same. It might have come from the same source, but how it gets treated is vital. If a data company doesn’t have good data hygiene practices things can get messy very quickly, making it hard to understand the data or undermining your valuable analysis.

These are our rules for ensuring data integrity:

  1. Always know where the data came from → We always record the precise source of every record, so that our users can always go back to the original source and validate each one of our records.
  2. Always know when the data was collected → We don’t just record where we got the data from, we also know when we collected the data, so if the source changes, we can change.
  3. Never overwrite source data → We know that some of our data needs to be improved, if a date is incomplete or a better category can be added, in doing this we always add to the data, rather than overwrite the underlying data.
  4. Generate metadata → Our clients want to be able to filter data on attributes that we create, the language of a record for instance. We augment our data with useful metadata every time we gather a record.
  5. Handle duplicates with sensitivity → We see a lot of duplicates and some records that look like duplicates but aren’t. So we don’t provide a binary ‘on’ or ‘off’ analysis of duplicates, we look at eight key attributes and then score these to provide a good understanding of whether something is a duplicate.
  6. Matching needs manual checks → Entity matching is incredibly hard to get right, algorithms can help, but in the end, every match that isn’t an exact match needs to be checked, to make sure that a match is correct. That’s what we do, because the details matter and if we get a contract award wrong, then it can impact investment decisions.
  7. Be ready to highlight anomalies → We wished that some of the records we gathered were better formed, had better information, or just had the data that they were supposed to have. We have to accept that this isn’t always the case. So where things aren’t right, we don’t shy away, we don’t pretend that everything is rosy, we tell our users where the problems are, and let them budget what’s best.

At Spend Network, we know that data quality matters. We won’t tell you things you want to hear just to get a sale, we’ll tell you what we know. We want to build partnerships, not future problems.

If you’d like to know more about our data or our research services, get in touch.

October 20, 2021

Canada Levels Up Procurement For Young Entrepreneurs.

Canadian not-for-profit organisation Futurpreneur,  has partnered with the Canadian Aboriginal and Minority Supplier Council (CAMSC), on a project to diversify the Canadian...
October 19, 2021

Delivery Guaranteed.

Our customers are the special ones. We make the bricks, they build the houses. They're the ones that make it easy to...
October 14, 2021

Contracting Emissions

We are launching a new analysis; the carbon emitted by the public sector's supply chain (scope three emissions). We're calling it Contracting...
October 12, 2021

The Services Sector Opens Up

The WTO have announced a new agreement on the regulation of the services sector in over sixty countries. The deal ensures more...
October 7, 2021

Greece Spends €3 Billion in French Defence Deal.

A few weeks back, when the US, UK and Australian governments announced their surprise military pact, France wore the brunt of the...
October 5, 2021

UK Government £650 Billion Pipeline.

The UK Government has released plans for the investment of £650 billion of public and private cash into infrastructure projects across the...
October 20, 2021

Contracting Emissions

We’ve built an analysis of carbon emissions by contract. Our team has shown some real dedication to get this far with the...
September 28, 2021

You Don’t Care About Open Data.

We only charge for commercial use of our data. Non commercial users get it for free. That means that our data is...
October 20, 2021

Zombie Tenders On The Rise

Across Europe, the number of single bid tenders that only receive one bid continues to grow. Despite the pandemic, the number of...
September 21, 2021

More Than Half Of Australia’s Procurement Is Uncompetitive

This week, the Australian Government House of Representative  standing committee  has been presiding over a an inquiry into government procurement practices. It...
September 15, 2021

Our story. Reinventing a business.

Spend Network exists because I almost lost everything. Where we came from: I'd built a consultancy with twenty staff and clients across...
August 31, 2021

UK and Central America Increase Trade Ties

The UK’s export credit agency UK Export Finance (UKEF), has signed a partnership with Banco Centroamericano de Integración Económica (BCIE), Central America’s...
August 26, 2021

New Canadian Government Opportunities For Indigenous Businesses

We recently discussed new procurement changes in Australia with the Western Australian government procurement process, which includes specific policies to increase opportunities...
August 18, 2021

Will Europe Get Stricter On Procurement?

The EU countries have agreed to pass the International Procurement Instrument (IPI), one that was initially proposed by the European Commission back...
August 17, 2021

Has Protectionism Found A New Home?

One of the first acts of the Biden administration was to direct public buyers to buy from American firms. China is apparently...

Newsletter

Compelling research, insights and data directly into your inbox.

Recent media stories

Search

Scroll to Top