7 Rules for Data Integrity

December 17, 2020
Data

Not all data is the same. It might have come from the same source, but how it gets treated is vital. If a data company doesn’t have good data hygiene practices things can get messy very quickly, making it hard to understand the data or undermining your valuable analysis.

  1. Always know where the data came from → we always record the precise source of every record, so that our users can always go back to the original source and validate each one of our records.
  2. Always know when the data was collected → we don’t just record where we got the data from, we also know when we collected the data, so if the source changes, we can change.
  3. Never overwrite source data → we know that some of our data needs to be improved, if a date is incomplete or a better category can be added, in doing this we always add to the data, rather than overwrite the underlying data.
  4. Generate metadata → Our clients want to be able to filter data on attributes that we create, the language of a record for instance. We augment our data with useful metadata every time we gather a record.
  5. Handle duplicates with sensitivity → We see a lot of duplicates and some records that look like duplicates but aren’t. So we don’t provide a binary ‘on’ or ‘off’ analysis of duplicates, we look at eight key attributes and then score these to provide a good understanding of whether something is a duplicate.
  6. Matching needs manual checks → Entity matching is incredibly hard to get right, algorithms can help, but in the end, every match that isn’t an exact match needs to be checked, to make sure that a match is correct. We do this because the details matter, and if we get a contract award wrong, then it can impact investment decisions.
  7. Be ready to highlight anomalies → We wished that some of the records we gathered were better formed, had better information, or just had the data that they were supposed to have. We have to accept that this isn’t always the case. So where things aren’t right, we don’t shy away, we don’t pretend that everything is rosy, we tell our users where the problems are, and let them budget what’s best.

We think that how we approach quality matters. We don’t tell you things you want to hear just to get a sale, we tell you what we know. We want to build partnerships, not future problems. If you’d like to know more about our data or our research services, get in touch.

contact@spendnetwork.com

December 2, 2021

We Don’t Want Your Trust.

Not because we're shifty. We're not. You don't have to trust us. Specifically, you don't have to trust our data. If you...
November 30, 2021

Intelligent Procurement Fraud Is Coming.

Are you ready for artificial intelligence fraud? We've already written about the almost magical capabilities of artificial intelligence here. We're using it...
November 18, 2021

What Do You Want To Predict?

Machine learning is changing the world. Spend Network is already using machine learning to categorize all of our documents. The results are...
November 17, 2021

New Government IT Procurement Opportunities For Smaller Businesses.

The autumn budget included an announcement that HM Revenue and Customs will receive approximately £750m over the next three years to upgrade...
November 11, 2021

What does good data look like?

If you're lucky, poor data will only make you look inefficient, but if you're unlucky, it will make a real mess of...
November 9, 2021

Greek Defence Spends Billions On Cyber, Navy and Aerospace.

In early September, Greek Prime Minister Kyriakos Mitsotakis outlined major defence procurement plans as a way to overhaul its loss-making Greek defence...
November 4, 2021

Pressures Mounts On UK Procurement Processes

The UK Government has come under renewed pressure after being ordered to reveal the names of companies that were directly awarded significant...
November 2, 2021

South Korea Targets SME Growth Through Health Procurement

When selling to governments, SME's often have challenges with matching the buying power and supply chain strength of larger organisations. For the...
October 26, 2021

List Of UK Tender Sources.

Our list of tender sources in the UK extends to over 180 individual publishers. If you use Spend Network you can access...
October 20, 2021

Canada Levels Up Procurement For Young Entrepreneurs.

Canadian not-for-profit organisation Futurpreneur,  has partnered with the Canadian Aboriginal and Minority Supplier Council (CAMSC), on a project to diversify the Canadian...
October 19, 2021

Delivery Guaranteed.

Our customers are the special ones. We make the bricks, they build the houses. They're the ones that make it easy to...
October 14, 2021

Contracting Emissions

We are launching a new analysis; the carbon emitted by the public sector's supply chain (scope three emissions). We're calling it Contracting...
October 12, 2021

The Services Sector Opens Up

The WTO have announced a new agreement on the regulation of the services sector in over sixty countries. The deal ensures more...
October 7, 2021

Greece Spends €3 Billion in French Defence Deal.

A few weeks back, when the US, UK and Australian governments announced their surprise military pact, France wore the brunt of the...
October 5, 2021

UK Government £650 Billion Pipeline.

The UK Government has released plans for the investment of £650 billion of public and private cash into infrastructure projects across the...

Newsletter

Compelling research, insights and data directly into your inbox.

Recent media stories

Search

Scroll to Top