7 Rules for Data Integrity

December 17, 2020
Data

Not all data is the same. It might have come from the same source, but how it gets treated is vital. If a data company doesn’t have good data hygiene practices things can get messy very quickly, making it hard to understand the data or undermining your valuable analysis.

  1. Always know where the data came from → we always record the precise source of every record, so that our users can always go back to the original source and validate each one of our records.
  2. Always know when the data was collected → we don’t just record where we got the data from, we also know when we collected the data, so if the source changes, we can change.
  3. Never overwrite source data → we know that some of our data needs to be improved, if a date is incomplete or a better category can be added, in doing this we always add to the data, rather than overwrite the underlying data.
  4. Generate metadata → Our clients want to be able to filter data on attributes that we create, the language of a record for instance. We augment our data with useful metadata every time we gather a record.
  5. Handle duplicates with sensitivity → We see a lot of duplicates and some records that look like duplicates but aren’t. So we don’t provide a binary ‘on’ or ‘off’ analysis of duplicates, we look at eight key attributes and then score these to provide a good understanding of whether something is a duplicate.
  6. Matching needs manual checks → Entity matching is incredibly hard to get right, algorithms can help, but in the end, every match that isn’t an exact match needs to be checked, to make sure that a match is correct. We do this because the details matter, and if we get a contract award wrong, then it can impact investment decisions.
  7. Be ready to highlight anomalies → We wished that some of the records we gathered were better formed, had better information, or just had the data that they were supposed to have. We have to accept that this isn’t always the case. So where things aren’t right, we don’t shy away, we don’t pretend that everything is rosy, we tell our users where the problems are, and let them budget what’s best.

We think that how we approach quality matters. We don’t tell you things you want to hear just to get a sale, we tell you what we know. We want to build partnerships, not future problems. If you’d like to know more about our data or our research services, get in touch.

contact@spendnetwork.com

May 12, 2022

MoD £2 Billion in Nuclear Contracts.

Britain's Ministry of Defence announced this week that over £2 billion worth of defence contracts have been awarded to start the third...
May 10, 2022

New Support For Canadian SME Traders.

The Government of Canada recently released its 2022 Federal Budget. Within it was an announcement of an intention to establish a Trade...
April 29, 2022

Progressive Procurement Yields Results

Last year, we shared the results of the landmark Aboriginal Procurement policy (APP), first introduced by Western Australia’s Premier, Mark MacGowan in...
April 29, 2022

Cutting Carbon Is A Procurement Issue

Just a few months ago, we shared an article on findings by the Boston Consulting Group and the World Economic Forum that showed procurement...
April 26, 2022

OECD: Promoting Gender Equality Through Procurement

COVID-19 showed that women have been disproportionately affected by the economic and social fallout from the pandemic(OECD, 2020[1]), as the pandemic exacerbated...
April 21, 2022

Progress In The Open Government National Action Plan

We’ve been working closely with The Cabinet Office to develop and refine a robust and repeatable methodology for the identification and matching...
April 19, 2022

Analysis Is A Service.

Understanding a market or a phenomenon has to account for when something happened. If you build an analysis that says 20% of...
April 14, 2022

Consultants Still Winning Huge Government Contracts.

Consultancies have won contracts worth more than £700m from UK government in Covid related contracts since the start of the pandemic in...
April 12, 2022

Two Solutions To A Hard Problem

In our last post, I suggested that public procurement has a big problem: an inability to account for, or record failure. Today...
April 7, 2022

Public Procurement Has A Big Problem

The world has changed. We live in a world where commerce, industry, and work are powered by data. Performance is measured and...
April 5, 2022

Ukrainian Invasion & UK Procurement

Last week, the UK Government released a Procurement Policy Note (PPN) that sets out how contracting authorities can further cut ties with...
March 29, 2022

Aviation Procurement Inquiry

The Defence Committee, made up of a number of MP’s appointed by the House of Commons to examine the expenditure, administration, and...
March 22, 2022

Solving Search

Search is difficult. By that we mean building a service that delivers the right information to people who are searching for it....
March 15, 2022

EU Medical Procurement Recommendations

The European Federation of Pharmaceutical Industries and Associations (EFPIA) has released a paper that examines procurement practices across the EU in their...
March 10, 2022

You Can Shove Our Data……

Right into the heart of your business. That’s right, we want you to put our data in your own database. We don’t...

Newsletter

Compelling research, insights and data directly into your inbox.

Recent media stories

Search

Scroll to Top