Not All Data Is The Same: Rules For Data Integrity.

February 28, 2021
Procurement

-Ian Mackill

Not all data is the same. It might have come from the same source, but how it gets treated is vital. If a data company doesn’t have good data hygiene practices things can get messy very quickly, making it hard to understand the data or undermining your valuable analysis.

These are our rules for ensuring data integrity:

  1. Always know where the data came from → We always record the precise source of every record, so that our users can always go back to the original source and validate each one of our records.
  2. Always know when the data was collected → We don’t just record where we got the data from, we also know when we collected the data, so if the source changes, we can change.
  3. Never overwrite source data → We know that some of our data needs to be improved, if a date is incomplete or a better category can be added, in doing this we always add to the data, rather than overwrite the underlying data.
  4. Generate metadata → Our clients want to be able to filter data on attributes that we create, the language of a record for instance. We augment our data with useful metadata every time we gather a record.
  5. Handle duplicates with sensitivity → We see a lot of duplicates and some records that look like duplicates but aren’t. So we don’t provide a binary ‘on’ or ‘off’ analysis of duplicates, we look at eight key attributes and then score these to provide a good understanding of whether something is a duplicate.
  6. Matching needs manual checks → Entity matching is incredibly hard to get right, algorithms can help, but in the end, every match that isn’t an exact match needs to be checked, to make sure that a match is correct. That’s what we do, because the details matter and if we get a contract award wrong, then it can impact investment decisions.
  7. Be ready to highlight anomalies → We wished that some of the records we gathered were better formed, had better information, or just had the data that they were supposed to have. We have to accept that this isn’t always the case. So where things aren’t right, we don’t shy away, we don’t pretend that everything is rosy, we tell our users where the problems are, and let them budget what’s best.

At Spend Network, we know that data quality matters. We won’t tell you things you want to hear just to get a sale, we’ll tell you what we know. We want to build partnerships, not future problems.

If you’d like to know more about our data or our research services, get in touch.

February 28, 2021

Do Framework Agreements Have Value?

-Ian Makgill Framework agreements are like umbrella agreements, and are usually made with a group of providers to supply a set of...
February 28, 2021

NSW Aims To Reserve Procurement Budgets For SMEs

We're always pleased to see governments around the world improving their procurement processes, by broadening opportunities for all types of business to...
February 28, 2021

Creating Synergy Between Politics & Procurement.

The need for administrations to act at pace is often at odds with the processes and procedures needed for good procurement. If...
February 28, 2021

Canada Launches Green Procurement

We are always pleased to see governments taking steps towards better procurement practices. Recently, the Canadian Government took a step forward for...
February 4, 2021

Post Brexit Procurement – What Will Change?

With the Brexit transition period officially behind us, it's worth considering the potential impacts of Brexit on Government procurement into the future....
February 28, 2021

Where Next For Data Led Procurement in Europe? A Discussion.

The They Buy For You  Project (TBFY) concluded on 31 December 2020. To mark the occasion, we look back at three years...
February 28, 2021

Spending $400bn – A Demanding Task For Biden.

-Fiona Hunt As my colleague Ian wrote this week, governments are increasingly looking to procurement to deliver better social outcomes. President Biden...
January 31, 2021

Social Procurement Demands Data

-Ian Makgill Social value is going to be a critical feature of public procurement in the coming years. The urge to 'build...
January 31, 2021

Where Are All The Small Businesses?

While 2020 was an incredibly rocky road for many businesses globally, in public procurement it was SMEs in particular who lost out,...
January 31, 2021

Are UK, France and Germany Having a Competition Crisis?

We recently completed a study of European government procurement changes in the three years up to and including 2020, to assess the...
January 31, 2021

How Long Does It Take A Department To Publish A Contract Notice?

Public sector organisations have, according to guidance, thirty days to publish details of their contracts online. So, thirty days after a contract...
January 31, 2021

Controversial Food Box Contractors Face Scrutiny

There is increased scrutiny for the suppliers of food boxes provided to the chronically ill and those asked to shelter through the...
January 31, 2021

£550 Million Missile Contract Signed.

Yesterday defence Defence Minister Jeremy Quin announced a £550 million contract was awarded for new surge-attack missile The contract award promises 'hundreds...
January 31, 2021

Time To Build More Open Products For Government

-Ian Makgill, Founder Spend Network Just before Christmas, DXC (formerly Hewlett Packard) was awarded a contract for £430,000 by the Business Services...
January 31, 2021

New Year, New Tool. Introducing Our New Classifier.

Our New Solution To Classification In 2020, we developed an advanced classifier. This tool adds multiple labels to procurement notices based on...

Newsletter

Compelling research, insights and data directly into your inbox.

Recent media stories

Welp Magazine
December 23, 2020
FT PPE Story
The Financial Times
December 9, 2020

Search