Not All Data Is The Same: Rules For Data Integrity.

February 28, 2021
Procurement

-Ian Mackill

Not all data is the same. It might have come from the same source, but how it gets treated is vital. If a data company doesn’t have good data hygiene practices things can get messy very quickly, making it hard to understand the data or undermining your valuable analysis.

These are our rules for ensuring data integrity:

  1. Always know where the data came from → We always record the precise source of every record, so that our users can always go back to the original source and validate each one of our records.
  2. Always know when the data was collected → We don’t just record where we got the data from, we also know when we collected the data, so if the source changes, we can change.
  3. Never overwrite source data → We know that some of our data needs to be improved, if a date is incomplete or a better category can be added, in doing this we always add to the data, rather than overwrite the underlying data.
  4. Generate metadata → Our clients want to be able to filter data on attributes that we create, the language of a record for instance. We augment our data with useful metadata every time we gather a record.
  5. Handle duplicates with sensitivity → We see a lot of duplicates and some records that look like duplicates but aren’t. So we don’t provide a binary ‘on’ or ‘off’ analysis of duplicates, we look at eight key attributes and then score these to provide a good understanding of whether something is a duplicate.
  6. Matching needs manual checks → Entity matching is incredibly hard to get right, algorithms can help, but in the end, every match that isn’t an exact match needs to be checked, to make sure that a match is correct. That’s what we do, because the details matter and if we get a contract award wrong, then it can impact investment decisions.
  7. Be ready to highlight anomalies → We wished that some of the records we gathered were better formed, had better information, or just had the data that they were supposed to have. We have to accept that this isn’t always the case. So where things aren’t right, we don’t shy away, we don’t pretend that everything is rosy, we tell our users where the problems are, and let them budget what’s best.

At Spend Network, we know that data quality matters. We won’t tell you things you want to hear just to get a sale, we’ll tell you what we know. We want to build partnerships, not future problems.

If you’d like to know more about our data or our research services, get in touch.

September 29, 2022

£900k Government Fund To Help Charities Win Public Contracts.

The Department for Digital, Culture, Media & Sport (DCMS) is running a Voluntary, Community and Social Enterprise (VCSE) Contract Readiness Fund grant...
September 27, 2022

New EU Procurement Instrument Now Law.

About a year ago, we wrote an article on the new procurement instrument approved by the European Union. In recent weeks this...
September 20, 2022

Blacklisting Gets Tested.

Back in June we wrote about blacklisting of suppliers and the Government’s intention to prevent poorly performing suppliers from bidding for government...
September 8, 2022

UK Risks Its Place On Anti-Corruption Body

The UK has been placed ‘under review’ by the 77-country-strong Open Government Partnership (OGP) due to its failure to meet mandatory criteria...
September 8, 2022

Thurrock Exposes Transparency Blind Spot

An investigation by The Bureau of Investigative Journalism (TBIJ) into investments by Thurrock Borough Council has led to the resignation of the...
September 6, 2022

New Zealand Government Reviewing Procurement System.

It’s always encouraging when we see governments around the world looking to improve their procurement transparency and efficiency. The New Zealand Government...
September 1, 2022

Collecting Data For Sustainable Procurement In Construction

Over the last few weeks we have been looking at setting a sustainable procurement framework in the construction industry, and what kind...
August 30, 2022

Selecting Data For Sustainable Procurement In Construction

It is estimated that around 40-50% of natural resources are transformed into construction material, and that as much as 30% of all...
August 25, 2022

Setting A Sustainable Procurement Framework For Construction

When procuring construction projects, it can be useful to underpin sustainability criteria on existing policy and regulation. When assessing the enabling framework,...
August 18, 2022

Big Net Zero Contract Win For Small Cornish Business

A small Cornish company has purportedly won a £70bn contract to help deliver the country's transition to Net Zero. The Penzance based...
August 16, 2022

Supporting Sustainable Procurement In ICT

One of the key challenges of sustainably procuring ICT lies in the lack of transparency in supply chains. To overcome this challenge,...
August 11, 2022

Why Is Sustainable Procurement Important For The ICT Sector?

The extraction of raw materials, manufacturing, transportation, use, and disposal of ICT products is associated with a number of environmental, social, and...
August 10, 2022

Shifts Towards Sustainable Sourcing

A while ago,  we shared an article on findings by the Boston Consulting Group and the World Economic Forum that showed procurement is responsible...
August 9, 2022

Supporting Sustainable Procurement In The Construction Industry

Construction projects are usually long and complex, involving the participation of different stakeholders throughout the different project stages. There are certain factors...
August 4, 2022

Why Is Open SPP Important In The Construction Sector?

The construction industry is estimated to account for 6% of global GDP, with Africa's construction market valued at around USD 5.4 billion...

Newsletter

Compelling research, insights and data directly into your inbox.

Recent media stories

The Times
May 30, 2022
CIPFA
August 3, 2021

Search

Scroll to Top