Data / By /

Ensuring Data Integrity.

November 29, 2022
Data

Not all data is the same. It might have come from the same source, but how it gets treated is vital. If a data company doesn’t have good data hygiene practices things can get messy very quickly, making it hard to understand the data or undermining your valuable analysis.

  1. Always know where the data came from → we always record the precise source of every record, so that our users can always go back to the original source and validate each one of our records.
  2. Always know when the data was collected → we don’t just record where we got the data from, we also know when we collected the data, so if the source changes, we can change.
  3. Never overwrite source data → we know that some of our data needs to be improved, if a date is incomplete or a better category can be added, in doing this we always add to the data, rather than overwrite the underlying data.
  4. Generate metadata → Our clients want to be able to filter data on attributes that we create, the language of a record for instance. We augment our data with useful metadata every time we gather a record.
  5. Handle duplicates with sensitivity → We see a lot of duplicates and some records that look like duplicates but aren’t. So we don’t provide a binary ‘on’ or ‘off’ analysis of duplicates, we look at eight key attributes and then score these to provide a good understanding of whether something is a duplicate.
  6. Matching needs manual checks → Entity matching is incredibly hard to get right, algorithms can help, but in the end, every match that isn’t an exact match needs to be checked, to make sure that a match is correct. We do this because the details matter, and if we get a contract award wrong, then it can impact investment decisions.
  7. Be ready to highlight anomalies → We wished that some of the records we gathered were better formed, had better information, or just had the data that they were supposed to have. We have to accept that this isn’t always the case. So where things aren’t right, we don’t shy away, we don’t pretend that everything is rosy, we tell our users where the problems are, and let them budget what’s best.

We think that how we approach quality matters. We don’t tell you things you want to hear just to get a sale, we tell you what we know. We want to build partnerships, not future problems. If you’d like to know more about our data or our research services, get in touch.

 

May 30, 2023

Space To Innovate- New UK Gov Opportunity

The UK Government's, Defence Science and Technology Laboratory (Dstl) Space Systems Programme has announced it is seeking proposals to aid UK Defence...
Read more...
May 25, 2023

EU Commission Move To Electronic Procurement

The European Commissions Directorate Generate for Informatics (DIGIT), has launched a new dynamic purchasing system for software licences and associated services (SIDE...
Read more...
May 22, 2023

Biosecure Procurement Trial

The UK Government yesterday announced new measures for domestic tree seed production and biosecurity procurement requirements. The next round of the Domestic...
Read more...
May 18, 2023

New Industry Body White Paper Review of Covid Procurement

Since early on in the pandemic, there has been significant focus on the procurement of Covid related PPE, due to unprecedented nature...
Read more...
May 11, 2023

EU Council Joint Procurement Under European Peace Facility

The European Union Council has adopted an assistance measure worth €1 billion under the European Peace Facility (EPF) that will further contribute...
Read more...
May 10, 2023

Measuring Sustainable Public Procurement

One of the projects we are most proud of is working with The Open Contracting Partnership to create a Sustainable Public Procurement...
Read more...
May 3, 2023

From Hero to Zero – UK Digital Marketplace Shuttered

Within living memory, the UK’s Crown Commercial Service (CCS) had planned to reshape a large swathe of public procurement in the image...
Read more...
May 2, 2023

We Sell Data, Not Seats.

Why we sell data not seats? We're data analysts. We really understand the data we gather. We've been doing it for 15...
Read more...
May 2, 2023

New Tech Disrupts Traditional Government Procurement

A Silicon Valley startup, GLASS, has created a revolutionary new tech, Glass Commerce, an e-commerce marketplace enabling compliant transactions between government buyers...
Read more...
April 25, 2023

Green Public Procurement: New Report

At Spend Network, we are always on the look out for governments, think tanks, research bodies, and other organisations who are working...
Read more...
April 25, 2023

EU Agrees on Anti-Coercion Instrument

The European Commission and European Parliament’s Committee on International Trade have agreed on an anti-coercion instrument (ACI) which will enable public procurement...
Read more...
April 13, 2023

£256 Million Contract for Cyprus Awarded by UK Government

The UK Ministry of Defence  has awarded a £256 million contract, for work, repairs, servicing, and facilities management for the UK defence...
Read more...
April 5, 2023

Endorsements For Export Agencies

Endorsements are a win-win for export agencies. Exporters face new challenges, both in terms of increased barriers to trade following the pandemic,...
Read more...
April 4, 2023

Looking For Reasons To Export to Government?

If you are looking to grow exports, you should start with the largest and most transparent industry in the world, and here...
Read more...
April 5, 2023

Lockheed Martin Selected As Preferred Bidder.

US defence giant Lockheed Martin has won a contract to deliver Australia’s first sovereign-controlled military satellite communication constellation. Lockheed Martin was recently announced...
Read more...

Newsletter

Compelling research, insights and data directly into your inbox.

Recent media stories

Search

Recent articles