Does Your Data Have Integrity?

Does Your Data Have Integrity?

October 20, 2022
Data

Not all data is the same. It might have come from the same source, but how it gets treated is vital. If a data company doesn’t have good data hygiene practices things can get messy very quickly, making it hard to understand the data or undermining your valuable analysis.

These are our rules for ensuring data integrity:

  1. Always know where the data came from → We always record the precise source of every record, so that our users can always go back to the original source and validate each one of our records.
  2. Always know when the data was collected → We don’t just record where we got the data from, we also know when we collected the data, so if the source changes, we can change.
  3. Never overwrite source data → We know that some of our data needs to be improved, if a date is incomplete or a better category can be added, in doing this we always add to the data, rather than overwrite the underlying data.
  4. Generate metadata → Our clients want to be able to filter data on attributes that we create, the language of a record for instance. We augment our data with useful metadata every time we gather a record.
  5. Handle duplicates with sensitivity → We see a lot of duplicates and some records that look like duplicates but aren’t. So we don’t provide a binary ‘on’ or ‘off’ analysis of duplicates, we look at eight key attributes and then score these to provide a good understanding of whether something is a duplicate.
  6. Matching needs manual checks → Entity matching is incredibly hard to get right, algorithms can help, but in the end, every match that isn’t an exact match needs to be checked, to make sure that a match is correct. That’s what we do, because the details matter and if we get a contract award wrong, then it can impact investment decisions.
  7. Be ready to highlight anomalies → We wished that some of the records we gathered were better formed, had better information, or just had the data that they were supposed to have. We have to accept that this isn’t always the case. So where things aren’t right, we don’t shy away, we don’t pretend that everything is rosy, we tell our users where the problems are, and let them budget what’s best.

We know that data quality matters. We won’t tell you things you want to hear just to get a sale, we’ll tell you what we know. We want to build partnerships, not future problems.

If you’d like to know more about our procurement data, our API or our research services, get in touch.

November 29, 2022

Data and Public Procurement.

There are some good reasons for including procurement data in your alternative data strategy. Governments are big and getting bigger. Before the...
November 29, 2022

Ensuring Data Integrity.

Not all data is the same. It might have come from the same source, but how it gets treated is vital. If...
November 24, 2022

Govt Issues £4.2bn Tender To Update Legacy Services

Crown Commercial Services together with HM Revenue and Customs (HMRC), have released a contract notice for the Digital and Legacy Application Services...
November 22, 2022

‘Unboxing’ £120 Million Of Public Spend

It may have been branded twice, but it seems nothing could save The Festival of Brexit, AKA Unboxed, from tanking in the...
November 17, 2022

We Sell Data, Not Seats.

Why we sell data not seats? We're data analysts. We really understand the data we gather. We've been doing it for 15...
November 15, 2022

Saab Voices Opposition To Canadian Gov Procurement

The Public Services and Procurement Canada (PSPC) agency has begun negotiating the final terms of delivery with Lockheed on an order from...
November 10, 2022

Your Data, Your Rules

If your company is hiring data analysts, you need data. You don’t need a login to a website which restricts the queries...
November 28, 2022

Life Moves Pretty Fast

We've been talking for a while about AI and possible impacts on procurement. We use AI in our own service to almost...
October 27, 2022

Lessons Learned In Public Procurement

The National Audit office is working on a Lessons Learned report, highlighting opportunities for change in public procurement. Topics will include commercial and...
October 13, 2022

Australian Government Covid Procurement Findings

As the world emerges from the pandemic, governments are reviewing their procurement of Covid related goods and services. The Australian Government tasked...
October 11, 2022

Hungary Passes Transparency Bills.

Last week, Hungary's government passed the first two of 17 anti-corruption legislation bills, in a bid to avoid losing €7.5bn in financing...
October 6, 2022

Babcock Lands Polish Defence Contracts.

The UK Government has this week signed major major defence equipment agreements with Poland’s military capability. At the Zamość Military Base in...
October 4, 2022

New enRichMyData project offers innovative solutions for data enrichment

Several European and international organisations, covering large corporations, SMEs, research centres and universities, are joining forces to develop new methods and tools...
September 29, 2022

£900k Government Fund To Help Charities Win Public Contracts.

The Department for Digital, Culture, Media & Sport (DCMS) is running a Voluntary, Community and Social Enterprise (VCSE) Contract Readiness Fund grant...
September 27, 2022

New EU Procurement Instrument Now Law.

About a year ago, we wrote an article on the new procurement instrument approved by the European Union. In recent weeks this...

Newsletter

Compelling research, insights and data directly into your inbox.

Recent media stories

Search

Scroll to Top