7 Rules for Data Integrity

December 17, 2020
Data

Not all data is the same. It might have come from the same source, but how it gets treated is vital. If a data company doesn’t have good data hygiene practices things can get messy very quickly, making it hard to understand the data or undermining your valuable analysis.

  1. Always know where the data came from → we always record the precise source of every record, so that our users can always go back to the original source and validate each one of our records.
  2. Always know when the data was collected → we don’t just record where we got the data from, we also know when we collected the data, so if the source changes, we can change.
  3. Never overwrite source data → we know that some of our data needs to be improved, if a date is incomplete or a better category can be added, in doing this we always add to the data, rather than overwrite the underlying data.
  4. Generate metadata → Our clients want to be able to filter data on attributes that we create, the language of a record for instance. We augment our data with useful metadata every time we gather a record.
  5. Handle duplicates with sensitivity → We see a lot of duplicates and some records that look like duplicates but aren’t. So we don’t provide a binary ‘on’ or ‘off’ analysis of duplicates, we look at eight key attributes and then score these to provide a good understanding of whether something is a duplicate.
  6. Matching needs manual checks → Entity matching is incredibly hard to get right, algorithms can help, but in the end, every match that isn’t an exact match needs to be checked, to make sure that a match is correct. We do this because the details matter, and if we get a contract award wrong, then it can impact investment decisions.
  7. Be ready to highlight anomalies → We wished that some of the records we gathered were better formed, had better information, or just had the data that they were supposed to have. We have to accept that this isn’t always the case. So where things aren’t right, we don’t shy away, we don’t pretend that everything is rosy, we tell our users where the problems are, and let them budget what’s best.

We think that how we approach quality matters. We don’t tell you things you want to hear just to get a sale, we tell you what we know. We want to build partnerships, not future problems. If you’d like to know more about our data or our research services, get in touch.

contact@spendnetwork.com

August 3, 2021

New Carbon Reporting To Impact 1,000 Suppliers

New rules to help reduce CO2 in public contracting show that at least 720 suppliers will be required to provide a detailed...
August 3, 2021

Trade, Policy And Procurement Meet Again.

The UK's Trade and Agriculture Commission has published its final report, along with twenty-two recommendations, one of which is to direct the...
August 3, 2021

McGowan Government Enhances Aboriginal Procurement Policy

In 2018, Western Australian Labour Premier Mark McGowan introduced a landmark Aboriginal Procurement policy. The objective of the policy was to improve...
August 3, 2021

Are Trade Agencies Becoming More Like Hedge Funds?

Trade promotion, where a country promotes its companies and products to be bought by another, has a certain image. One imagines suave...
August 3, 2021

Meeting The Research Challenge

Our clients expect us to answer tough questions. Questions like, "what's the total value of public contracting worldwide?" or, "who's the leading...
August 3, 2021

US Consults On Social Procurement

The US Government's Office of Management and Budget (OMB) has requested submissions on the "methods and leading practices" to assess whether agencies'...
August 3, 2021

Harnessing Procurement To Tackle Climate Change

As experts in procurement and open data, we are pleased to see the recent announcement that global confectionery giant Mars is harnessing...
June 29, 2021

Western Australian Govt Rolls Out New Procurement Process.

We're always pleased to see governments reviewing their procurement processes with a view to greater transparency. The Western Australian Government has recently...
June 9, 2021

Commit To Net Zero To Win Government Contracts.

....So says the Cabinet Office, with new measures released on World Environment Day that will require businesses to publish clear and credible...
June 2, 2021

Major Procurement Project For Inverness Castle

This week sees Highland Council publish the main construction and refurbishment tenders for the Inverness Castle Transformation project, through Public Contracts Scotland...
May 27, 2021

Six Considerations When Marketing To Governments.

We might say marketing to the government is similar to marketing to any other target market, particularly another B2B target market. And...
May 17, 2021

Data Mastery.

It's easy to say that you're a 'world leading' enterprise. It's much harder to back it up. But we genuinely believe that...
May 4, 2021

Recommendations For Tackling Gender Disparity For Governments

Clearly, the UK Government is a leader in the field of open data and transparency: it is this commitment to transparency as...
June 2, 2021

Does Government Business Correlate To Gender Disparity?

As part of our Procuring Inequality report, we've analysed whether winning government business equates to a gender pay disparity. By linking gender...
May 5, 2021

The Worst Performing Suppliers To Government In The Gender Pay Divide.

As our analysis of the gender pay gap within strategic suppliers to government from our Procuring Inequality report, showed a broad overlap...

Newsletter

Compelling research, insights and data directly into your inbox.

Recent media stories

Search

Scroll to Top