Deduplication Routines

Analysis, Data / By / February 28, 2023 / 1 minute of reading

Deduplication Routines

February 28, 2023
Analysis, Data

Deduplication is a deceptively complex problem to handle at scale. There’s a simple, ugly and brutal way to do it, where you compare two records and determine if they are exactly the same, but this doesn’t work if the records are very slightly different. A missing comma is enough to render that algorithm useless.

Doing deduplication properly means more than just mashing together texts, its about building the infrastructure and referencing that allows you to narrow the body of potential duplicates, whilst also providing accurate and intuitive analysis of what constitutes a duplicate.

Augmenting texts with machine learning and then using that to refine the matching process and similarity scoring means that our deduplication algorithm is 99% accurate across a body of over 50,000 live tender opportunities each day.

At Spend Network, we have government procurement data for more than 150 countries globally. We take the hard work out of collecting, organising and analysing procurement data at scale. Get in touch to find our more about our data products and services.

March 21, 2023

AUKUS Defence Procurement Deal

Australian Prime Minister Anthony Albanese joined the US President Joe Biden, and the UK Prime Minister Rishi Sunak, in San Diego this...
Read more...
March 14, 2023

Australian Government Responds to COVID-19 Procurement Review

Many governments around the world have faced pressure regarding their procurement of COVID-19 treatments and vaccines, so it is encouraging to see...
Read more...
March 8, 2023

EU Joint Ammunition Procurement

EU ministers will meet today in Stockholm to decide whether to  jointly procure  ammunition to aid Ukraine and replenish domestic stockpiles. EU...
Read more...
March 7, 2023

Denial Of Service By Wall Of Text

Clarkesworld Magazine is a publisher of science fiction short stories. They accept and pay for submissions from anyone. As of last week,...
Read more...
March 1, 2023

Canadian Government Announces Rail Procurement Process

This week, the Canadian Government announced a major step in the procurement process for their High Frequency Rail project: the launch of...
Read more...
February 23, 2023

Policy Proposal: Analysts In Place

Data is critical to the functioning of government. Policies and services live or die on the quality of data. Successes and failures...
Read more...
February 23, 2023

Hallucination As A Service

I asked Chat-GPT to find me some academic papers that highlight the usefulness of combining two different machine learning techniques. I hoped...
Read more...
February 20, 2023

Future Trends in Government Procurement

Over the years, government procurement has undergone significant changes, with new trends emerging that are reshaping the way governments buy goods and...
Read more...
February 15, 2023

Making Your Own Export Platform?

If you are building your own export platform, sourcing, organising and analysing your data has its challenges. Here are some of the...
Read more...
February 14, 2023

Tackling Modern Slavery in Supply Chains – Guidance

On 26 March 2020, the UK became the first country to publish a Government Modern Slavery Statement, setting out the steps government...
Read more...
February 8, 2023

Securing The Competitive Edge.

Are you an exporter looking for leads for your exporters? Do you want to help your in-country teams build more partnerships within...
Read more...
February 7, 2023

The Benefits of Data Tech 101

Government procurement, the process by which government entities purchase goods, services and works from private businesses, is being revolutionised by the integration...
Read more...
February 1, 2023

EU Awards New Gas Aggregation Contract

In December 2022, EU energy ministers agreed to create a platform for the collective demand aggregation and joint procurement of gas, with...
Read more...
January 25, 2023

Does Procurement Need Blockchain?

Blockchain is fundamentally a database, but rather than a database where one item is allowed to replace another, each change to the...
Read more...
January 25, 2023

Mixing Politics & Procurement.

The need for administrations to act at pace is often at odds with the processes and procedures needed for good procurement. If...
Read more...

Newsletter

Compelling research, insights and data directly into your inbox.

Recent media stories

Search

Recent articles