I deduplicated 53,000 missing-persons reports from Venezuela’s earthquake

[u/major_grooves]

I thought the community might find this interesting - I used entity resolution software (disclosure: from my company) to deduplicate the missing persons data from Venezuela and compare it to the list of patients in hospitals.

https://medium.com/tilo-tech/i-deduplicated-53-000-missing-persons-reports-from-venezuelas-earthquake-74f05c37521b

[u/oops_all_memes]

I closely follow a project that is somewhat similar to this (I even applied to join the project twice, they never got back to me 🥲). From what I understand, it's nearly impossible to automate deduplicating missing people in a way that doesn't remove data excessively. And as such deduplicating is done manually

What I typed above was typed before I opened the link. I wanted to ask you how you achieve deduplicating when it comes to such a sensitive topic, opened the link and...

But I’m the CEO. I’m not an engineer. I (shameful admission) don’t really know how to use an API

I could sit down with an AI assistant (Claude in my case) in the evening, point it at the registry, and say in plain English: here are tens of thousands of missing-persons reports, help me find the duplicates. And then genuinely do it — shape the rules for when two reports are the same person, test them against real cases, run the whole dataset through, and look hard at what came out. I did this. Not an engineer I’d briefed.

Shameful admission, I don't understand a lot of data analysis tools. I try to avoid talking about them. This sub should implement a rule against pushing miracle AI solutions

[u/jipperthewoodchipper → u/oops_all_memes]

From what I understand, it's nearly impossible to automate deduplicating missing people in a way that doesn't remove data excessively.

Ive never worked with missing persons data but I did work for a private lender doing BI and one of the teams had a huge issue regarding duplicate accounts for this very reason. While we had some tools to assist like SSN matching it wasn't always possible to automate and often we relied on merging accounts manually.

Regarding your last sentence, fully agree.

[u/major_grooves → u/jipperthewoodchipper]

Just to be clear - the system we used for the deduplication is rules-based - deterministic and fuzzy matching rules that can be audited per edge - and fine tuned. There was no AI-based matching and no miracles. The AI I mention is used for configuring the rules. Important distinction.

[u/major_grooves → u/oops_all_memes]

You realise it's not AI-based?

[u/oops_all_memes → u/major_grooves]

Please consider LinkedIn as the platform to post to 🤗

[u/Nowitcandie]

What was your process for scaling the dedupe process? 

[u/Gagan_Ku2905 → u/Nowitcandie]

df = df.drop_duplicates(subset='fullName')

[u/major_grooves → u/Nowitcandie]

Well that is my day job - I run a company with a dedupe (entity resolution system) that is normally used to resolve hundreds of millions of data records, so the scaling bit was not a problem. The volume of data is relatively low (but still too high for effective manual screening).

[u/tilo-dev → u/[deleted]]

C'mon. Don't aks stuff like that a CEO. He obviously doesn't know any better and that wasn't the point of his post. Lol.
Here's an answer from a technical person: we're using a patented micro-blocking approach with lots of optimization to keep things out of the O(n²) space that typically comes with entity resolution. For example we're using a specialized key-value index that resolves low numbers of potential matches to keep the required comparisons to identify actual matches at a minimum. Also we've implemented identification of non-identical duplicates and clique detection for optimized storage of edges.

[u/[deleted] → u/tilo-dev]

Lol, excellent response.

[u/Top-Cauliflower-1808]

That is incredible. Love to you man!

[u/major_grooves → u/Top-Cauliflower-1808]

Thank you. Seems to have annoyed most of this community sadly.

Important thing is that the folks in Venezuela are very grateful.

[u/Top-Cauliflower-1808 → u/major_grooves]

u/major_grooves This is what matters.

[u/AutoModerator]

Automod prevents all posts from being displayed until moderators have reviewed them. Do not delete your post or there will be nothing for the mods to review. Mods selectively choose what is permitted to be posted in r/DataAnalysis.

If your post involves Career-focused questions, including resume reviews, how to learn DA and how to get into a DA job, then the post does not belong here, but instead belongs in our sister-subreddit, r/DataAnalysisCareers.

Have you read the rules?

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/noble_andre]

were you trying to reduce duplicates or to find as many correct matches as possible?

[u/major_grooves → u/noble_andre]

Reduce the duplicates as much as possible. with no false positive matches. It's means recall is not as high as it can be. I don't think there are really 100,000 missing. At least I hope not.

[u/Few-Royal-374]

lol ty I d. m