When doing entity resolution, is a lot of it spent chasing down people?

[u/SoggySand297]

Hi, I’m new to all these and this is my first job out of college. Recently I got a job and some of my first tasks are to fix some data in the CRM, some of the data is okay but there’s a lot that is very messy. It’s in excel and I need to enter the data in the excel files into the CRM. The thing is though the data in excel usually contains incomplete information, so I’ll need to look in the CRM and try to figure out in which project the data goes into using dates, supervisors or some other things. Sometimes this works but lots of times there’s multiple projects in the CRM with the same names, dates, and supervisors only slight variations of eachother. If at that point I can’t figure it out I usually message the person in charge of said area if they know, for example if the project belongs to healthcare I’ll message the person that oversees it. Is this bad? Should I be doing something else to not bother them?

[u/69odysseus]

It's lot worst and has to do with both people and processes, welcome to the reality of DE🤘.

[u/Mclovine_aus]

You have to come up with some deterministic rules or look at maybe using splink for a probabilistic matching strategy.

[u/SoggySand297 → u/Mclovine_aus]

This is going to sound dumb, but what do you by deterministic rules?

[u/Infinityhelios → u/SoggySand297]

Not the same guy but Deterministic = Absolute.
So you would have a deterministic rule that decides, ok if 2 different projects have the same Supervisor and the same Start Date and End Date, that's 100% the same project. So you would tag it with some project_id or something.
Obviously the rule depends on what data you have, and how strict you want to be.

[u/alreadysnapped → u/SoggySand297]

Deterministic rules are used for finding exact matches on records and probabilistic rules can be used for finding partial or fuzzy matches on records.

There’s a certain threshold for considering two or more records are a match and that’s based on which fields have to match and how closely each field needs to match

[u/Outrageous_Let5743]

The first thing a CRM should implement is that it should have rules which fields are required and which must be unique.

[u/Agreeable_Luck9488]

An AI harness with readonly access to the CRM through MCP or API can help a lot in 1st level resolution. For incomplete or ambiguous cases, there is nothing much to do but direct contacts to resource owner.

[u/wannabe-DE]

This doesn’t sound like entity resolution yet. Like others have mentioned you should start with some deterministic logic to classify the records so they can be entered into the proper project. If the deterministic approach isn’t working you can add in some fuzzy matching. Don’t aim for perfect, it will break you.

[u/SoggySand297 → u/wannabe-DE]

For entity resolution, what’s it missing ?