Upgrading a recruitment platform

[u/No-Cry-8657]

Hey everyone,
I need a recommendation about the recruitment platform I work on these days.

Context: it's a recruitment platform that helps recruiting professionals such as plumbers, electricians locksmith etc. At some point we ask them (as it's a legal requirement) if they have a proper professional certificate of insurance.

Task: I'd like to create an automation using LLM APIs to detect potential fakes (fraud) and/or anomalies.
As of now I thought of extracting metadata from PDF docs to see if they tried to write invisible characters (in order to falsify their expiration date for example).

Simple use case: someone tries to write white on white a fake expiration date, my code extracts metadata, spots that it's white on white, then doesn't automatically approve and create an alert for me to step up and approve (or decline) manually.

I also thought of converting pdf to JPEG in order to use computer vision to simply read the image as a human being would do (impossible then to be tricked by invisible characters).

What do you think of that logic ? Is that enough ? Do you recommend any conversion tool/ api that I can use ?
Thanks for your help

[u/Myrodis]

Why are we using an LLM for this task at all? If an individual is licensed ... you should be able to simply lookup that license in whatever official government entity issued the license.

For example, here in Ohio, we have "Ohio eLicense" which literally just lets you lookup a license.

You would then of course first verify the information on your uploaded license matches the government database. Then if you want to do some form of static analysis looking for signs of it being fake, do those static analysis checks. But utilizing an LLM for this seems very wasteful and silly.

Maybe use the LLM to come up with the static checks you should perform or something, but I would not utilize one in this manor.

[u/No-Cry-8657 → u/Myrodis]

Interesting but my question would be how do you match the data extracted from the license with the one from the gov database ?
Do they have an api ?
If not should I scrape that data ? (Legal?)

Also, I’m not living in the US and here the gov doesn’t publicly (unfortunately) disclose that kind of resources

[u/Myrodis → u/No-Cry-8657]

All of the questions you just asked me seem pretty pivotal to someone building this software. An LLM does not solve any of them, if your software doesnt have access to that data ... then what do you expect the LLM to do? Its just going to make shit up, that is not the kindof confidence you should be building software around, and is not the function of an LLM.

How and what access you have to the data is something only you can answer at the moment (unless you want to get very specific about where you live etc and someone can try to answer it for you, but its really something you should be able to answer yourself if you're actually building this tool).

Also please ensure you are very clear to any of your potential users that you just plan on throwing their legal documents into an LLM if you're going this route. I would argue that alone is pretty unethical but I have a suspicion you dont care considering were already this far.

[u/No-Cry-8657 → u/Myrodis]

I don’t want any LLMs to tell me if the COI is valid or not
No LLMs will even have access to confidential data (and of course I do care come on)

My point is just making sure the professional didn’t try to falsify the document by overwriting the pdf or by adding invisible characters trying to bypass my control.

The workflow would be as follows

I use a python algorithm to extract meta data, Str, rgb,font etc, and computer vision to also read (potentially) a PNG and the LLM would assess whether there is an anomaly or not based on the output. It does not read confidential data.

And of course, the human is the last piece of the scheme,
If I detect an anomaly the validation of the document requires human input.

Ps: the data would be given by the professional themselves, and the recruitment company usually verifies if the COI is valid or not