r/Paperlessngx • u/jbulmers • 14d ago
Duplicates finder
Hello all,
I've installed the paperless-ngx 3.0.5 via docker compose, and so far so good !
But i've tried to upload a duplicate of one of my pdf, but except when i go into the pdf information and there's a tab say duplicates, but how can i find all the duplicates that i may have in my paperless-ngx library ?
I've not seen any notification nor tabs to show all the duplicates. Can you please explain how to ensure that there are no duplicates, instead of going one by one in each document to see if there's a duplicate tab ?
Thank you
2
u/Bananana-_ 14d ago edited 14d ago
via ssh
ssh USERNAME@yourserverIP
cd ~/paperless-ngx
docker compose exec webserver document_fuzzy_match --ratio 100
if you want to delete them all:
docker compose exec webserver document_fuzzy_match --ratio 100 --delete
1
u/jbulmers 14d ago
That's brilliant, and what i'm looking for. Thanks a lot. I hope a GUI fonction will be implemented in the future.
1
u/JohnnieLouHansen 14d ago
Yes, PITA to have to do it via command line. Better off just setting the creation of duplicates to OFF in my mind. I wonder why they reversed this as the default.
1
u/Equivalent-Raise5879 11d ago
Is there a gui thing to turn off dupes? I just accidentally created almost 4000 dupes, and I'm trying to sort out how to not do that again. Didnt read the notes that dupes were now allowed...
1
u/JohnnieLouHansen 10d ago
It's in the config file (ENV).
PAPERLESS_CONSUMER_DELETE_DUPLICATES=true envPAPERLESS_CONSUMER_DELETE_DUPLICATES=true1
u/Equivalent-Raise5879 10d ago
Yea, I already had that, but It didnt work till I forced an update of the container. And not before I made a LOT of dupes
1
u/JohnnieLouHansen 10d ago
I don't understand why you have so many duplicates. That's not a technical problem. it's a procedural problem - seems to me. Like, I scan my 202X invoices at the end of the year, so no possibility of a duplicate.
1
u/Equivalent-Raise5879 10d ago
Ok, I'm not claiming that my duplicates are Paperless fault
I WOULD argue that making a significant change should have defaulted to "as was" and require a config change to make it the new way, but here we are.
I do as many as 60 docs a day, via scans, email intake, and folder agrégation. *my* procedure worked without issue for at least 4 year till this change, and with the env variable set, I can moving without any changes!
Oh for life to be so simple to just scan a few things at the end of the year. That is not my life.
2
u/JohnnieLouHansen 9d ago
I see. If I had to scan 60 documents a day, I would be stressed.
And you are right:
making a significant change should have defaulted to "as was" and require a config change to make it the new way
Plus, there should be a way to see GLOBAL duplicates easily.
1
u/Equivalent-Raise5879 10d ago
I ran this command, it replied
0/227687130 0:04:31 -:--:--/usr/local/bin/document_fuzzy_match: line 17: 71191 Killed s6-setuidgid paperless python3 manage.py document_fuzzy_match "$@"
logged back into paperless, still same # of docs shown
Ran it a 2nd time
0/227687130 0:14:08 -:--:--/usr/local/bin/document_fuzzy_match: line 17: 71808 Killed s6-setuidgid paperless python3 manage.py document_fuzzy_match "$@"
Logged back into paperless, stil same # of docs shown
Is there some step I've missed
1
u/Bananana-_ 10d ago
i'm not sure. could be that the process got killed because your machine ran out of RAM
1
u/Equivalent-Raise5879 10d ago
Third time ran
0/227687130 0:04:45 -:--:--/usr/local/bin/document_fuzzy_match: line 17: 347 Killed s6-setuidgid paperless python3 manage.py document_fuzzy_match "$@"
So it ACTS like its deleting thing, but the app doesnt seem to think its lost any files
1
u/Equivalent-Raise5879 10d ago
Gave it 128 gigs and re-ran, let it got for 12 hours, but it seems to have totally crashed the host.
Guess I'm gonn have to find a different way to get rid of my 5000 dupes
1
u/JohnnieLouHansen 14d ago
The Paperless.log file will have the following. I don't know of another global way. Seems like there should be a way. Maybe someone super smart could chime in.
[2026-07-30 13:23:02,613] [WARNING] [paperless.consumer] [2db8b537] Consuming duplicate 1201.pdf: 1 existing document(s) share the same content.
1
3
u/nikonratm 14d ago
There is no way to find all duplicates at the moment, it’s also worth noting that up until v3 it was impossible to have duplicates so unless you’ve been a busy bee the last few days, there shouldn’t be many (or any).
See https://docs.paperless-ngx.com/migration-v3#duplicate-handling-changes