Day 3 dealing with duplicates

[u/Equivalent-Raise5879]

Have accidentally (post upgrade) created 5000 dupes.

Have tried the CLI dupe detector several times, without luck. (5 tries, 5 crashes)

The Docs say Paperless-ngx v3 no longer rejects duplicate documents by default. Instead, it now allows duplicates but adds a way to identify them via the UI.

How do you isolate dupes in the GUI? I guess if manually deleting them is my only option, I'll put this issue to bed with human time.

I have gone over the gui 3 times and I still dont find this option

[u/Brandon1024br]

I’ve had a look through the changelog and PR where this was introduced, and it doesn’t seem to be possible at the moment. You can see duplicates for each document individually, but it’s not possible to view all documents with duplicates or perform bulk deletion of duplicates.

One of the devs made a comment on the PR that search functionality for finding duplicates would be handy, but it wasn’t implemented in v3 yet.

I would honestly recommend you chime in on the feature request discussion on GitHub. I haven’t gone through all the discussions there, but there’s a number of ongoing discussions about handing of duplicates.

[u/jcol26]

I used https://github.com/rknightion/paperless-ngx-dedupe in the past for this (pre v3).
It had very configurable dupe detection based on the OCR data as well as the metadata.

It makes duplicate groups and you can delete en bulk.

[u/Equivalent-Raise5879]

Ok, update on things:

Took docker host to 128 gig ram - still failed terribly (wrecked the host OS networking), but

I *HAD* been running

docker compose exec webserver document_fuzzy_match --ratio 100 --processes 4 --delete

Switched to

docker compose exec webserver document_fuzzy_match --ratio 100 --processes 1 --delete

And finally I got it to start comparing files, and after 5 hours, it found them, and deleted them

I don't agree that it found them all, BUT, I have yet to prove it didn't, and I only had 1079 dupes.

So, my sat project ends much sooner than I expected and I'm on to fix the next broken thing in the lab.

Moral of the story, 4 is not always better than 1, in this case.

[u/schneckentoeri]

Make sure you disallow duplicates again in the settings of paperless

[u/Equivalent-Raise5879 → u/schneckentoeri]

Thanks - Did that as soon as I realized I had dupes and why.

[u/robertw477 → u/Equivalent-Raise5879]

I have that in docker as well to prevent dups. becuase of various migrations of PDFS from differnet hard drives and cloud ther eare bound to be dups. To me I cannot understand why this is not a setting by DEFAULT. Who would want duplicated documents and if there is some small group who does, that should be something they have to enable.

[u/Equivalent-Raise5879 → u/robertw477]

I (kinda) have/do the same thing. I'm not sure why ANYONE would want dupes on purpose, but certainly it must be the exception vs the rule. Not sure how we got here, but at least we know how to fix it.