Why I Built DuplicateFinder Pro

I didn't set out to build software. I set out to shoot a graduation.

Even with just one camera, the same small annoyance shows up on every shoot: a RAW file sitting right next to the JPG preview the camera generated for it, both looking like separate files worth keeping, both quietly eating space for no reason. It's easy to ignore when it's one folder. It stops being ignorable the moment you're not working with one camera anymore.

Eight cameras. Ten hours. A Canon R5C running Cinema 4K at the anchor position, a FujiFilm 100S working a second angle, GoPros scattered across the room catching what the primary cameras couldn't reach. By the time the last GoPro came off its mount, I had footage and photos scattered across half a dozen memory cards, each camera with its own file-naming convention, its own frame rate, its own idea of what "organized" means.

That night — and every event after it — the same ritual played out: dump everything onto the drive, then spend hours by hand figuring out what was actually a duplicate and what only looked like one. A RAW file and its camera-generated JPG preview, sitting in the same folder, doing nothing but taking up space. A backup copy of a camera dump, quietly duplicated into the named event folder after I'd already organized it. The same photo, exported twice at different quality settings, now living in two different folders with two different filenames, invisible to any tool that only compares names.

That's the point where DuplicateFinder Pro actually started — not as a product, just as a script I wrote so I'd never have to do that by hand again. It didn't work. Not at first.

I tried the obvious fix first — match files by filename. It was worse than doing nothing. On one real event folder, a filename-only approach produced 349 false positives — files it called duplicates that weren't duplicates at all, just similarly-named shots from different moments in the ceremony. Deleting on that basis would have meant losing irreplaceable footage of someone's baptism because two files happened to share a naming pattern.

That failure is what actually taught me what the problem was. Filenames lie. Content doesn't. So I rebuilt it around perceptual hashing — comparing what's actually in the image, not what it's called — so a RAW file and its edited JPG export could be matched correctly even sitting in two completely different folders, renamed, re-compressed, re-organized. The same principle extended to video: exact-content matching that, on one real test folder, correctly identified 25.5 GB of genuine duplicate footage — camera dumps that had already been safely backed up into a named event folder, sitting there doing nothing but eating storage.

Because this had to work on real professional archives — some over 25TB, scans running for hours — it couldn't just be fast. It had to be safe. A scan that crashes four hours in, on a drive that size, is worse than useless. So the engine checkpoints itself at every phase boundary, caching hashes as it goes, so a paused or interrupted scan picks back up instead of starting over. And nothing gets deleted outright — flagged files move to a 24-hour quarantine first, with a full restore path, because the cost of a wrongly deleted family photo is not a cost any tool should ask a photographer to risk.

I'm a working event photographer first — I shoot weddings, baptisms, graduations, and church services, where a single service can run four hours and involve every camera I own running simultaneously. I'm also a developer. DuplicateFinder Pro exists because I got tired of losing evenings to a problem every event shooter runs into — and one no generic "duplicate finder" app actually understood — because none of them were built by someone who'd actually stood in that room, at hour six, watching the card fill up.

If you shoot events with one or more cameras, this was built for exactly your kind of chaos.