My father likes taking thousands of photos, transfer them from the camera to his laptop, and then make backup upon backup upon backup of the same goddamn files scattered throughout his absolute mess of a folder structure. His system drive is an SSD, that local storage space comes at a premium, so I've granted him access to an area of my spinny rust RAID1 array, where he can then upload his shit over network.
Now for obvious reasons I'd like to avoid my precious storage wasted by him sending tens of thousands of files which all have several duplicates at different file paths on the same physical hard drive.
Can anyone recommend a free piece of software that can sift through a given set of folders, look for duplicate files, and present the results to me for final action? Attributes such as file name, size, and date modified would be enough for what I need.
Uh... so he keeps everything on one SSD? Did you explain the concept of a backup to him?
Eli Cook
that sounds like a job for a 6 line bash script, yeah
Levi Robinson
I recommend rsync
Liam Cox
Fdupes
Ian Hernandez
In an easily stealable portable device, yes. He's a sixty-some year old boomer, you know how it goes.
Jacob Anderson
>use an image size optimizer for each file >sort by file size to find dupes solves 2 problems easily
Daniel Walker
Dont ZFS and BTRFS have offline dedup?
Jace Roberts
So you'd fuck up all the original files to get rid of duplicates?
Bravo.
Blake Lewis
>what is lossles compression
Nicholas Gray
>I've granted him access to an area of my spinny rust RAID1 array don't. tell him he needs to get his own home NAS or cloud space.
Brandon Rivera
make a script that compares md5 and transfers those that are unique. idk.
Ryan Powell
Ultimately if he can't manage to clean his shit up he'll just be paying for new higher capacity drives when the volume runs full. The last thing I need is another separate NAS to manage (if he got one for himself that would for sure be up to me, too), which is why I offered some space on my already established hardware.
Thanks for all the suggestions so far
Nathaniel James
Depends on what format the files are in. Even if PNG is lossless you still lose data when converting RAW to PNG.
Kevin Lewis
It actually turns out CCleaner (the non-pozzed version from before Avast took over) has a nifty little duplicate finder tool that quickly spits out a detailed list of directories which hold identical files based on a number of conditions you define in the search. Thanks guys!
buy some oldass harddisks and some ancient PC from craigslist (cost you less than
Jaxson Garcia
Visipics my dude
Ryan Cruz
For the future, he should use Adebis Photo Sorter Very simple software that copy photo (jpeg and raw) in YYYY/MM/DD folder structure (or other). Its a proprietary abandonware but I like it. Personally, I use this + Nextcloud server for backup. I have like 600GB of photos.
Luis Gutierrez
If he doesn't change the files themselves, only the name, can't you just MD5 them then delete duplicates? Should be a pretty simple python or bash script, no?
Nathan Reyes
This RAID1 array is a headless computer?
Zachary Peterson
borgbackup is the answer, pic related is what i call with cron
>Now for obvious reasons I'd like to avoid my precious storage wasted by him sending tens of thousands of files which all have several duplicates at different file paths on the same physical hard drive. How about you simply create a TrueCrypt/Vera Crypt container of whatever space you want to give him and simply mount the container and give him the access to this file?
If you create a container of 1TiB or 500MiB or whatever he can not get over this limit. I think there are other ways to create limited space for your father. Something with VDI files or some shit.
Why the fuck are you even giving him access to your server did he threaten you to give him your space or what? >user you ungrateful fuck you horde all your HDDs for yourself give your father some I buy him a external HDD and call it a day.
Personally i did have pretty illiterate father I have gotten away with taking his money to "buy him 8TB HDDs" and Buying the 8TB HDD using it in my computer and taking a old 8 GB USB stick i used and giving this to my father. He was happy and it all checked out he did have his blazing 8 whatever space it was enough for him for the next years.