Hey Jow Forums, I'm looking for a specific type of backup solution.
I'm going to be backing up from time to time to external HDDs. Ideally I want something automated or semiautomated to sync the new data when the HDDs are plugged in. The thing is, I will be backing up very large files and I want something smart like syncthing that will only add the changed parts of the files instead of coping the entire files again every time.
Something cross-platform is preferred because I currently use windows but plan on moving to GNU fully soon. If not possible, two similar solutions (one for each OS) is acceptable.
And last but not least, I would really rather a free software option, or at least open-source. Suggest proprietary software only as a last resort, please.
Any ideas? Also, does anyone here do backups in a similar manner?
Backup solution
Other urls found in this thread:
Duplicati
Sounds good, but not exactly what I'm looking for. I want my backups to be fully accessible directly from the drives, without the overhead of restoring from the backup first.
Also, it seems like this would also bloat excessively with time. I want to record the changes made to the files, not additions that will change the files when restoring.
I might consider using this for cloud backup, though, so thanks.
Absolutely no warranty, but if you feel lucky enough you can use my implementation of a backup tool for big files.
I use it to backup my 2 tb big files to external drives. The current implementation is windows only but i think wine should make it work on linux without any problems. Good luck in finding a solution.
>tfw trusting all your data AND your backups to the hacker known as 4channel.
Seriously, though. It sounds interesting. Thanks for the share.
GNU/Veeam
It's open source, no need to worry
A shell script.
I'll take it if you have one.
Why not write one yourself?
* You want diff backup
* You want the files to be directly accessible
* You are storing large files
* You want to do this on Linux
A solution that werks on Loonix and isn't some gentooman's github is ZFS snapshots + send/recv. Have all your drives formatted as ZFS and use send/recv to push snapshots to your backup.
lolwut?
If I knew how to do that I wouldn't need to ask Jow Forums for assistance. I don't even know what tools would be necessary for my use case. I will take nudges in the right direction if you have them.
Unfortunately zfs is not an option. My hard drives are already formatted as ntfs and they'll sadly have to stay like that for the rare occasion I might need to access them on windows. Also, on GNU I'll be using btrfs, but zfs. But thanks.
Does anyone else have some ideas?
I don't know, man. How big are the files that we're talking about here?
Nothing extreme like the 2 TB this user uses their program with, but some can be upwards of 20 GB which already would be very overkill to move back and forth every week on a USB 3 connection. Not to mention the unnecessary writes to the HDD. That's why I need to send only the changes to the files, which apparently is called delta.
rsync --inplace --no-whole-file
Thanks. After researching for a bit it does seem like rsync would probably be the logical choice. Unless anyone else has a better solution.
This might be a solution if you don't expect any speed improvements. It will write less to the hdd but it will also take much longer than a regular copy, just saying it. Rsync has to read the section from the hdd before it can compare their contents. This will lead to increased copy times.
cron and rsync - google it, it'll take about 2 min to setup if you've got half a brain.
Why would reading from the external hard drive take longer than writing to it?
Because if you copy over you just override the bytes without knowing if anything had changed.
Rsync has in addition to writing the bytes the overhead of reading all bytes, also the bytes which didn't change and calculate the checksum for each chunk
I'm actually reading about udev and rsync now. Apparently with that I can make the backup happen when I plug the drive in, which is the level of automation that I wanted. Of course it's no a solution for windows right now, but I think it will come in handy when I move to GNU.