I finally organize all 11 terabytes of data I collected over the past 2 years...

I usually buy 30-40 dollar 2tb refurb drives.

Just to add on to your post, there's a purpose-built variant of the Odroid XU4 that's intended to be used as a NAS called the HC1 (for 2.5" drives) and HC2 (for 3.5" drives).

It's a cute, stackable little thing.

Attached: ODROID-HC1-Figure-0.png (800x535, 795K)

Probably okayish for the next 1 to 2 years. My new 8tb drive arrived yesterday but it's still just jbod in my desktop. I went from 8tb to 16tb so for now I copied all my other data over to the new drive to make it redundant for a little while before I need the storage again.

I'm currently saving to get a dedicated NAS/server but I don't want a prebuilt so it's actually kind of expensive since I want at to be able to use at least 8 drives with preferably ZFS. I just hope I can sort my finances before some disaster strikes.

I guess I don't have that much data but I do have a premium OneDrive account about 50% full of porn and anime

Attached: 6d95393.jpg (800x1138, 112K)

That's also an option. Agreed, they are cute and stackable.

But as a downside, they don't have a HDMI out and less USB ports, so they're not really suitable for any secondary use as HTPC / web browser machine.

Also, actually stacking 3+ pretty firmly puts them in a price range comparable to building an x86 machine.

>I want at to be able to use at least 8 drives with preferably ZFS
You could save a good bit of money while achieving the same performance and nearly the same features if you just used Linux md/dm RAID...

MD Raid doesn't save you from bit rot though, only from drive death, unless I'm very much mistaken.

>defragging archives
Pls tell me you did not do this

Well mainly the deciding factor would be the RAM prices whenever I will build the NAS. If they're still sky high I'd probably just go without ZFS. However if RAM prices were to go down I'd only be looking at a small premium for using ZFS.

Oh and I'm not going to use ECC. Prices are even higher and if a RAM kit can manage at least 10 memtest+ runs it probably has a 99.9% chance of doing fine.

This and write holes are the primary reason I want to use ZFS. Datasets and snapshots are a nice extra.

29TB sadpanda including metadata and fakku backup, along some MMD models and videos and japanese artist twitter backups where i still have to delete food pics.
400GB reaction image folder that started back in 2009.

Storage devices used:
12x4TB WD Red.
113x500GB 2.5" HDDs i get from work that are used as backups of the backup that i have in my house, hangar and other places in case of a SHTF situation.
23x250GB 2.5"HDDs used for anime backups.
2x1TB 970 EVO.
1x500GB 960 EVO.
2x250GB 850 EVO.

Also that is a cute happy potato, i still feel bad about the ending.

Attached: tkmiz-967570855965310977-20180224_182403-img1.jpg (1536x2048, 860K)

Use a journaled file system. Dumbass

I'm not terribly worried about the write hole, but the checksumming has saved me several times from drives coughing up bad data.

GB reaction image folder that started back in 2009.
jesus user, I've been at it longer than you and my folder of images saved from *chans only totals about 40K files and 15.5GB. Do you just scrape everything all the time?

The size is mostly related to the amount of videos i have, it has Videos, flashes, stuff that has been slowly removed from the internet related to Jow Forums in any way, the Do/k/ument is 80GB alone.
Ar/k/ is 450GB but i have it outside my reaction image folder.

pardon the stupid question, but i think this is fitting since the subject is data hoarding.
How do you guys label your Jow Forums images? Reaction images and shit like that? Do you just save them with the original user post number, or you guys label it with something else?
Sometimes my image folder get so big, I lose my reaction images in a sea of shit, and when I want to use 'em, i can't find it.

I just put the reaction images inside a folder called reaction faces that i use often, screencaps for screencaps, and so on.
I do rename the ones that i use frequently.

>not deleting shit after you watch it

why?

>its time to defrag
either you shouldn't have to since moving a file usually defragments it, or you fucked up and were doing multiple parallel file copying simultaneously via Windows Explorer while having low free space

Reaction images I (try to) keep in a separate folder than just general non-reaction images that I happen to like/find funny. If there's text on them I use that for the filename.

You've never ever wanted to watch something again after you've seen it once, user? I certainly have, fairly frequently.

>You've never ever wanted to watch something again after you've seen it once, user? I certainly have, fairly frequently.

Generally not. Why not just re-download when you want to watch it again?

i have tons and tons of images dating back to '05~06 but I've never had the patience of renaming then, so the filenames have just the post number of when I picked them up.
It's complete chaos, no order. No subfolders, just one fucking folder with 10, 20gb that I've been hoarding since I was a teenager.

thanks for the tips guys

Because there's a good chance it won't be there anymore. Shit gets DMCAed from Youtube and porn tubes all the time, and that's only getting worse. Torrents die after not very long. Shit happens like the big Nyaa shutdown.

And then I'll be both sad that I can't watch the thing I wanted to watch and angry at myself for not saving it when I had it. You may recognize the voice of experience here.

Because some shit disappears eventually, and finding someone that has it might be a problem.
>Tfw all those youtube videos deleted i didn't save

> Using wandows
Found your problem

So I can make webms to post on /tv/ later if I want.

>29tb of anime titties

why do people do this

So if the panda ever dies i could share it so clones spread.
See Mangatraders dying and Mado being created from the backup Blue had, without it it would have never existed.

>defrag
Faster to copy to another drive and copy back.

>defrag
Holy shit

It can actually possibly fix bit rot on a drive if you use RAID6 and occasionally scrub.

Or you can augment your main files with par2 data. That also allows to detect and fix bit rot.

Lastly, bit rot isn't a huge issue for media files and stuff. Even if one bit in some Terrabytes is corrupted eventually, you'll likely not notice it very much.

> 29TB sadpanda including metadata and fakku backup
Sadpanda fits on just 29TB? Or is it just your partial set?

just had to retorrent my jamesbond collection casue some of them were fucked up. now i got them in 720p woot

>>It can actually possibly fix bit rot on a drive if you use RAID6 and occasionally scrub.
Not really, since it doesn't do checksumming. It can notice that block X is inconsistent with its parity information, but it can't tell whether that data is corrupted or the parity is. It just has to pick one. Same is true of RAID 1, it can find the mismatch, but it has no way of determining which drive gave it the correct data.

>Or you can augment your main files with par2 data. That also allows to detect and fix bit rot.
Or you could use a filesystem like Btrfs or ZFS that has this built-in so that you don't have to bother with all that bullshit.

>Lastly, bit rot isn't a huge issue for media files and stuff.
Yeah but you can have protection from it just by choosing something better than MD RAID, so why wouldn't you? You're already going to the trouble of buying extra drives to defend against data loss, after all.

Japanese, English, Korean and Chinese languages only.
Doujinshi, Manga, ArtistCG and GameCG, Non-H categories only.
Chaika, Robot rips(Fakku rips) and etc.

Sadpanda alone is 23TB, the other 6TB is Twitter, Pixiv, Fantia/enty/Patreon, and MMD because fucking Japanese love to delete stuff often for no reason.

> Not really, since it doesn't do checksumming.
Of course it can. What are you talking about, RAID5 is a special case of Reed-Solomon [which does not necessarily allow for correcting bit rot because its not necessarily clear what copy got damagfed]. But RAID6 makes it pretty clear which is the defective copy in all cases, so it can correct for bit rot, bad sectors and the like just fine.

>.Or you could use a filesystem like Btrfs or ZFS that has this built-in so that you don't have to bother with all that bullshit.
Or you can NOT use something as bloated and slow as ZFS that does these calculations on every read/write in many cases and gobbles RAM in various configurations and has performance fluctuations on top of that. Oh, and ZFS also scales like shit once you get to 8-10 drives - it will barely perform like 4-5 drives if you add SSD caches and what not.

Instead you just periodically scrub the array or verify your PAR2 data while you're asleep and/or at a rate that doesn't interfere with whatever you're doing. It'll result in the same or better performance on a significantly weaker machine.

That's not to say that with significant hardware, ZFS can't run well enough for home use, but I find it overall pretty silly in terms of what compromises you have to take to use it.

> Yeah but you can have protection from it just by choosing something better than MD RAID, so why wouldn't you?
Just explained that.

Home use is where ZFS gives you the least amount of performance worries. If you're just streaming anime or something, you're doing sequential reads. Preferably you'd like them to saturate gigabit ethernet (~100MB/s) but eh. This is not a demanding performance standard to reach. The massive amounts of RAM, log devices, L2ARC, and all that are all about making spinning discs suck less for heavy random read/random write workloads - which is exactly what home use doesn't do.

I always double up on these to mirror

I had some youtube channel backed up after they went down in 2009. Recently found out I lost the files in a system upgrade and I'm pretty bummed out

A good exchange anons; here's my data hoarding advice: Don't start obsessively screencapping every little mildly interesting thing you see.
>fucking Japanese love to delete stuff often for no reason
This. Adds a certain impetus to grab things quickly after finding them instead of waiting, I guess.

Attached: data_hoarding.png (923x2215, 342K)

OP some of those pictures might have embedded files in them.

Attached: 1507844405469.gif (500x383, 195K)

>Preferably you'd like them to saturate gigabit ethernet (~100MB/s) but eh.
With mdadm RAID5/6 this works on the low end of current onboard x86 - Atom / Pentiums and 1GB RAM or so. And the transfer rate will be quite stable.

The problem is how much more ZFS will require to get there.

> The massive amounts of RAM, log devices, L2ARC, and all that are all about making spinning discs suck less for heavy random read/random write workloads
You need only like $1k in HDD and you need to build a full gaymen-tier machine [without the GPU but with a SSD] to just run RAIDZ2 at ~1/3 raw drive speed with fluctuations.

Of course people can have a lot more drives at home. [No, not everyone does, but cost-wise this isn't even an obstacle to the upper end of poorfags / low end middle class.]

>defrag
blast from the past dude, totally rad. yesterday was windows 95, today is windows 98, tomorrow is windows 101

> Japanese, English, Korean and Chinese
> Doujinshi, Manga, ArtistCG and GameCG, Non-H
That's a pretty good coverage regardless. Didn't expect it to be only 23TB.

> the other 6TB is Twitter, Pixiv, Fantia/enty/Patreon, and MMD because fucking Japanese love to delete stuff often for no reason
Yea, I also noticed that. Don't know why they're doing it.

Is there any trick that you're using there?

The best way I found to get these with some tags and other metadata (but not comments etc. unfortunately) is to download them with Hydrus. I couldn't sufficiently into web spiders to archive the actual web pages in a sensible format.

My sadpanda (including Fakku) folder is around 6TB. Do you hoard EVERYTHING? I'm so confused to how you got it to 29TB.

I only keep Japanese and English, sometimes Chink/Gook if there's no RAW. A lot of duplicates too, because I don't delete RAWs if the retard translation groups downscaled the resolution, which is sadly a common thing for them to do.

>11 terabytes
what the fuck are you hoarding

>Do you hoard EVERYTHING?
Yes.
And see
>Why
Dunno, looking at how mado turned out from the backups of MT and the FTP project at the start inspired me to do it.

What exactly is it that you fagets hoard? how much anime and video gaymes do you really need?

>stuff that has been slowly removed from the internet related to Jow Forums in any way

Bruh, torrent that shit, im sitting on 32TB of mirrored free storage due ded data center scavenging and 1GB FTTH and literally nothing to use it for

Unrelated to actual topic, but there already are "cures" for depression. Ketamine in small doses or psychedelics in microdoses have a lot of anecdotal evidence in their favor. Obviously, neither of them are high-tech.

>am out of sata ports
what do

>How you data hoarders doing today?
I know this is pleb tier but my overall music collection just surpassed 500GB, most of it is mp3s so by number of songs thats quite a bit, and there are over 100 albums left to download.

Also Ive been meaning to stash anime since nyaa and bakabt were kill a year ago. I still havent done it but there are a little over 100 shows that I want to download that ill probably never watch anyway. I want to fill 1TB at least.

Attached: 50ff7cea0422d2e5f293a0faf3f755bc6f76bbef93413bdce679025a9a6f5c12.png (500x500, 103K)

>hoarding
>uses a SHIT FS
kek... fucking shit for brains.

Attached: 1505794326097.jpg (500x375, 18K)

Convert your images to HEIC.

Attached: 1523439167174.gif (235x250, 97K)

PCIe SATA port adapter.

Does Bookmarks and links hoarding count?
>tfw 5K bookmarks and a 1MB txt file with links
>tfw i'll never finish sorting it out

>am out of pci slots
what do

PCIe splitters.

>Inb4 already splitted
Just build a fucking NAS.

>how much anime and video gaymes do you really need?
All of it.

Attached: 1529173988351.png (500x520, 310K)

There used to be a pretty great addon to help you sort/keep your bookmarks sorted. Too bad the Web Extension shit killed it.

addons.mozilla.org/en-US/firefox/addon/bookmark-sorter/

people like you who need to ask this question, will be begging us for content once it is no longer accessible

Ignorant Windows user here. When you say "checksumming" is that automatic or manual? Does the filesystem/software either notify you automatically or correct it automatically?

And regarding bit rot, do you have any evidence that it has fixed or avoided any issues? That is, for those of you running ZFS/ReFS/btrfs, does it actually record instances where something was caught or corrected, or do you just feel safer knowing it's your underlying filesystem?

I originally had an AMD E350 (basically atom equivalent) in my file server and it was fine, but these days I run Windows on it with a bunch of other services (WSUS, web server, qBitTorrent, etc) and that all runs fine on a Sandy Bridge Celeron. Idle power is around 50 W though.

I have a separate machine acting as a TV server with tuner cards though and ideally I'd want them in a single box: Linux bare metal with drives in ZFS, a Windows VM for WSUS, a Linux VM for TvHeadend. I think Ryzen 1700 or something is perfect for this and these days idle power usage is pretty low even on desktop platforms I think.

Only thing holding me back really is the time to move everything over. Would probably need a week, plus I'd likely buy some more HDDs to transfer everything over more easily.

>format external HDD as ext4
>can't access data in wangblows

Attached: 1523756595741.gif (460x345, 492K)

The lack of a decent NTFS / FAT32 defragmenter on Linux is a real pain in the ass.

>Drives
>HDMI out
Pls explain

I horde chinese cartoons. Going to buy a 4TB soon and shift a bunch of it down to the new drive.

Attached: aminejune2018.png (3520x1081, 480K)

Isn't using ZFS without ECC, a big NONO?

ask microsoft to write one if they

AM4 & Ryzen supports ECC.

>When you say "checksumming" is that automatic or manual?
The par2 files that the other user was talking about are manual. You have to run a program to generate them and use them to check the files they have reconstruction information for, and they're regular files like any other ones, not filesystem metadata. ZFS and Btrfs do this internally. You never see the checksums as a file, its part of the on-disk format of the filesystem. Whenever you read a block from the disk, the filesystem also reads the checksum. It recalculates the checksum against that block. if they match - great. if not, the filesystem uses whatever redundancy is there to rebuild the data (or generate the correct checksum, if the data was correct but the checksum wasn't) and then hands that corrected data to the application. There'll be a note of it in the system log and you can look at it with the filesystem's admin tools, but the application itself reading data never knows anything happened.

You can do (and in fact are strongly advised to do) what's called a "scrub", which is just reading everything on the device and doing this checksum verification. If there are any corrupted areas on any disk (that are in use by a file), they'll be found and fixed. And yes, I most certainly have had scrubs turn up corrupted data.

Remember that this depends on redundancy - RAID 1, 5, or 6. ZFS and Btrfs can both use single drives, but then when a checksum mismatch happens, there's no other disk to get the corrected data from. All the filesystem can do in that case is fail the read and inform you that a certain file is damaged.

It's better if you do, but it's not as dire as people say it is. ZFS and no ECC is still a big improvement over no ZFS and no ECC.

Faulty memory can fuck your shit up. However a good RAM kit almost never creates an error. If you run you ram through more then 10 memtest+ tests without even a single error appearing the chance of something happening will be really fucking small. The reason to get ECC is to actually guarantee that it won't create errors. So don't cheap out in an enterprise but at home a normal ram kit could suffice if thoroughly tested.

Umm Sir

I prefer the term "Data Archivist"

former data hoarder here
I started off with 32 TB and now I'm at 900 gb
it feels much better, but I'm wanting to trash my droves and get a single 1 tb ssd and maybe keep a drive for backup
it feels much better having everything in one place

Was it a bad move to get a synology Nas? I don't really have any admin knowledge and just wanted it to werk. Plus with how shitty the cad is it was pretty much the same cost to build my own custom.

Thanks for that explainer. You mentioned that you've had scrubs turn up corrupted data. Do you think that's due to bit rot or something else? My impression is that bit rot is rather rare, so is that corruption caused by other sources?

The creators of ZFS say it's fine to use ZFS on any hardware and RAM.

What do you use to download galleries?

>My impression is that bit rot is rather rare
That impression is wrong. It's not something you see all the time, but over the course of, say, a year, with a multi-terabyte array, it's almost sure to turn up.

Started chopping down my 2k+ browser tabs, shit was slow as fuck today but I had many better days.

Is this code for some kind of pedo thing?

Send me drives.

why is the thread in pic ordered like in reddit

>it feels much better having everything in one place
that place being /dev/null? how do you have everything in one place with just 900GB? why did you stop hoarding?

>Isn't using ZFS without ECC, a big NONO?
It's not any worse than using any other filesystem without ECC. If you don't give a shit about your data then non-ECC memory is fine.
Well, if you do regular scrubs then there could be issues but it's still about as bad for your data as running on any other filesystem without ECC.

Lack of ECC isn't nearly as bad as lack of checksums in the filesystem. Scrub-of-death is a myth. Yeah, you want ECC if you can get it, but if you can't, it's not anywhere near as big a deal as people like you make it out to be.

No, seriously, what is the rationale behind defragging an archive?

>it's not anywhere near as big a deal as people like you make it out to be
I, too, like to write data to the disk without knowing whether or not the intended data is the actual data. If you're writing to the disk and a bit gets flipped (pre-checksum), the checksum will be of the corrupted data.

this
you gotta make torrents to preserve it, I'll gladly setup a seed box just for you

user are you there? How did you get that aesthetic?

I host a booru locally on my rack server that i upload all my 4chins images to, and add tags so i can just search for something like "reaction + mac" and have shitposting material right at my fingertips.

journals are bloat. the ext filesystems wastes up to several terabytes of space for nothing if you dont disable it.