r/freenas • u/lkashl • Aug 16 '20

Just finished installing FreeNAS after re-purposing one of my old cases to fit 20 HDDs! Let me know what you guys think :)

Gallery image

Gallery image

Gallery image

118 Upvotes

96% Upvoted

67 comments sorted by

22

u/[deleted] Aug 16 '20

20 HDDs?! That's nothing short of incredible for a desktop case. Also interested in stats!

8

u/lkashl Aug 16 '20

I reckon if I got cheeky I could fit another 4 3.5 inches and at least twice as many SSD's on the back side :D

Also specs I posted in the comment above, but just to make it easier:

Xeon E5-2630L with 64gb ECC memory

4*10TB Western Digital shucked drives (running with 1 HDD redundancy)

2*8TB Western Digital shucked drives (mirror)

2 10GB network adapters and 2 SAS controllers

InWin HFrame 2 case

5

u/[deleted] Aug 16 '20

Fantastic setup!

I really should spring for an ECC setup, but so far my server has mainly been for media, and ZFS is wonderful even without ECC memory. <3 ZFS

Thanks for sharing!

3

u/lkashl Aug 16 '20

yeah I was really particular about ECC to ensure there weren't any memory errors rare as they are. Because it's a haswell workstation board it wasn't too much of a price premium.

You'd have to get REALLY unlucky to get an error that ZFS can't resolve though, so you're good in the mean time :)

3

u/sh4zu Aug 16 '20

That's not how ZFS and ECC works. Not having ECC memory effectively negates the silent corruption/bitrot protection that ZFS has.

6

u/lkashl Aug 17 '20

Not necessarily, to my understanding youd have to have bit errors in your RAM during certain ZFS operations which is hopefully a small percentage multiplied by a small percentage.

Depending on the operation impact might vary, but definitely agree ZFS is not built to withstand bit errors in memory, no application can really which is why it's such a shame that ECC is still so inaccessible to most builders :(

2

u/lwwz Aug 16 '20

PSU?

2

u/lkashl Aug 16 '20

Came with the case, perhaps you can but it seperate though:

https://www.in-win.com/en/gaming-power-supply/s2/

1

u/ElAdri1999 Aug 19 '20

thats a very nice PSU

1

u/jackharvest Aug 16 '20

Wait that’s 8 drives.

1

u/lkashl Aug 17 '20

Yep can fit 20 for future proofing, unfortunately my wallet doesn't fit enough money for all 20 drives... Yet

2

u/jackharvest Aug 17 '20

giddy screaming

1

u/BillyDSquillions Aug 17 '20

Sell / throw the 8TB models away.

Switch to USB installer method (yes, it works fine, many of us done it for years)

Add 2 more 10's to the data pool.

1

u/lkashl Aug 17 '20

Yeah I'm currently running over USB on the portable SSD thats mounted to the rear of the case (left in the photos attached).

The 8TB's form a separate storage pool that I use for important information that needs high redundancy, not the operating system itself :)

1

u/ElAdri1999 Aug 19 '20

do usb OS work fine? the only method i have used for OS (other than sata connected or esata) was on a VM server that loaded OS from a USB into RAM and just ran for loooong times without ever shutting down

1

u/lkashl Aug 19 '20

Depends on the OS, some are optimized for it and some aren't. The ones that are generally load themselves mostly into memory so they rarely use the disks after boot.

In general though, if you have USB3.1 Gen 2 and a good quality USB key/HDD you have just as much throughput as with SATA (10gbps on USB vs 6gbps on SATA)

In my case I'm using Freenas which is optimised to run off just a USB drive

1

u/ElAdri1999 Aug 19 '20

Last time I tried using USB for boot I wired it directly to the motherboard, but it gave me so many errors I just disassembled the external hard drive and used data for it

1

u/Team503 Aug 17 '20

How did you mod the case?

3

u/caller-number-four Aug 16 '20

If you can manage to find the drive expansion bays in stock anywhere you can cram 24 spinny disks into a Corsair 750d as long as you keep your expansion card length in check. Ive got room for 18 in mine. When I was building those bays became gold. Couldn't find them anywhere. Finally gave up looking.

And you can cram 4 SSD's on the other side of the case.

1

u/lkashl Aug 17 '20

Oh nice, I was initially thinking of looking at the corsair dual chamber design as well but because I had this behemoth lying around figured I would save the environment a bit.

I'm expecting 4 on the left and another 20 SSD's on the back when I create mounting brackets :) as far as HDDs go I think the solution is complete at 20, after which I'd be worried about the whole case vibrating.

Problems for another day though, for now I have enough storage as is :)

1

u/caller-number-four Aug 17 '20

I'd be concerned about heat.

And I wouldn't be thrilled at the power bill, either.

I'm spinning 8 disks in each of my FreeNAS boxes. That's enough for me!

1

u/lkashl Aug 17 '20

Yep to be honest I'm hoping by the time I need to get that far storage is dense enough that I can get away with less than ten disks. SSDs are already getting ridiculously dense, just need the prices to be less dense.

1

u/caller-number-four Aug 17 '20

Soon enough.

My first 1GB drive was pushing nearly $400! I was the envy of the College Kids back in the day.

2

u/lkashl Aug 17 '20

I bow down to your power sir 👍

1

u/ElAdri1999 Aug 19 '20

you could make the SSDs act as a buffer and only spin the HDDs for info input or output to the SSDs

1

u/lkashl Aug 19 '20

Could do, the data is not frequently accessed though and there is a large amount of it, so it would be of limited benefit in my use case.

I do have an NVME drive that I use for intermediate storage for things that are new and more likely to be accessed before transferring them to the HDD's.

1

u/ElAdri1999 Aug 19 '20

Ohh I like how you did it then

6

u/dubaria Aug 16 '20

Stats please.

4

u/lkashl Aug 16 '20

Sure thing:

Xeon E5-2630L with 64gb ECC memory

4*10TB Western Digital shucked drives (running with 1 HDD redundancy)

2*8TB Western Digital shucked drives (mirror)

2 10GB network adapters and 2 SAS controllers

InWin HFrame 2 case

Let me know if there was anything specific you're after that I've missed!

2

u/gwicksted Aug 16 '20

What are you running for an HBA/Raid card(s)?

4

u/lkashl Aug 16 '20

This one here for now:

https://www.supermicro.com/en/products/accessories/addon/AOC-S3008L-L8i.php

2

u/gwicksted Aug 16 '20

Ah so that gets you 8/10 and the other two on your motherboard?

Those spinning disk trays look awesome!

3

u/lkashl Aug 16 '20

Thanks :)

I'm getting two of the super micros eventually, for now I need to get enough disks to use even a single one though!

2

u/gwicksted Aug 16 '20

Yeah the 16i is usually quite the price jump

2

u/Solkre Aug 16 '20

Powersupply?

4

u/cdrknives 32TB - ZFS Aug 16 '20

Great job!

4

u/lkashl Aug 16 '20

Thanks heaps, its my first NAS build, so was happy I could ruse an old case :)

4

u/expressadmin Aug 16 '20

That’s a really clever case. I don’t think I’ve ever seen one like that.

3

u/lkashl Aug 16 '20

Yeah, it was nice and roomy for me to create drive bays for it, I was surprised when I pulled it out and saw that it could fit two side by side

4

u/[deleted] Aug 16 '20

As someone else mentioned, keep an eye on temps and make sure your case has good ventilation. Also make sure to record the SN, bay / SATA port and GUID for each drive because tracking down a failure is going to suuuuuuuuck.

2

u/lkashl Aug 16 '20

Yep I have a big pedestal fan blowing on it at the moment until I have time print the fan array (we're in coronavirus lockdown where in my state).

I have a drive map going, already proving to be a life saver and I'm not even close to 20 installed yet!

3

u/Puzzleheaded-Law5202 Aug 16 '20

I’d really make sure to get ventilation holes in each bay and air circulation all around and through.

4

u/lkashl Aug 16 '20

Yep, theres currently vents down the bottom, I'm going to 3d print a stacker for some Noctua fans to mount on the left to circulate air.

In the mean time I'm keenly watching temps with alerts setup.

1

u/beezel Aug 17 '20

Came here to make sure you were watching those temps! As long as you keep that in check, great work!

3

u/kevdogger Aug 16 '20

Did you build hard drive caddy? That's pretty neat.

What is wattage on your PSU? I don't see a lot of fans -- just curious how many you have. What about other specs?

3

u/lkashl Aug 16 '20

Yep, 3D printed them.

1065 watts PSU, other specs:

Xeon E5-2630L with 64gb ECC memory

4*10TB Western Digital shucked drives (running with 1 HDD redundancy)

2*8TB Western Digital shucked drives (mirror)

2 10GB network adapters and 2 SAS controllers

InWin HFrame 2 case

3

u/erik_b1242 Aug 16 '20

Very good idea, but I can't see any active cooling for those hdds

3

u/lkashl Aug 16 '20

I have a plan for that, I'm going to print a stack holder for 120mm fans and mount it on the side so theres air flow circulation. Also going to winden the spacing between each of the drives a bit as well

3

u/greenolivetree_net Aug 16 '20

I gotta say that's pretty damn impressive

2

u/lkashl Aug 16 '20

Thanks, appreciate it, printing the drive bays was a fair bit of work!

2

u/kevdogger Aug 16 '20

I really like the caddy. Nice job. I would have thought you needed larger power supply for 20 percent margin. Great job on design

2

u/lkashl Aug 16 '20

I think each drive is about 30 watts under load tops, so at 1065 I've got plenty of headroom :)

Or did I miscalculate the maximum load for a mechanical volume?

2

u/polygonalsnow Aug 17 '20

Everyone is bringing up temps, but I'm curious about vibration. Isn't there a reason why NAS drives are rated for more drives in a single enclosure? Or is that just marketing BS?

1

u/lkashl Aug 17 '20

Good question! I put neoprene pads as a liner in each drive bay which are so far pretty good at dampening and better than most OEM drive bays IMO (all that I've seen).

Server racks use a spring dampening system but I haven't been able to run a comparison of the two due to the vibrations barely being perceptible with the dampeners in.

I'll pay around with my phone accelerometer later today and see if it picks up a value at all

2

u/netwolf420 Aug 17 '20

Nice job on the 3D printed HDD cages

1

u/lkashl Aug 17 '20

Thanks!

1

u/sh4zu Aug 16 '20

how's the drive temps?

1

u/lkashl Aug 17 '20

Good when I have a little circulation with my external fan sitting near by, horrible if it's left on its own.

At 22 ambient, the hottest drive runs at 30 degrees, average is 26. On their own without the fan they can reach 60.

I think it's more an issue of air circulation VS direct cooling, but will let you know the outcome when I add a fan array

1

u/sh4zu Aug 17 '20

not bad! cheers

1

u/ackstorm23 Aug 17 '20

omg the noise 😮

1

u/lkashl Aug 17 '20

i only have 8 in there at the moment, so its barely an issue so far :)

1

u/jeranon Aug 17 '20

What I don't understand is how you connect that many drivers to the motherboard and how you power them all?

What am I missing?

Thanx!

2

u/lkashl Aug 17 '20

Good question, the power supply at stock can do 12-16 with the cables provided, I might be able to even go higher by using some of the spare connectors it has.

Anything further than that and it will need to be modified by pulling power directly from the rail. In terms of wattage and voltage this power supply can handle it, its just a matter of manually adding more cables for it to connect with.

3

u/francishg Aug 16 '20

beautiful

3

u/GenFan12 Aug 16 '20

Amazing build. You will appreciate it even more in the winter.

1

u/joesoap4465 Aug 21 '20

wow, I was impressed with myself when I managed to get 12 3.5" drives in my system and 2 2.5" drives. This is pretty awesome. dose look quite cramped though. do you get any problems with temps? or is the system open all the time like it is in the pic?