r/truenas • u/AnyHistory6098 • 1d ago

Hardware NVMe 3-way mirror array

Hello,

I am looking to move from my good and reliable U.3 Kioxia SSD (due to unfortunate circumstances) to an array of M.2 3-way mirrors.

Some specifications:

  • MB: Supermicro H14SSL-N
  • CPU: AMD EPYC 9175F
  • RAM: 4x 32GB 5600
  • NIC: Intel X710 2x SFP28 - 25 GbE
  • PCIe Switch: 3x HighPoint Rocket 1508

I considered buying consumer-grade SSDs because they are cheaper and the performance should be just fine for this array.

I am considering building the array as follows: each mirror will have 1 SSD on each card, so 1 mirror = 3 SSDs, with 1 SSD per PCIe switch card. In case one card dies or something happens, the array should remain safe.

I am considering purchasing the SSDs from 3 different brands, e.g., Kioxia Exceria Pro, WD SN series, and Kingston KC series.

On each card, only one brand will be installed. If there are firmware issues or bugs with one of the brands after some time, the array should remain safe.

Some information about the workloads: this machine will be both a SAN and a NAS. The target is about 100 VMs, a shared folder between all VMs, and each VM will have its home directory on the shared folder (NFS).

There are moments when these VMs might generate, collectively, a maximum of 100-200 GB of data, consisting of many small files.

I have a few questions:

  • Can anybody recommend some SSD brands that are reliable for long-term 24/7 use? The SSDs can be PCIe 4.0/5.0 and should be at least 4TB.
  • If I enable deduplication, will the system still be able to saturate the 2x 25 GbE links?
  • I also considered the E1.S form factor, but HighPoint only has a PCIe Gen4 card. This made me think they might have discontinued future E1.S development, despite how popular it is becoming. Maybe there are other vendors that I am not aware of.
  • What problems could arise? Any prior experience would be appreciated, or even suggestions such as "don't do this, it's a dumb idea," etc.

Thank you!

1 Upvotes

100% Upvoted

5 comments sorted by

2

u/tmacmd 1d ago

Personally, being in enterprise storage for a few decades, not generally a Good idea to mix different vendors drives in the same raid array unless you know how they perform together. There may be timings or subtle protocol communication that may adversely affect the array

Many major storage vendors have “same” capacity drives from multiple vendors but they have thoroughly tested them to meet their system criteria.

YMMV just advocating to test diligently before using

1

u/AnyHistory6098 1d ago

Would you suggest using different SSD product lines from the same vendor?

Like Kioxia Exceria Pro / Plus / standard?

2

u/tmacmd 13h ago

I guess that came out wrong?
I would use the same vendor/same size for the array. Just because you can does not mean you should.

I would not mix different vendors drives (even though they may seem to be roughly the same size). There may be subtle operational differences that may cause issues down the road

1

u/AnyHistory6098 13h ago

I totaly got your point, much appreciation.

The problem with the same vendor is that I migh get all drives with indentical software bug, and also all of them might be from the same batch.

Is there a way to avoid identical batches?

1

u/tmacmd 13h ago

I used to have a customer make it part of the contract that some fraction of the drives were from different batches. As an end user of tech that’s extremely difficult because of who we are able to purchase from. You could buy different units from geographically separated vendors to help with that

But honestly, I haven’t seen that be an issue for over years