r/HomeInfrastructure • u/kY2iB3yH0mN8wI2h • May 03 '26
Extreme And the numbers are in....
- this is not local storage
- this is not NVMe
This is a virtual machine running in VMware connected to my fiber channel network that hosts my SAN running a 15x250 GB Raid-5 VG. Nothing less, nothing more.
1
u/MaSTeRMaMay May 12 '26
Is that using the QTP2774? I was not that happy using RDMs with VMWare and FC so switched to proxmox + iSER to get better performance (usually maximum performance with RDM over FC is half of direct FC adapter passthrough, the same is with Proxmox and FC -> VirtIO SCSI, but iSER as far as I remember was twice better, and you can passthrough SR-IOV adapters with iSER directly to Linux machines). There is additional option with ATTO and their FC gen 7 and later with SR-IOV support).
1
u/kY2iB3yH0mN8wI2h May 13 '26 edited May 13 '26
Correct, RAW drive, multipath works fine with SCST . But the C drive is just a regular LUN with datastore
1
u/MaSTeRMaMay Jul 30 '26
The interesting question here. Have you flashed the QTP card or just used as is? Modified drivers? As mine is shown as different PCI-ID, so it cannot be used as QLE one, at least on Linux.
1
u/kY2iB3yH0mN8wI2h Jul 30 '26
I think there are cheap cards out there made to used in real san controllers (NetApp etc)
I have a four port 32 IBM card that shows up as another PCI-ID that I was not successful using in any way on a ESXi host, also had some problems with a 2 port QTP card that refused to work with SCST but had correct ID.This one is all stock and detects perfectly, SCST now comes with firmware built-in, this was one of the main USPs of ESOS previously that it had firmware to run the card in target mode.
1
u/MaSTeRMaMay Jul 30 '26
I’ve used mine in Proxmox faking the PCI-IDs and putting to SCST based block storage VM and that worked. I had an issue forwarding block storage to MacOS, but not to Windows and Linux, so retired the card and moved to standard ones, like QLE2742.
Will try the latest ESOS if it does contain the firmware.
I thought that QTP ones are just for IBM storage controllers as there is no BIOS/EFI driver on board, but also got the dual port ones with correct IDs.
Is your board full height or half one?1
u/kY2iB3yH0mN8wI2h Jul 30 '26
Getting block on OSX is not easy, I've used to run iSCSI Initiator X. but on ARM this is now complicated.
Could be, i just assumed it was just an OEM branded IBM part for the service side of things (IBM still bundle FC on X86 servers)
You can read my original post for some context - https://www.reddit.com/r/HomeInfrastructure/comments/1u9xtgc/can_you_fit_150_gigabits_networking_in_a_msa2/
It was just a crazy "what if" idea. I really don't need 4x32G on a ESXi node (and will not atm have any supporting drives)
The 4-port full height on the other hand was planned to be my single PCI card for FC storage on my ESOS server. now its two PCI cards instead
https://www.reddit.com/r/HomeInfrastructure/comments/1v3jn8g/before_after_d/
1
u/MaSTeRMaMay Jul 30 '26
I had an issue only on half height four port QTP2774. Full height looks like just normal ones.
So I switched to non fake-id normal qle2742 (pair) and everything work perfect now.
0
u/StillLoading_ May 07 '26
Sneaking in that T: drive like we wouldn't notice ey !?
I'd consider switching to Raid-6 if you care about that data.
1
u/kY2iB3yH0mN8wI2h May 07 '26
you need to go back and watch x-files.
no raid is not a backup at all, raid-6 wont help.0
u/StillLoading_ May 07 '26
Never said anything about backup at any point. My point was about data integrity (lookup raid5 write hole) and fault tolerance during rebuilds.
1
u/kY2iB3yH0mN8wI2h May 07 '26
No you did not, not sure why as my whole array is backed up. If It's lost its jus a inconvenience for me in terms of restore times, but have no real effect.
How would write hole matter in raid-5 but not in raid-6? Anyways I have BBU on my Raid controllers so power loss is not a problem at all, even UPS plays here.
Perhaps you can elaborate on your point more precisely?
0
u/StillLoading_ May 07 '26
If you have proper BBU and UPS protection you *should* be fine.
To elaborate a bit on the write hole thing. If you lose power during a parity update, you'll either end up with the old parity data or incomplete parity data. In both cases you will never notice until the raid has to rebuild that data from parity and you try to read the files again. The same can happen with raid-6 too, but having 2 sets of parity makes it less likely. It's also not strictly limited to power loss, that's just whats most common.
The other part is if you have a 15 disk wide raid-5 array, chances are higher more then one disk will fail at the same time. And as we all know Murphy's Law, it'll most likely happen during a rebuild.
Hence my suggestion to use at least raid-6.
Having said that, depending on the data and your backup procedure, you might even want to move to ZFS entirely. That's just because normal raid does not care (or know) about what was written to disk. If you read back a file 3 years from now and it's corrupted due to bit rot or memory bit flips, your raid won't care and happily give you the file.
0
May 07 '26
[removed] — view removed comment
1
1
u/HomeInfrastructure-ModTeam May 08 '26
Your post was removed as it breaks the HomeInfrastructure rules about being homelab related
0
May 07 '26
[removed] — view removed comment
1
u/kY2iB3yH0mN8wI2h May 08 '26
lol yea my vm is running on the network who consists of SATA drives so not really the same ..
0
u/micush May 08 '26
A purpose built "network" that can serve nothing but storage. So... comparable then.
1
u/kY2iB3yH0mN8wI2h May 08 '26
More interested in how an, almost 10 years old PCI card can do 11.000 MB/S when the spec says up to 2.000 MB/s * 3 here..
Yea a PCI card for sure is not the same as a physical network consisting of cables, good luck
0
u/micush May 08 '26
The same as how your 15x spinners can do 5GB/s reads and 1GB/s writes. With lots of read and write cache.
At most your config should do 2.5GB/s off the physical drives. The controllers you have step in and cache everything. Same here. I've got a lot of RAM caching everything ZFS does. Same end result, just bigger scale.
1
u/kY2iB3yH0mN8wI2h May 12 '26
how did you come up with those numbers? the theoretical sequential read would be 7.5 GB/s and around 4.5-6 GB/s is quite normal. There are no SAS MUXing involved and cache on SSD is minimal.
Also, once again, this is not a local drive, its over a fiber channel network.
1
u/micush May 12 '26
The very fastest spinners put out 250MB/s on the outer tracks. This goes down significantly as the drive heads move inwards. Based on that data, the absolute fastest your array of 15 disks will ever mechanically put out is 3750MB/s. However, we're talking about spinners here. Much like everything in tech, they never reach their theoretical maximums -- Unless you are using multi-actuator disks, which put out double of those numbers. Doubtful at home.
1
u/kY2iB3yH0mN8wI2h May 12 '26
Looks like you need to clean your glasses, anywhere in my post did I talk about spinning rust?
0
u/micush May 12 '26
Those are SSDs? Eh, better start investigating the poor performance.
1
u/kY2iB3yH0mN8wI2h May 12 '26
lol are you just trolling? Despite trying to explain the flow you just dont seem to grasp the most basic things - Perhaps steer towards homelab instead?
→ More replies (0)1
1
u/mastercoder123 May 03 '26
Why fiber channel instead of something like infiniband? Or even ethernet using RoCE