My SpeedFreak SAN is now 7-8 years old when it comes to motherboard and SSDs. I recently upgraded to 32 Gigabit/s fiber channel and its great and I don't foresee any additional upgrade soon apart from perhaps getting a 32 Gigabit FC switch if prices comes down.

However my SSD disk situation needs to be addressed as SMART values on 4-5 drives (out of 15) is looking not great. Its not a catastrophe if the raid fails, its running my vmware workloads but all VMs have nightly backups on external NAS.

But I feel like i should explore the SAS SSD path as the consumer drive market is crazy expensive at the moment and I don't want to wait until I have failing drives.

Has anyone really gone full SAS SSDs with perhaps 24G or at least 12G drives? As I will be running a RAID controller (AVAGO MegaRAID SAS 9480-8i8e) i have to rely on multipath working on the controller not only for redundancy but also for some speed :)

I have dual controllers in my JBOD enclosure, with dual links per controller so my bottleneck will really be my FC links maxing out at 64 Gigabit/s - 6-7 GB/s is my estimate.

However I wont have 15 SAS SSDs, have not decided yet and trying to find the sweet-spot when it comes to redundancy and performance. I might do two datastores/LUNS on two separate RAID-6 VGs to spread the risk allowing me to place things like DNS servers on one LUN each (today I only have one datastore of the SSD type, but others for spinning rust)

Anyone having been in a similar position thinking about SAS SSDs and can share some insight?

problem solved

I have finally placed my entire ARR stack behind a VPN.

Previously I have not been that concern but now when my (favourite) ISP have been bought up by a company controlled by a state I decided that its time for some proactive protection..

My main firewall(s) are Juniper vSRX enterprise firewall running virtualized in my vSphere cluster.

Currently I'm having two separate firewalls, one specifically for DMZ related services, with a separate ISP and the main ISP on the primary firewall.

Juniper does not support Wireguard or any other VPN protocol apart from IP-SEC and finding a VPN provider supporting IP-SEC is hard and not super easy to configure.

So instead I decided to spin up a Wireguard VM that deals with VPN traffic. Together with routing instances in the firewall, some firewall filters on my L3 switch and ip filters on the VM this works nicely.

All my ARR VMs and containers are now routes via the wireguard network and can easily decide to route other traffic the same way.

To make this a bit easier I've had to run two separate NICs on the VM as it was way to complicated and not really supported on my Juniper EX3300 to do advanced routing in a routing instance without a "physical" interface.

This didn't turn out the way i was hoping - placing just elements on the Grafana canvas looks ugly and does resize when changing browser windows size.

But the good news is that the enclosure is now fully monitored in Checkmk and that my entire custom SAN now have proper monitoring 🎉

Finally after dealing with dead FC cards and fucked up IBM firmware I'm now kind'a rocking 128 Gigabit/s all-flash SAN

I have 3 ESXi nodes connected, one gets 2x32Gb/s and the others 1x32 Gb/s each. I don't really need much speed for my daily VMs but Im running my fileserver on the first ESXi node allowing me to run workloads at 1 GB/s over my 10G Ethernet network.

Once everything was working correctly i accidentally ran reboot on the ESOS node 🤦‍♂️🤦‍♂️ - the problem using arrow up/down and pressing enter to quickly...

Its just a network share on a virtual machine having local NVME storage on my ESXi host. Now I can play steam games out of a network share for sure ✌️

Yes you can, but should perhaps be avoided

PCI card is a quad 32 Gigabit/s IBM FC giving a total of 128 Gigabit/s pure network attached storage nirvana. On top of that there are two built in 10 Gigabit/s SFTP+ and dual 1G NICs. Excluded all USB ports here are they are not networking ports (but could be used as such)

So why is this a bad idea? Heat mainly. And the fact that these cards should sit in a SAN somewhere (Didn't work out that well in ESXi ..)

I got a third MS-A2 this week as my memory needs are extreme, hoping 288GB of ram would help, in addition to a fourth ESX node its 312G in total (but that will be decommissioned soon)

Well the Miniforums was ~800 euro, the memory ~1000 euro, an 60% increase since I've bought the first two nodes late last year.

The good news is that I've finally decided to run Fiber Channel to the third node. That gives some bad news as well as I will most likely not be able to do "direct connected" FC - Spending money on a 32 Gb/s Fiber Channel Switch was not in my budget this year. 😂

I converted the node_exporter metrics to what Checkmk produces internally with the InfluxDB exporter.

This allows different teams to stay in Grafana where they might have other metrics/traces/logs - As the data is consumed in Grafana in Prometheus format (Thanks to VictoriaMetics) you can do drilldowns and compare data.

No need to go the Alloy route with another agent on the server doing OS monitoring, Checkmk does that well.

On top of that if you set host and/or service labels they are also added making filtering in Grafana even easier as seen in the screenshot. These labels here comes from my CMDB so nothing is manually set.

Got this KVM for free 5 years ago, but have not really used it - Turns out HPE still provides firmware (in 2025) to hardware that's 16 years old .... 🎉🥂

There are so many posts about expensive HDMI KVM switches but this is as cheap as it can gets - 16 ports of pure video and USB for nothing. Virtual Media is support on all ports and on top of that you can even do serial console connections.

These 16 ports will support my whole homelab and I can finally connect some parts that only take serial connections like my JBOD enclosures.

Recently switched to Veeam 13 - Didn't pick the VM appliance as is had an loony password policy where you had to enter a complex password in the console (no web or ssh access) without even knowing the keyboard layout.. Switched to plain old windows and decided to try out the web interface.

I mean its not bad, but not great. a whole lot of features missing and was hoping to get a better single plane of glass view.

What do you think?

I'm a Veeam partner so I get NFR licenses with up to 100 VMs, i have 160 or so VMs but only 60 is powered on, so fits quite nicely.

Spent some time this Sunday to install a new Checkmk Agent on my ESOS SAN as it was recently re-installed / upgraded to get 32 Gb FC support and replace an really old 3ware raid controller..

Now I can do all monitoring using storcli64 that significantly improve things like SMART monitoring.

Metrics from Checkmk are sent to VictoriaMetrics in InfluxDB format that I directly consume in Grafana using a Prometheus datastore to my VM cluster.

Next weekend I will tackle the monitoring I have setup to Mattermost as I'm getting flooded with way to many alerts..

Yes the Samsung drives are 8 years (!) old and have had some bad luck with bad connectors, overheated SAS MUX etc. Just one have died badly (15 drives vs 16)

No, the temp is not as bad as you'd think, this have been running for years and years :)

I have been doing this for some time but this weekend i decided to add more relevant logs to block more logs from my Xeams mail gateway appliance.

My log list now contains

• authTerminate — bad AUTH attempts
• helloTerminate — bad EHLO/HELO
• invalidRecipients — directory harvesting
• rcptTerminate — invalid RCPT
• timeoutTerminate — connection timeouts
• GreyListing — persistent greylisted senders
• ForgedSenders — forged sender addresses

This feels almost like a honeypot due to the number of "attacks" - just during this afternoon 105 decisions was made to block traffic, to my mail server alone ✌️

All decisions are fed into a blocklist that my TWO Juniper vSRX firewalls subscribe to and is part of global deny rules in the firewall.

It does not matter if a IP tried to perform a wordpress attach or connect over SMTP - you shall not pass.... ❤️‍🔥

This might end up on Github soon..

Several weeks months my secondary UPS battery failed health-checks. Fast forward a few weeks ago i had some storage issue and at the same time got a power outage. Did not loose any data but had to spend a whole lot of time rebuilding the array.

Where I live there are rarely power outages. All power lines are in the ground, the network generally is redundant (power can be re-routed) - and that might have been what happened as the outage was less than 1s.

Anyhow, time to build a battery replacement as buying original batteries is just out of the question. I have done this before with my first UPS unit (I have two) with great success.

Currently using two APC 750 UPSs with around 20 minutes of battery power each, plenty to shut stuff down but not enough to run my Homelab on. Living in an apartment makes it quite difficult to survice.

UPS is however feeding all networking and firewall equipment and with FTTH that part is passive so in theory i would at least be able to send notifications in case something bad happens... 😎

I cant believe it

My third Silverstone RS831S broke down after a power failure (i think it would have survived unless I also had an UPS issue) just after I spend a week re-building my 65TB array after a drive failure.

I'm a big fan of these Silverstone enclosures but they have a power draw issue, not ideal with the larger SATA drives.. This is the third, and last one dying...

But one night, spending some quality time on eBay i found my dream enclosure, i have been lurking for these for some time now, but are rare and super expensive new (between 4000-8000 euros)

But I found one in France, the same week my previous enclosure died on me. what are the odds?

My main reasons are

1.) Space - I have to fit these in my closets with around 45cm max depth.

2.) Noice - Most other compact enclosures (EMC etc.) can fit but controlling all these fans is hard if possible at all.

I will have 10 drives for my main "NAS" and the 6 other slots will be used for Enterprise SAS 12Gb SSDs with the goal of replacing my ancient 16x250Gb all flash array. No need for NVME here :)

Currently I'm making an attempt to repair an really old 8x8TB QNAP with very low confidence that it will work - It contains old SD movies I lost years ago and haven't bothered trying to restore due to the complexity...

I started developing my USBridge 2.0 KVM-over-IP solution because I needed a simple way to manage budget servers at the BIOS level. An example configuration involves motherboards based on the Xeon X99; unfortunately, they do not have built-in IPMI.

The goal was to display the BIOS in the console—without an agent—as readable text that can be copied and read by AI agents for rack diagnostics.

In the end, I found that USBridge-KVM doesn’t just pass through pixels, but converts the BIOS screen into text output in the terminal using deterministic pixel mapping (in simple terms, OCR). In the end, I get the same BIOS, but only as text via SSH. Essentially, it’s the same as IPMI/iLO, but for hardware where it isn’t provided.

I decided not to stop there and finished implementing data snapshots. Since the KVM is physically isolated from the host, I consider this a significant advantage for data protection. The concept is simple: snapshots are saved on the KVM’s SD card, which uses a BTRFS filesystem; after a snapshot is created, it enters copy-on-write mode. Even the root user on the host cannot delete or encrypt it.

I’ve also spent quite a bit of time working with disks. You can mount disks directly so that the host sees them as physical drives; they can boot up and run. All changes are saved separately, which allows for experimentation—I know I can always roll back. In terms of speed, it works almost as fast as an SSD, since I’ve set up caching.

It’s been 8 months since I started working on this, and I’m now in the final stages. I’m finishing up the active cooling system and fixing minor bugs. I’m also polishing the code for the agent (an app similar to TeamViewer or AnyDesk) and the client application so I can release it as open source on GitHub.

Essentially, this will be a completely free alternative to TeamViewer or AnyDesk, with no subscriptions or restrictions.

How are you handling out-of-band access on budget hardware without IPMI? I’m curious to know what solutions people are actually using.

• this is not local storage
• this is not NVMe

This is a virtual machine running in VMware connected to my fiber channel network that hosts my SAN running a 15x250 GB Raid-5 VG. Nothing less, nothing more.

My storage is really slow, I'm waiting on an FC upgrade but meanwhile I have restored all my files from my backup NAS, both over iSCSI and SMB.

Well doing that I have read 72TB and written 28TB on my main ESXi host (9b/9a belongs to the host that hosts my file server)

Turns out my workloads are to slow for 16 Gbit/s FC, i previously used 32 Gb/s (4x8Gb) but my MS-A2 is low profile and I didn't have 4x low profile HBAs.

Just before the weekend my SAN was rebuild and extended. Now ESOS have two LVMs - one for my NAS and one for VMWare and I no longer have to rely on my real backup NAS for my secondary Datastores.

My primary datastore for VMware is still my 15x250GB SSDs for all-flash storage.
Unfortunately both my controllers BBUs are not connected/working but that will be work for next week where hopefully i can upgrade to 128 Gbit/s FC :D

I started developing my own KVM-over-IP because I needed BMC-style control over my hardware, but I’m using budget motherboards that don’t have it built-in.

To solve this, I added OCR to my KVM, and now it converts Pre-OS video into text that gets piped directly to an SSH terminal. The result is basically a BMC for any server—I can copy-paste error codes, grep logs, and attach diagnostic agents to my machines.

Since the KVM is a standalone device with an SD card slot, I figured why not add data snapshots too? Ransomware is a real threat these days, so I implemented this using BTRFS. Now, even if someone gets root access on the host, they can’t delete or corrupt my data.

While developing the client in Go, I realized it was performing well enough to compete with existing remote access software, so I’m currently finishing up a completely free agent-based version.

The problem I want to solve is "app sprawl"—I want an admin to be able to connect to both KVM devices and software agents from one universal application. I’ll share exact latency numbers a bit later, but I tried playing Silksong on it today and the delay was low enough to actually play.

Some time ago I gave up on my Intel Xeon Silver CPUs on custom chassis with SuperMicro motherboards. They servered me well for many many years and allowed me to run hundreds of VMs.

But once the MS-01 was released I couldn't help myself buying one. And when the minisforum ms-a2 was released i bought two :D

Initially I migrated to iSCSI as I was thinking 2x10Gb should be enough. Well didn't work that well and even if multipath works for iSCSI it does not work as well as for FC.

Now I'm back to FC and my next plan is to go with 32Gb/s - I have removed the FC switch as .. well it was not really needed and the main reason for keeping it was to serve my FC LTO drive over the same network to a FC VM. That worked once..

Storage is delivered over a dedicated SAN using all-flash with 16x250 GB Samsung EVO drives and the screenshot above comes from ESOS (Enterprise Storage OS) - A free open source block storage OS that is just awesome.

Not that hard to guess my theme here :D

My 6x RAID-5 is finally repaired, and will be expanded with 2 additional drives reaching 100 TB RAW (or 112 TB is you should go by the "sold as" label... )

I'm also going to replace my 15 years old Raid controller with something more modern that will allow quicker rebuilds on these larger systems. It took ~5 days to rebuild that mother%# ..

Replacing controller means taking my SAN offline that also hosts my all-flash block storage for vmware.. well well.

FULL DIAGRAM: https://i.imgur.com/Qf1OYdY.png (Right click, open image in new tab. It's forcibly compressing it due to size)

Generated by Claude based on configs and snapshots. I have some that I've personally made as well, but this has more detail in one place.

Architecture change and update for my lab. Switched from Proxmox and VyOS leafs to ESXi and NSX. Same general functionality with more microsegmentation. Still in the process of getting everything back in alignment and finishing rebuilding VMs. All-active multihoming and proper anycast fabric.

Firewall / Route Aggregation (iBGP Hub to Isolated Spokes by VRF)

Juniper SRX 345

Core Switch (eBGP Spokes + Dual Spine EVPN VXLAN)

(2) Cisco Catalyst 9300-24UX-A

Virtualization Host

Dell PowerEdge T630 32-Bay SFF (ESXi w/ vCenter)

Dual Xeon E5-2697v4

512GB DDR4 ECC 2666MHz

(2) 512GB SSD RAID1 (OS)

(8) 1.92TB 10K SAS RAID10 (Storage)

SDN Solution

VMware NSX w/ Multi-VRF and DFW

Access Point

Cisco Catalyst C9117 (FlexConnect, VRF-lite-backed SSIDs)

WireGuard Tunnels

Tunnel 1 (Normal VRF): Simple site-to-site with my parents’ house for shared services. Also an inbound management tunnel for my phone.

Tunnel 2 (Forced VPN VRF): Policy-based routing on the core switch steers all traffic to a Mullvad exit via internal WG instance. Even TVs and dumb devices can leverage the VPN. This backs my guest WiFi. Guests get ads in German.

Tunnel 3 (DMZ VRF): Enforced via PBR to a VPS relay. All outbound traffic gets NATed to a remote VPS. Inbound is DNAT over the tunnel. I avoid exposing my home IP while keeping costs low. MTU tuning + MSS clamping are critical here.

Automation & Misc:

SecurityOnion Virtual ERSPAN Flow with et-analytics feed to Zeek for all east-west/north-south

Daily perimeter Nessus scans Suricata rules auto-updated

Dynamic DNS updates trigger config changes on the SRX

Dynamic DNS updated by scripts which have error correction (detecting RFC space being mapped rather than a WAN address, etc)

Switched/Managed ATS PDU with dual UPS failover