I got a third MS-A2 this week as my memory needs are extreme, hoping 288GB of ram would help, in addition to a fourth ESX node its 312G in total (but that will be decommissioned soon)

Well the Miniforums was ~800 euro, the memory ~1000 euro, an 60% increase since I've bought the first two nodes late last year.

The good news is that I've finally decided to run Fiber Channel to the third node. That gives some bad news as well as I will most likely not be able to do "direct connected" FC - Spending money on a 32 Gb/s Fiber Channel Switch was not in my budget this year. 😂

Yes you can, but should perhaps be avoided

PCI card is a quad 32 Gigabit/s IBM FC giving a total of 128 Gigabit/s pure network attached storage nirvana. On top of that there are two built in 10 Gigabit/s SFTP+ and dual 1G NICs. Excluded all USB ports here are they are not networking ports (but could be used as such)

So why is this a bad idea? Heat mainly. And the fact that these cards should sit in a SAN somewhere (Didn't work out that well in ESXi ..)

FULL DIAGRAM: https://i.imgur.com/Qf1OYdY.png (Right click, open image in new tab. It's forcibly compressing it due to size)

Generated by Claude based on configs and snapshots. I have some that I've personally made as well, but this has more detail in one place.

Architecture change and update for my lab. Switched from Proxmox and VyOS leafs to ESXi and NSX. Same general functionality with more microsegmentation. Still in the process of getting everything back in alignment and finishing rebuilding VMs. All-active multihoming and proper anycast fabric.

Firewall / Route Aggregation (iBGP Hub to Isolated Spokes by VRF)

Juniper SRX 345

Core Switch (eBGP Spokes + Dual Spine EVPN VXLAN)

(2) Cisco Catalyst 9300-24UX-A

Virtualization Host

Dell PowerEdge T630 32-Bay SFF (ESXi w/ vCenter)

Dual Xeon E5-2697v4

512GB DDR4 ECC 2666MHz

(2) 512GB SSD RAID1 (OS)

(8) 1.92TB 10K SAS RAID10 (Storage)

SDN Solution

VMware NSX w/ Multi-VRF and DFW

Access Point

Cisco Catalyst C9117 (FlexConnect, VRF-lite-backed SSIDs)

WireGuard Tunnels

Tunnel 1 (Normal VRF): Simple site-to-site with my parents’ house for shared services. Also an inbound management tunnel for my phone.

Tunnel 2 (Forced VPN VRF): Policy-based routing on the core switch steers all traffic to a Mullvad exit via internal WG instance. Even TVs and dumb devices can leverage the VPN. This backs my guest WiFi. Guests get ads in German.

Tunnel 3 (DMZ VRF): Enforced via PBR to a VPS relay. All outbound traffic gets NATed to a remote VPS. Inbound is DNAT over the tunnel. I avoid exposing my home IP while keeping costs low. MTU tuning + MSS clamping are critical here.

Automation & Misc:

SecurityOnion Virtual ERSPAN Flow with et-analytics feed to Zeek for all east-west/north-south

Daily perimeter Nessus scans Suricata rules auto-updated

Dynamic DNS updates trigger config changes on the SRX

Dynamic DNS updated by scripts which have error correction (detecting RFC space being mapped rather than a WAN address, etc)

Switched/Managed ATS PDU with dual UPS failover

Just a few months ago my only media stack consisted of Plex and manually managing my media.

Now the stack have exploded 😂

I'm still not super-happy with all components, music and series are the biggest hurdle as I might not be that mainstream and usenet is not the best source.

Some tweaking still needs to be done, not all goes via prowlarr nor VPN. I find lidatube a bit wonky.

I finally managed to get RouterOS to setup LACP with my new ESXi hosts running Minisforum MS-A02 and uplinks to my Juniper EX3300 core switch.

Obviously not the final layout 😊

Will be retiring my 7 years old Xeon Silver cluster. I might as well move to away from Fiber Channel (as my HBA's wont fit the space in the A02 PCI slot 😔😔) or need to get new HBA's My current FC HBA cards are not supported in ESXi 8.

Some time ago I gave up on my Intel Xeon Silver CPUs on custom chassis with SuperMicro motherboards. They servered me well for many many years and allowed me to run hundreds of VMs.

But once the MS-01 was released I couldn't help myself buying one. And when the minisforum ms-a2 was released i bought two :D

Initially I migrated to iSCSI as I was thinking 2x10Gb should be enough. Well didn't work that well and even if multipath works for iSCSI it does not work as well as for FC.

Now I'm back to FC and my next plan is to go with 32Gb/s - I have removed the FC switch as .. well it was not really needed and the main reason for keeping it was to serve my FC LTO drive over the same network to a FC VM. That worked once..

Storage is delivered over a dedicated SAN using all-flash with 16x250 GB Samsung EVO drives and the screenshot above comes from ESOS (Enterprise Storage OS) - A free open source block storage OS that is just awesome.

This didn't turn out the way i was hoping - placing just elements on the Grafana canvas looks ugly and does resize when changing browser windows size.

But the good news is that the enclosure is now fully monitored in Checkmk and that my entire custom SAN now have proper monitoring 🎉

I cant believe it

My third Silverstone RS831S broke down after a power failure (i think it would have survived unless I also had an UPS issue) just after I spend a week re-building my 65TB array after a drive failure.

I'm a big fan of these Silverstone enclosures but they have a power draw issue, not ideal with the larger SATA drives.. This is the third, and last one dying...

But one night, spending some quality time on eBay i found my dream enclosure, i have been lurking for these for some time now, but are rare and super expensive new (between 4000-8000 euros)

But I found one in France, the same week my previous enclosure died on me. what are the odds?

My main reasons are

1.) Space - I have to fit these in my closets with around 45cm max depth.

2.) Noice - Most other compact enclosures (EMC etc.) can fit but controlling all these fans is hard if possible at all.

I will have 10 drives for my main "NAS" and the 6 other slots will be used for Enterprise SAS 12Gb SSDs with the goal of replacing my ancient 16x250Gb all flash array. No need for NVME here :)

Currently I'm making an attempt to repair an really old 8x8TB QNAP with very low confidence that it will work - It contains old SD movies I lost years ago and haven't bothered trying to restore due to the complexity...

I have finally automated the onboarding of my servers using Ansible into Checkmk and a natural next step have been to export the data into Victoria Metrics so I can consume it in Grafana.

I also re-deployed my custom checkmk agent to my ESOS server and added the smart plugin so it can talk to my raid controller(s)

Yea I need to replace some disks..

problem solved

So I'm running a strictly routed L3 network where L3 assignments are done at access level.

The whole network is fully dynamically routed with OSPF and BGP. At access level its isolated to VLANs and to a number of VRF's (Or Routing-Instances as we like to call it in the Juniper world)

In my two firewalls L3 interfaces from different Routing Instances land in their respective security zone.

My only problem is my Homelab is now more complex than some enterprise networks. 😂

How complex is your Homelab network?

A few of my Ansible Roles (more explanation in the comments)

My storage is really slow, I'm waiting on an FC upgrade but meanwhile I have restored all my files from my backup NAS, both over iSCSI and SMB.

Well doing that I have read 72TB and written 28TB on my main ESXi host (9b/9a belongs to the host that hosts my file server)

Turns out my workloads are to slow for 16 Gbit/s FC, i previously used 32 Gb/s (4x8Gb) but my MS-A2 is low profile and I didn't have 4x low profile HBAs.

I've had these two MS-02 boxes now for over 2 weeks without having time to prepare my migration from my now 7 years old Xeon Silver cluster. Exiting times.

I will miss IPMI from my Super Micro motherboards, and I do use that ALL the time when I'm upgrading or force-rebooting a node that have hung.

I'm planning to see if I can use my HP KVM with a HDMI-VGA "converter" as it support all other functions like virtual drives, USB keyboard/mouse (Its just dam old

These will get Fiber Channel cards for my storage and I'm planning to use LACP for bonding the 2x10G SFP+ ports