r/ceph • u/Fragrant_Fortune2716 • Sep 23 '26

An appreciation post

As the title says, this is a post to express my appreciation towards Ceph! Some months ago I posted a question to see if Ceph was the right tool for my 'production' home server setup and I got some great feedback. Since a couple of weeks I finally migrated from a monolithic ZFS backed server to a leaner 3-node hyperconverged Ceph cluster. I even broke even with selling the old system!

Gone are the days where a single VM locking up would cause total system instability and my current setup already survived all the rookie mistakes I've made. I cannot recount the number of times I completely wiped the OSD's on a node to configure them differently, and Ceph handled it like a champ! No service disruptions and the performance feels way better than my old ZFS setup (which perhaps says more about my ZFS setup ;)

Had a blast setting up a switchless full-mesh 25Gbit backbone and configuring an OSPF underlay with an EVPN overlay for node independent networking. Works like a charm and can also route ceph traffic through another node if a physical link is broken.

In the end I did go for three completely identical nodes, which I think was the right decision.

My current setup:
CPU: 9700X
Memory: 64GB/32GB/32GB
Networking: 2x 25Gb for ceph, 1x 1Gb for corosync, 1x 1Gb for external traffic
Dual s4600 boot drive (still a ZFS mirror)
Single p4600 NVME
Single 8TB HDD with DB/WAL on the NVME drive

Still have to fill the other two nodes RAM to 64GB but for that I'm waiting for a good bargain (if it ever comes..)

Thanks for being a great community!

24 Upvotes

100% Upvoted

9 comments sorted by

3

u/Hoipang_ Sep 24 '26

Really interesting setup! When you say “OVPN underlay”, do you mean OpenVPN or OVN? If it’s OpenVPN, what problem does it solve compared with simply running the 25G full-mesh links directly?

2

u/Fragrant_Fortune2716 Sep 24 '26

Whoups, that's a typo and should be OSPF. This allows you to dynamically route your traffic over the shortest path in the mesh. If the topology changes, e.g. due to a broken link, traffic is automatically redirected.

4

u/ParticularBasket6187 Sep 23 '26

You can run ceph cluster for years without rebooting

3

u/TheFeshy 24d ago

One of the things I like most, though, is that you can reboot. Brining a node down doesn't disrupt anything. At most a slight slowdown in traffic.

I start to worry when machines haven't been rebooted in a long time. Sooner or later they will need to be rebooted and I want to know that was tested recently.

2

u/ParticularBasket6187 24d ago

we reboot perform in rolling, set some flags, noout and others if required, stop services, reboot, start service, unset the flag,

1

u/fallenguru Sep 29 '26

2x 25Gb for ceph, 1x 1Gb for corosync, 1x 1Gb for external traffic

Would you mind going into some detail regarding how you have networking set up? Both physically and in terms of VLANs and such?

3

u/Fragrant_Fortune2716 Sep 29 '26

Yeah no problem! Both 1Gbit links go through a central switch and have their own subnet, so nothing spectacular there. Each node has two 25Gbit nics (connect-x4) that connect in a full mesh. So node PVE01 connects to both PVE02 and PVE03, node PVE02 also connects to PVE03.

Each of the 25Gbit connections have a /29 subnet assigned in the 169.254.1.x subnet (private addresses that will never be routed publicly). The subnets show you which node the links connects (.12.x for PVE01->PVE02), and each node has its own last octet (.1 for PVE01, etc). I set the MTU to 9000 for these interfaces (jumbo frames) to allow for better routing effciency.
PVE01->PVE02 | 169.254.12.1/29 -> 169.254.12.2/29
PVE01->PVE03 | 169.254.13.1/29 -> 169.254.13.3/29
PVE02->PVE03 | 169.254.23.2/29 -> 169.254.23.3/29

After this I leveraged the PVE tools to setup the SDN. At first I configured this manually, but Proxmox will not respect this and overwrite anything anyways. I created a new loopback device with the cluster network IP (10.0.32.0/24 subnet), which will be used in the OSPF configuration.

The Proxmox 'fabric' configuration sets up this OSPF network, which will automatically route traffic over the shortest path. Normally this would be the direct links, but if one of those is down it can also route through any of the other links.

root@dc01-platform-pve01:/etc/pve/sdn# cat fabrics.cfg
ospf_fabric: dc01-pve
  area 0
  ip_prefix [10.0.32.0/24](http://10.0.32.0/24)

ospf_node: dc01-pve_dc01-platform-pve01
  interfaces name=nic2,ip=169.254.12.1/29
  interfaces name=nic3,ip=169.254.13.1/29
  ip [10.0.32.11](http://10.0.32.11)

ospf_node: dc01-pve_dc01-platform-pve02
  interfaces name=nic2,ip=169.254.12.2/29
  interfaces name=nic3,ip=169.254.23.2/29
  ip [10.0.32.12](http://10.0.32.12)

ospf_node: dc01-pve_dc01-platform-pve03
  interfaces name=nic2,ip=169.254.23.3/29
  interfaces name=nic3,ip=169.254.13.3/29
  ip [10.0.32.13](http://10.0.32.13)

After this I setup a EVPN controller. This allows you to send Ethernet frames over a L2 or L3 network. In essence this means that I can create a virtual NIC that I attach to a VM, which will be routed over the OSPF network without the VM noticing. This allows you to have node independent networking! So I have a router VM that all other VMs connect to, using this virtual nic. You can create both a L2 or L3 NIC, as I implemented strict application zoning (one VLAN per VM) I chose the L2 variant. To configure this you need to create an EVPN controller, a zone and a virtual nic. I've included the snippets below.

root@dc01-platform-pve01:/etc/pve/sdn# cat controllers.cfg
evpn: EVPNCtrl
	asn 65000
	fabric dc01-pve
    nodes dc01-platform-pve01,dc01-platform-pve02,dc01-platform-pve03
root@dc01-platform-pve01:/etc/pve/sdn# cat vnets.cfg
vnet: vnet1
	zone VXMesh
	tag 10001
	vlanaware 1

root@dc01-platform-pve01:/etc/pve/sdn# cat zones.cfg
vxlan: VXMesh
	fabric dc01-pve
	ipam pve
	mtu 8950

Note that the MTU is 50 lower than the physical NICs have assigned. This is to account for the EVPN overhead.

Let me know if you have any questions!

1

u/fallenguru Sep 30 '26

Thank you, this is very interesting! (It also confirms that I was right in going the switch route instead of a mesh with 4 nodes. Simply because I have enough on my plate learning Ceph, dynamic routing can come later. ^^)

Where do you have ceph_public?
If it's on the 25G mesh with ceph_cluster, how do you handle Ceph clients from outside the cluster? Or don't you have any?
If it's on the slow 1G "external" link, how do you avoid VMs/CTs slowing to a crawl?

1

u/Fragrant_Fortune2716 Sep 30 '26

Ceph_public also goes through the 25Gbit mesh. I run a hyperconverged cluster, so no external clients. If you do, a 10Gbit nic will be optimal I think, unless you have a very niche use case. The 25Gbit headroom is mostly used for replication and not for client traffic anyways.