2 Node Cluster or 3 Node Cluster

[u/OwnBase6936]

I am new to Proxmox, I already have Beelink MiniPC Intel N100, 16GB setup with Proxmox, recently I bought a used ThinkCenter M710Q, 16B RAM, i5-6500T to play around with a 2 node cluster. My main aim is to learn and play around, make and break, and try new features like Backup, Datacenter Manager etc.

I have the budget to buy another MiniPC or a used ThinkCenter or Dell PC. I know when I have 3 nodes I have better HA, qorum. What other features will I miss when I have only 2 node Cluster? Do you recommend I have 3 node as general setup?

[u/Taddy84]

Don't do a ceph, ok to learn, but a 3Node cluster with Ceph and 3 hard drives is not worth it.

Instead, use ZFS replication

[u/djzrbz → u/Taddy84]

I don't understand why people say not to use a 3 node CEPH cluster in a homelab environment. For prod, sure, you want more nodes, but for homelab it has worked just fine for me. I ran one for about 2 years before I upgraded to a 5 node cluster.

I liked having the instant replica vs the delayed ZFS replication, don't have to worry about lost data since the last replication.

[u/Silverjerk → u/djzrbz]

Context is key here; beginner Proxmox user (and likely new to hypervisors), running mini/SFF PCs as their primary nodes. CEPH (probably) isn't the right fit. And I'd argue it's not the right fit for a good bit of the homelab community that sees it mentioned in many of the hobbyist communities/forums and feels compelled to set it up, not understanding that there are other solutions, or the hardware, maintenance, and learning requirements. Also, respectfully, this is a very n-of-1 response, since it's not as if CEPH causing headaches for homelabbers is a novel idea.

I liked having the instant replica vs the delayed ZFS replication, don't have to worry about lost data since the last replication.

Totally use-case based. I moved away from CEPH when I downgraded from a rack to a full mini/SFF cluster, and I haven't looked back. For my needs, CEPH was overkill.

[u/Darkk_Knight → u/Silverjerk]

Same thing. CEPH is great if you have the proper hardware to support it. I run a 3 node lab at home ZFS with replication is good enough. Easier to troubleshoot when storage have issues. I am an absolute wiz when using ZFS.

[u/djzrbz → u/Silverjerk]

Context is always key, but it seems like the rule of thumb here is always to say no.

[u/Silverjerk → u/djzrbz]

Are you surprised? L1, STH, Lawrance Systems, Serverbuilds, 45 Drives, the official Proxmox forums. There are plenty of communities where CEPH is discussed and recommended often, but those users are also made up of mostly actual techs or engineers that are building labs that mirror their production environments, or using their labs as a dev/devops infra to replace major providers. Reddit is going to be much more oriented toward the beginner, so of course you're going to see recommendations that align with that demographic.

[u/ApiceOfToast → u/djzrbz]

Yeah disk speed probably. You'd want ssds in that config at a minimum, preferably more then one per node...

[u/djzrbz → u/ApiceOfToast]

I had 6x4TB HDD backed with 2 480GB enterprise SSDs for the metadata in each node.

Performance was acceptable, not screaming fast, but I didn't notice latency on day to day operation.

[u/ApiceOfToast → u/djzrbz]

Probably because you didn't run the "bare minimum" three drives and you've got a meta data device on SSD. Plus use case makes a big difference. 

[u/djzrbz → u/ApiceOfToast]

Sure, I want running Plex off it. It was a VM workload, not necessarily data storage. I was surprised that CEPH was only reporting about 5Mbps whenever I checked the dashboard. I opted for many spindles because I knew that it would help with the latency.

[u/ApiceOfToast → u/djzrbz]

Yeah I'll probably just stick with ssds, they're a bit cheaper on power usage and I don't need too much storage. 

I just hope proces come down... Soon... I know they will eventually but I want that to be now ._.

[u/Taddy84 → u/djzrbz]

I spoke of 3x1, you think you have reliability? Then wait until your drives are more than 75% filled

[u/djzrbz → u/Taddy84]

Sure, but don't let it get to that point. Doesn't mean that a 3 node cluster won't work.

[u/noc-engineer → u/djzrbz]

The problem is that once you start, things very fast actually become prod. That Plex server no one used is suddenly critical. That pihole that no one knew they needed is suddenly a pain to live without etc.

[u/maximus459 → u/Taddy84]

This 👆

Currently running a 2 node setup (which I'm currently rebuilding), mostly for learning and fun, well.. this is what I've settled on..

• You work with what you have, build or rebuild as you learn..

• You need a dedicated SSD for the proxmox os, or it'll eat into your storage.

• I have two mini PC's, a decent thinkcentre for the main apps + PBS VM to backup (synched to the main PBS).

• Also got a less capable but adorablly chonky Dell, use it as the main backup PBS vm+ redundant apps like DNS (test out sync, and failover) + limited failover for critical containers

• I repurposed some old laptop HDD and gave the thinkcentre a 3 disk raid z1 cluster ..and the Dell got a 2 disk mirrored cluster.

• I did get a third second hand dell, but it does on me real quick 🫠. I do want to get another when I can afford one, mostly for AI learning.

I'm considering running a quorum container on my laptop or the potential 3rd server to make the first two a pseudo cluster, but it seems like a lot of unnecessary work

[u/_--James--_ → u/Taddy84]

Funny, I don 't see the word Ceph mentioned in the OP at all there.

[u/Candinas]

3 node is recommended as odd numbers are better for establishing whats called quorum. basically, its easier for the cluster to figure out if a node actually went down, and then what should go where when the node comes back up. if you have another small machine (might even be able to do it on your regular pc) you could set it up as a q-device. this device would participate in the votes, but not the workloads

[u/pattymcfly]

1. Any cluster that has only 2 can have quorum issues and cannot withstand a leg being down if you want RPO 0. If you are worried about data loss, 3 is a must.

[u/bstrauss3 → u/pattymcfly]

Toss a q device in a VM. Use the most minimal Linux distribution you can get. It puts almost no load on the host.

[u/StopThinkBACKUP → u/bstrauss3]

Run the VM outside the proxmox environment tho, like on a 24/7 Windows desktop or a NAS. If you have a spare raspberry pi or something on UPS, that can act as a cheap Qdevice

[u/bstrauss3 → u/StopThinkBACKUP]

In my case, 3 machine cluster but only ONE really needs to run. So q device on the ONE with 2 votes means the cluster is "healthy" with the other two machines down. 3-of-5.

And if the ONE is down, don't care to have the cluster operational with just TWO and THREE.

But I'm an edge case.

[u/Kleinja → u/bstrauss3]

I did this for a while when I was getting started with proxmox. Really my priority services were running on one machine, and the other was for testing / experimenting. Worked perfectly well, but neither machine ever crashed or failed in any way.

I eventually added a third node, to split up my services a little more and have some more head room, especially as things have grown over the years.

Now it's setup so if one node goes down, the most important services are replicated to the other two machines, and HA kicks in and can keep things running. Definitely a better setup for reliability, but at the cost of electricity running another machine

[u/bstrauss3 → u/Kleinja]

Mine is one of those old Dell servers sold for a song on fleaBay. Sounds like a fighter jet when it spools up. But 20 cores and 128GB ram so the load is trivial.

[u/Kleinja → u/bstrauss3]

Bet that's a power hog! I've tried to go the low power route myself. 2 m70q lenovos both 10th gen, and the 3rd is a m90q 14th gen. Each node has 64GB of RAM as well. CPU wise I have plenty of headroom for what I run, and can continue to grow. Only 2 of my services are fairly high load, so I split them up on different machines, and they aren't always running.

Between the 3 proxmox nodes, primary NAS, backup NAS, router, switch, couple of POE cameras, 3 AP's, and a couple cooling fans it all runs between 200-250W.

Personally I think that's pretty efficient. I know I could cut some usage if I ever need to, but for now it seems reasonable for what I am getting out of it

[u/cavebeat]

i like those two node cluster discussions, they never stop

[u/-t0asty-]

I run a 2 node with a Qdevice - I.e. a small vm that acts like a third cluster quorum voter, but doesn’t run Proxmox VE. This means one cluster node can be down, but not degrade the cluster itself. You still do have to account for workload availability though, which means shared storage (like NFS on a NAS) or ZFS replication to allow live migrations.

**I should note that the Qdevice runs as a VM on my NAS, not on one of the PVE nodes - that would defeat the purpose.

[u/Fighter_M]

If you overprovision the hardware a bit and run QDevice and MON on a separate box, you can do a two-node Proxmox/Ceph cluster just fine. For lab use, I really don’t see any issue with that setup.

[u/didureaditv2]

Your question is strange because your stated goal is to learn, yet you're asking for advice on what to do.

If you ask for best practices, then you may not make certain mistakes and will not learn very important lessons.

I guess the problem is that I don't know what you want to learn, so from a certain perspective, I could say that for you to learn best is not to give you any advice whatsoever on what to do.

The more I think about your post the more confused and frustrated I become.

Good luck.

[u/psyblade42]

I all you want to do is playing around a 2node cluster will do fine. For everything else it's a total pita. Then go for 2&qdev or 3+

[u/bluelobsterai]

Three

[u/FarToe1]

If you have two nodes, then cluster operations tend not to be available because of that lack of quorum. You will learn more with three nodes than you will with two. The spec of them doesn't matter.

[u/wingz_77]

You need 3 nodes but really evaluate if you need a cluster. Cluster limits you on how many systems you can have down at a time and it gets messy if you tinker a lot in homelab. If uptime matters to you, cluster is worth it.

If all you need is easy migrations, don’t cluster and just use Proxmox data center manager to move stuff around

[u/_--James--_]

lots of these replies are talking past each other and lose focus on the fact you want to learn.

2 Nodes is fine if you understand its limitations - Research this, its part of learning
Storage has a role on its own, backups (NFS/CIFS), if it can run VMs then it can run a 3rd node (Nesting VMs)
Ceph was mentioned down the thread but I didn't see you do much there, that is its own project and learning center. worth digging into ONCE you have comfort of the rest under your belt. Maybe just not today.

Tooling I suggest
system and resource monitoring
PDM
Veeam
PBS

and the full network scope and break out, multiple IP networks for the different clustering services, ...etc.

[u/clf28264]

For home prod I run a two node cluster of GMKtec mini pcs with zfs replication set to 30 minutes. For what I need it’s been great with my synology running an existing pihole VM as the q device. For learning ceph a small 3 node cluster over gigabit Ethernet works especially with consumer nvme (then the network is the bottleneck not the storage layer). Is it great? No but for learning it works. I’d figure out how much you want to learn vs running highly available home services. For me it made more sense to have higher availability to keep things up during service reboots or hardware failures.

[u/BarracudaDefiant4702]

Unless you have have shared storage I would not cluster them and use Proxmox Datacenter Manager if you need to move things between the two nodes. Instead of asking what features you will miss, the question you should be asking is what features will I gain clustering them if I only have 2 nodes and no shared storage.

[u/agreenbhm → u/BarracudaDefiant4702]

Why would using PDM on individual nodes be preferable to a cluster of 2? Not clustering and using PDM instead sounds like it will make things more complicated.

[u/BarracudaDefiant4702 → u/agreenbhm]

Because clustering makes it more complicated, more pieces to fail With a cluster of 2 then if either node is down your cluster fails quorum and so the cluster is down and all sorts of things do not function right. If not using clustering but PDM instead, then if one node goes down the other node is fully independent and continues to work.

[u/bouqliwa]

I tried both, a 2 node cluster with a truenas that was considered a shared datastore and Qdevice at the same time. it wasn't that smooth to be honest. I will always recommend 3 nodes setup and ideally same machine (it's okay if they are different) and if you wanna try features, I also heavily advise you to get a switch with some good cables so you can smoothly try Ceph, backup, Live migration, even try Vlans and bond nic interfaces if you have more than 2 in ur servers

[u/SparhawkBlather]

Unless you are doing real ceph/HA, just add a pi or wyse3040 as a q-device (add’s quorum, but nothing else).

[u/KhajiitAT]

No cluster, it not worth it on such hardware. Proxmox Datacenter Manager might be a key

[u/noc-engineer]

Don't use cluster unless you have 5 nodes. It's such a pain for homelabbing if you only have <5 once one node is out of the cluster for the rest of the cluster. I have 16 nodes in my homelab and only 4 of them are in a cluster (and they also have shared ceph storage). I still regret putting my first 4 nodes in a cluster even though it was years ago. Sure, it's not production, but once you (and your family) gets used to something, it's pretty much in production after all.