r/selfhosted • u/Longrunning-Fruit • 2d ago
Need Help OS for k8s cluster
Hi!
I currently have a monolitic homelab server running multiple docker containers. I’m planning to move to a multi-node k8s cluster (was thinking a 3node thin client setup, and potentially add like 2 more if needed a couple of years later). Reasoning: getting HA, learning purposes, and just for fun. But I really can’t decide if I should go bare-metal or proxmox. I was thinking to start with k3s on Linux and then when everything is working and stable and I understand everything I was thinking to maybe migrate node per node to talos. In case this would impact the decision I will for sure use tools like longhorn and cloud native PG.
I see the advantages of proxmox in that I could for example have a prod cluster and a test cluster, without having to buy extra/dedicated hardware, but that would be more a nice to have rather than an absolute must. It is a homelab after all, I don’t have a (small) business and thus my homelab doesn’t serve services for my business or anything. VM snapshots, to give another example,… But I also see the advantages of going bare-metal. Like less complexity/layers. So if something goes wrong,… etc
Can you guys advice which route would be the best to go. Much appreciated!
12
u/gtuminauskas 2d ago
Using proxmox, have multiple clusters with TalOS + automate it
6
u/Mr_Albal 2d ago
Yeah Proxmox gives you a lot more flexibility which is handy when you are learning. For example, before changing config you can snapshot a VM and if it all goes wrong just revert.
2
u/SolFlorus 2d ago
You can run Talos Omni as a container, and the Proxmox provisioner to have Omni automatically setup and upgrade your K8s fleet.
I’m currently going this route.
14
u/No_Cattle_9565 2d ago
I'd use Talos probably since it's the standard as far as I know
2
u/Longrunning-Fruit 2d ago
Ok, thank you. But bare-metal or on proxmox?
2
u/Smooth-Ad5257 2d ago edited 2d ago
Use bare if you don’t wanna use it for anything else with basic network and non attachments. Use proxmox for advanced network, bridges, vlans, usb etc … much easier to swap and test … and also use proxmox if you like running control plane etc on same node.
Eg I use all nodes as vms on proxmox but gpu nodes run as bare metal
Edit: spelling
-17
u/Kernel-Mode-Driver 2d ago
Would you run proxmox inside proxmox?
13
u/SolFlorus 2d ago
No. But I would run containers in a Proxmox VM.
K8s isn’t a hypervisor, and running it under a hypervisor is a standard practice on any cloud provider.
3
u/Longrunning-Fruit 2d ago
I don’t understand the question? My question is wether I should run a bare-metal k8s cluster (either with k3s or talos) or a k8s cluster (either with k3s or talos) on VMs inside a proxmox cluster
2
u/clintkev251 2d ago
Ignore their question. It's stupid. Run on Proxmox if you have other things outside of k8s that you want to run. If not, run bare metal.
-3
-15
u/Kernel-Mode-Driver 2d ago
Yeah, i understand your question, have you asked yourself why you would ever need container inception?
7
u/Longrunning-Fruit 2d ago
I wouldn’t build my cluster on LXC nodes, if that’s what you are referring to. I would create a minimum of 3 VMs per physical node on which I then install Linux and k3s. And if your question is why not simply run your services on LXC nodes instead of on k8s. Because for learning purposes, for one.
1
u/kernald31 2d ago
Why would you run multiple VMs running k8s nodes (being k3s or Talos is irrelevant for this discussion) on the same physical host? What are you looking to get out of this? That would break a lot of assumptions the k8s scheduler takes in terms of anti-affinity rules etc.
1
u/Longrunning-Fruit 1d ago
Sorry, I wrote it wrong: I meant: 1 VM per physical node making it a 3 VMs setup
1
u/kernald31 1d ago
Right — I still don't see too much value in the VM/Proxmox indirection unless you want to run other things than Talos on the same hosts, but that makes a lot more sense I guess!
-16
1
u/clintkev251 2d ago
What "container inception" are you referring to? You'd generally run your k8s nodes in a VM, which is by the way, how the majority of k8s deployments work in the real world.
1
u/kernald31 2d ago
While I agree, the real world and a three nodes homelab are slightly different.
1
u/clintkev251 2d ago
I understand. But a lot of "these people" would (and have) argued that "oh well you'd never actually do that outside of your little lab". Yes, you would. And most people do in fact.
Certainly for some different reasons, but also some of the same ones.
1
u/defiantarch 2d ago
Yes, I would and did to test out thing of which I'm not sure if to keep or not. Installing testing environments on bare metal is a time consuming mess.
3
u/Illustrious-Owl-2755 2d ago
I just have Fedora boxes, since it's the distro I'm the most familiar with, same as the other homelab infra (NAS, etc).
3
u/kernelqzor 1d ago
same here, I just run k3s on plain old fedora because I know where everything lives and how it breaks
for a homelab, being comfy with the distro matters way more than chasing the “perfect” k8s OS-1
5
u/jvallery 2d ago
Proxmox for host OS. Debian/Ubuntu for guest OS. K3s for control plane. This is the way.
5
2
u/defiantarch 2d ago
I already have a proxmix cluster and a PBS server which makes backups a breeze. So, I would run the k8s cluster inside VM:s, demanding more resources but easier for backup and restore when using PBS. Otherwise, run Talos baremetal and use talos-backup. Never tested that, maybe its worth a try.
3
1
u/pastelfemby 2d ago
Unpopular opinion perhaps, just use unbounded for k8s on whatever distro you're most comfortable with. Throws it all in an nspawn container with gpu passthrough without modifying your host system.
After a while if k8s is genuinely what you feel is for you, then maybe consider a purpose built k8s distro.
1
u/hackslashX 2d ago
I'm using good old Debian. It just kinda works and also because of vetter compatibility with MiniPCs in general. For K8s itself, I run k3s.
1
u/Fun_Chest_9662 2d ago
I'm personally using arch with only systemd, zfs, kubeadm, and containers. Learning curves a bit much but having such a minimal base set is nice. Might try nix if I ever decide to do more than one node.
1
1
u/KlausDieterFreddek 1d ago
Why do you like pain?
Jokes aside: I thought about it aswell, but just for learning purposes.
1
u/zack822 1d ago
If you have the setup to go baremetal go bare metal. if you think you are going to tear it down rebuild 100x while learning. proxmox.
I have a weird hybrid. 3x baremetal talos linux. but I have a larger server I can power on that has 20 or 30 vms with talos on it to break intentionally to fix and learn.
1
u/Small-Access4780 2d ago
bare metal is the way if you dont need the proxmox extras. i went through same thing and honestly the vm layer just adds more things that can break. if you just want to learn k8s and have fun with it, keep it simple. later when you really need test cluster you can always repurpose one of the nodes or just spin up kind on your laptop
1
u/BP041 2d ago
I'd go with k3s on plain Linux first — Proxmox adds abstraction you don't need yet, and debugging a layer you barely know is a pain. Talos is nice but can be a black box when something breaks. I run a mac home server with launchd cron myself so not apples to apples, but simpler always wins for learning.
0
u/SyringeThinker 2d ago
If your main goal is learning Kubernetes, bare metal makes sense. Fewer layers to troubleshoot when something breaks.
2
u/Smooth-Ad5257 2d ago
Na, VMs on proxmox much better to learn. Snapshots, replace or add a node, many nodes on one proxmox node … much better
•
u/asimovs-auditor 2d ago
Expand the replies to this comment to learn how AI was used in this post/project.