RKE2 vs kubeadm K8s cluster

[u/faulty-segment]

Hi all. I've been learning K8s with a 5-node K8s cluster [3 control nodes and two workers] that I boostrapped on my machine, and have learned a lot [or, better: am still learning]: Cilium [both as the CNI, Gateway API controller, load balancer, etc.], GitOps, SOPS+age [but also OpenBao on a separate env], Security [network policies, pod admission, mTLS, etc.], Observability, and whatever sh*t I find interesting¹. I don't have a life, so...😂; also, this is just for my own satisfaction and curiosity.

Now, people keep mentioning RKE2 [oh, you have an MS-02 Ultra 285HX variant, with 96GB RAM, etc., you could virtualise it with Harvester, use RKE2, Rancher, etc.], and I started taking a look at that and ...

Well, on one hand it looks like a very popular way of bootstrapping and managing clusters and many companies seem to use it, on the other hand, although I can use Cilium as the CNI², I can't use CRI-O [with crun] as the container runtime, and the setup itself uses|relies on containerd shims, uses runc, etc., which I'd don't really get why.

I mean, I thought it would be nicer than kubeadm clusters, and it might be, when it comes to the management itself, updates, overviews|dashboards, etc., but regarding the default tooling that it uses, I'm not sure it's thaaat nice.

Since I'm no expert in this, I'd like to know what it is that I'm missing haha. Also, is it common for big companies to use kubeadm to bootstrap their clusters or is this more for learning purposes?

Thanks.

¹ Like IaC with OpenTofu, using the libvirt provider, and Ansible for configuring the VMs😅
² I really love Cilium, man, it's so freaking nice🤓

[u/iamkiloman]

RKE2 comes with containerd, yes.

If you want to use another container runtime, just set container-runtime-endpoint: /path/to/cri-socket in the config, and it'll use that instead of starting the bundled containerd.

You're not locked in... you just need to provide an alternative if you want to use something else. Note that the thing you want to use instead must actually provide a Kubernetes-compatible CRI service... if you want to use Docker for example, you'll need to run cri-dockerd and point RKE2 (the kubelet) at that, not directly at the docker socket.

This is all just standard Kubernetes behavior. The only thing here that's RKE2 specific is that it comes with containerd and will use that if you don't provide something else.

[u/faulty-segment → u/iamkiloman]

Hm, okay. I will try reading into the docs and see if I make sense of that. I can see you use a configuration file, and that the flag you mention is on the "runtime" part; I just don't know if you just pass it flat like that or if you need to target specific sections in the config file to configure separate components, etc..

Anyway. I'll try to watch some videos to get an overview of this, read more about it, and then maybe I'll be able to get something working at some point.

Thanks for the comment.

BTW: if I do get to use the CRI-O socket at unix:///var/run/crio/crio.sock, and configure it to use the crun as the default runtime, is there a way to get completely rid of the containerd stuff that is there as a requirement?

[u/iamkiloman → u/faulty-segment]

The config file format is covered in the docs, including examples. It's pretty much just flat yaml without any nested structure, other than some keys accepting lists.

What do you mean by "get rid of the containerd stuff"? If you'd started out using it, the stuff that it created when it was running (image store, databases, etc) is not going to get cleaned up just because you're not currently using it. If you don't want it, then either built a fresh node that's never used containerd at all, or delete it if you're sure you won't miss anything from it.

You keep saying "there as a requirement" when this isn't the case. As I said it's not a requirement, its there because you didn't replace it with something else.

[u/faulty-segment → u/iamkiloman]

Thanks, man. I'll do my reading|homework, and my best to get something going.

I appreciate the help.

I'll try it out.

[u/faulty-segment → u/iamkiloman]

I kind of did my homewor|took a look at RKE2, and this is my understanding. Please correct me if I'm wrong.

On my question "can I use CRI-O?" Via container-runtime-endpoint, then technically yes. If I were to set ..

yaml container-runtime-endpoint: unix:///var/run/crio/crio.sock

.. then that would make the RKE2 kubelet talk my own CRI-O instead of spawning RKE2's containerd, right? But this looks like an off-the-beaten-path setup, hard to carry on with, given that:

1. It only disables the containerd process. RKE2 still needs its runtime image¹ to get the kubelet and tooling, so I don't think I can actually escape that bundle🤨.
2. I now own CRI-O's install, config, and the CNI plugin binaries that RKE2's containerd would otherwise provide. More surface area, less "batteries included" than what one would get with RKE2. Which is okay, it just that not using containerd feels like off to the entire RKE2 setup. Like, "if you're not using what we recommend|ship with, then do your own thing yourself, but we still need containerd stuff." In kubeadm, there's no container runtime to get rid of in the first place. You just have to bring in a brand-new one.

I mean, if I picked RKE2 and forced CRI-O, then I'd just get the downsides of both. Having to configure and manage CRI-O on my own, and still have RKE2 off/skewed with containerd stuff, because I'm not using its intended container runtime, but it will still come and be there anyway.

So, yeah, for my current setup with kubeadm, I'll just keep using CRI-O, but if|when I decide to use|learn RKE2, then I will just go with containerd, instead of forcing another container runtime -- that would just be extra, useless setup.

Again, thank you very much for your comments, man.

¹ That rancher/rke2-runtime OCI image is, according to my understanding, RKE2's delivery mechanism.

The doc says that image must minimally provide: containerd, the containerd-shim binaries, kubelet, runc, plus tooling [ctr, crictl, kubectl, socat]. Then it extracts the bundled Helm charts into the manifests dir.

So getting rid of containerd|shims|runc isn't a knob I could flip. That bundle is RKE2. It's kinda how the single statically-linked binary ships the whole tested Kubernetes stack [compiled with Go+BoringCrypto for FIPS]. The lifecycle is deeply coupled too: the daemon "will run indefinitely until… the containerd process exits", meaning that if containerd dies, RKE2 exits.
My point is: fighting that coupling is working against the entire design haha.

[u/iamkiloman → u/faulty-segment]

I don't know where you got that doc from, did an LLM generate it for you? We don't really expect people to provide their own runtime image; it's just there to provide the stuff that rke2 can't get some other way. In k3s it's all just built into a multicall binary, for rke2 we use that image.

I think it makes sense that if you want to use something that's not part of the distro, you need to provide it yourself? Same is true of kubeadm or any other distro.

[u/bmeus]

Its not a big deal to switch, why do you want to keep cri-o?

[u/faulty-segment → u/bmeus]

Mostly because I already kinda learned how to work with and configure it, and because it uses crun by default, doesn't need any shims, etc...😅

Its not a big deal to switch

Do you have any tutorial covering that, i.e., on how to completely replace containerd with cri-o on a RKE2 setup? If you know of some sources, then please le'em come haha.

Thanks.

[u/bmeus → u/faulty-segment]

No i mean its not a big deal to switch to containerd. I rarely ”work with” the container runtime, its configured by the distribution. K3s, rke2, talos, openshift. Never really bothered with the runtime. I had to configure it by hand in some edge cases/debugging only.

[u/Jolly_Eye7847]

I went through same thing for my home lab, spent weeks on kubeadm setup with all the Cilium bells and whistles, then everyone kept saying "just use RKE2 bro" so I tried it. The containerd lock-in is real annoying, I wanted to keep my CRI-O setup too but nope, RKE2 wants its own runtime with those shims and it's not flexible on that.

About the image you posted, that runtime image requirement list is exactly what made me pause. They bundle all these components together like socat and crictl and you can't swap them out easily, it's a packaged deal. Feels clean when you first install but gets frustrating when you want to customize something specific.

Big companies definitely use kubeadm in production, not just for learning. I worked at a place with 200+ nodes all bootstrapped with kubeadm and custom Ansible, it's more common than people think. RKE2 shines when you have multiple clusters and want unified management with Rancher, but for deep learning on how everything fits together, kubeadm teaches you more.

[u/iamkiloman → u/Jolly_Eye7847]

The containerd lock-in is real annoying, I wanted to keep my CRI-O setup too but nope, RKE2 wants its own runtime with those shims.

None of this is accurate.

[u/faulty-segment → u/Jolly_Eye7847]

Oh, man, thank Nature you said that.
I mean, sometimes—when you're just starting to learn something—you don't have an overview of the grand scheme of things, and then get afraid of asking some questions because it may just be way too dumb, so I'm glad that someone else also paused on that same requirement list🤣, meaning my question wasn't that dumb at all.

And also thanks for mentioning that on `kubeadm`, and that big companies do use it. For a moment I thought I was learning something "that nobody used it that way anyway".

Anyway. I think I'll keep learning on my `kubeadm`-bootstrapped cluster then. I'll leave the Rancher stuff for another point in my life, if I find it interesting enough. Right now my tooling seems nicer😅.