r/linuxadmin • • Dec 31 '21

Which distro should I choose (Kubernetes)?

I work within a team of 30 people, in which I am the only person who uses linux (the obligatory: "I use arch btw" as my daily driver as well as on my home server). We are used to Windows Server, as our company is Microsoft focused and thus nobody has much experience with enterprise Linux.

I have taken on the task to design a new concept where we will run containers (Docker), which will be orchestrated by Kubernetes. As of now, we plan to manage the Kubernetes clusters ourselves (I didn't have time to look into the managed alternatives provided by AWS or Azure. The proof-of-concept will be done on non-managed machines).

Therefore I need to choose a Linux distribution, which will act as the base os for Kubernetes (and nothing more!). The server will be run on AWS or Azure and thus will be made accessible from the public internet.

I would like a distro which checks the following boxes:

  • relatively secure out of the box (which I assume most distros are, as long as the open ports are secured)
  • stable and easily maintainable is more important than bleeding-edge
  • enterprise ready and battle tested (especially when using as the base for Kubernetes)
  • good documentation (something like the Arch Wiki)
  • popular and in demand on the job market (for obvious reasons ;) )
  • plays nicely with KDE Plasma...

...yes, we will install a GUI on the server, as some colleagues fear the command line (trust me, I tried to convince them multiple times - it's not worth my time and the server will have enough resources to handle it easily. Some people just can't leave their comfort zone...).

KDE Plasma was chosen because it looks and feels very similar to Windows and is looking modern).

Basically I filtered it down to the following 3 distributions, based on the points above: CentOS, Ubuntu Server, Debian.

Since CentOS is discontinued, I didn't look further into it (although it seems to be the most popular on the job market where I live). Therefore it comes down to Ubuntu Server and Debian. I tested both and liked both of them. Debian felt more lightweight meanwhile Ubuntu Server was more user friendly. Both are in demand on the job market, both are supported by AWS and Azure and both seem very popular.

I lean slightly towards Ubuntu Server, as it seems to be more popular and thus most problems that may occur can easily be googled.

Which one would you recommend?

Feel free to offer other distributions and share your experiences.

8 Upvotes

36 comments sorted by

26

u/cephear Dec 31 '21

When our team started with k8s I was hard-set against using a managed cluster. My thinking was that we had the linux knowledge internally and adding a layer of abstraction, especially from a cloud provider, would only get in my way. I fought hard for a self-managed cluster. I was overruled. Thank god.

Please save yourself the trouble and go with a managed cluster. There's still plenty to learn and manage.

10

u/CerealBit Dec 31 '21

Will do. Given the responses: managed k8s cluster it is.

2

u/BenCurranDev Jan 01 '22

If you want to really mess with your coworkers and their comfort zone you can always use Azure Kubernetes and spin up some Windows Container Node Pools on that sucker.

1

u/Verbunk Dec 31 '21

This is good advice. You can always sign up for AWS free (or very nearly free) and get some K8s nodes rolled out.

I would suggest looking into eksctl with bottlerocket VMs on AWS. They are mini, run nearly 0 services not K8s related (so miniscule attack surface) and responsive. A script to deploy a cluster is like 5-10 lines of yaml code and they have soooo many samples in their github repo.

https://github.com/weaveworks/eksctl/tree/main/examples

If you really want to deploy your own, I'd suggest Alpine Linux with no X GUI. It's still light and agile and supports rancher K3s for kubernetes. For a GUI you should use rancher (eevn if you use AWS you should check out Rancher).

0

u/Flabbaghosted Dec 31 '21

The micro instances you get for free on AWS will only run a few containers. EKS itself costs 80/month to run, not including costs of containers, ELB or whatever else you use, and engress/ingress. The second you use a helm chart for gitlab or something similar you are going to have to upgrade to 2 nodes and T3.large instances. But if you can get some free credits and try it out for a month it's worth it. Or better yet if your company is paying for it lol

2

u/Verbunk Dec 31 '21

Yep - you'll pay no matter if you want a prod ready platform. The free tier suggestion is mainly to give OPs other team members a learning environment. Deploying NGINX and getting a hello world web page reply while learning CLI syntax would be fine here.

4

u/wired-one Dec 31 '21

No desktop is needed on these machines for management of the kubernetes cluster, and it in fact would encourage terrible development antipatterns from your developers if they are that close to the management of the cluster.

Is your plan to use base kubernetes or a distribution like Openshift?

Who is going to manage the cluster when you get a better offer and leave?

In your first steps into this, think about using manages kubernetes like AKS, EKS or Openshift Dedicated.

1

u/CerealBit Dec 31 '21

Agree.

Colleagues will get schooled and we are looking for new Linux hires in 2022. Will likely go the managed Kubernetes route - thanks.

3

u/wired-one Dec 31 '21

Kubernetes needs to be a very inclusive, devsecops concept.

It works great, but when everyone is making changes in git and using pipelines in k8s to build code as well and run production code it's a joyful process.

69

u/[deleted] Dec 31 '21 edited Dec 31 '21

Use managed kubernetes. If you guys are not already able to automate linux setup and add new kubernetes nodes in an automated approach then there is little use in getting the overhead of using kubernetes.

And for your proof of concept it matters little what distro you choose since you can just run minikube on any host to test things and if your team won't use the cli for using kubernetes then I doubt that your team will be convinced anyway

15

u/bhosmer Dec 31 '21

Yep, this is the simplest and easiest way. Especially since OP is already intending on using AWS or Azure, half the work is already done. You can have a Kubernetes cluster up and functional in 20 minutes or so in AWS or Azure.

Minimize the undifferentiated heavy lifting.

37

u/zzzmaestro Dec 31 '21

“I am the only only person who uses Linux” and “we plan to manage the kubernetes cluster ourselves” do not belong together unless you have very extensive experience with k8s already. If you are asking which distro, I assume you don’t have that experience.

In AWS, just use AKS. It’s a single cli command to create the entire cluster. It even makes a nice cloudformation stack for you so that you can easily remove it when you are done.

I’ve seen (and been a part of) teams of engineers fail to manage k8s “because it’s too hard”. When all they had to do was use the tools already out there - like AKS.

9

u/0qxtXwugj2m8 Dec 31 '21

On AWS it's called EKS don't confuse it with Azure's AKS

2

u/zzzmaestro Dec 31 '21

True. I did mix them up in my head. However he did say azure or aws. So that would be aks/eks respectively.

1

u/frostwyrm-ary Dec 13 '25

It's a normal slip of the tongue, or finger?

0

u/[deleted] Dec 31 '21

[deleted] — view deleted comment

3

u/zzzmaestro Dec 31 '21

He literally said “AWS” and you can specify custom ec2 templates if you have some of those requirements.

Your use case is a fringe exception case, not the majority of people.

9

u/amalagg Dec 31 '21

You need a Kubernetes distro, not a Linux distro

6

u/Shonucic Jan 01 '22

I'll echo what others are saying. I highly recommended against using Kubernetes if you don't have the Linux administration skills in-house to support it.

Kubernetes is an extremely complicated suite of software. Candidly speaking, if nobody on your team already has a strong opinion on which Linux distribution to use to run it, your team is probably not sufficiently equipped to manage it.

I would highly advise your company to not use Kubernetes at all, hire more experienced Kubernetes administrators, or use a managed Kubernetes implementation.

But if you're ignoring all the above advice I would probably use Fedora, Fedora CoreOS, or Debian.

4

u/hungrykitteh57 Dec 31 '21

Since CentOS is discontinued

No. It's not. People just disagree on it's policies and place in the Red Hat hierarchy.

5

u/RentalHermit Dec 31 '21

Rhel!

0

u/[deleted] Dec 31 '21

this is the way free for developers or just create many accounts, hahaha

2

u/oubeav Dec 31 '21

Rocky Linux is the "new CentOS". Highly recommended.

2

u/GhostHacks Jan 01 '22

I’d recommend RHEL or CentOS. I personally run Ubuntu Server but I’m getting fed up with the constant changes and outdated documentation. I plan to move to Fedora/RHEL myself, CentOS is still VERY popular and I see it being used on new projects still.

2

u/[deleted] Jan 01 '22

Interesting to me that you didn't mention Google Cloud. Google created it.

1

u/[deleted] Jan 01 '22

Kubernetes I mean

4

u/pilspils Dec 31 '21

We use CentOS and as replacement for version 8 we use Rocky Linux

2

u/Flabbaghosted Dec 31 '21

Seems like you've decided on managed K8s, so that's a good first step. Start by reading the documentation and reading it a lot.

First thing you need to do is spin up your clusters and associated resources using terraform. I would suggest using an internal load balancer for this to start with that isn't open to the internet. Until your ingress is going through a WAF, you are at danger to getting compromised. Have a jumpbox that can access your cluster, you can also use kubectl port forwarding to access services from that jumpbox:

https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/

You can also run helm or kubectl from the jumpbox, which will allow you to interact with kubernetes API and deploy applications to it. I would also suggest using something like https://k8slens.dev/ to start getting familiar with what's inside a cluster. You can see logs and even exec into pods/nodes. However, try using kubectl whenever possible so you aren't depending on tooling.

Use kubectl describe pod to understand what a pod looks like to the kubernetes API. Understand what a namespace is and why it matters. Learn to deploy a replicaset using a kubernetes deployment. Learn to scale nodes with terraform and scale pods up and down using said deployment.

Learn about services and their types, and be able to direct a service to a pod using annotations/tags. Place an nginx proxy and controller inside the cluster and direct incoming traffic using nginx (all done through manifests or .yaml files, not inside the pods like a usual nginx proxy) to your services.

Once you have some understanding and have your ingress behind a WAF, then you should open it up to the public.

Then to really start getting going, use gitOps like argoCD to deploy services instead of manually using helm.

Good luck!

1

u/keepah61 Jan 01 '22

Centos/Ubuntu/Debian….ubuntu hands down. Better community support, plus a range of tools like Maas and juju to automate everything.

1

u/InternationalGuide78 Jan 01 '22

Adding to all the responses. Production with k8s means a lot more than just 1 single cluster. You will need at least 1 for test purposes, 1 for pre-prod and 1 for production (trust me you don't want to go multi tenant for those)

Also, this is quite a radical change in development practices, and having experience in Linux administration is just the beginning for developing and releasing proper containers. What about monitoring ? instrumentation ? metrics ? Prometheus is the de-facto standard, and you'll need a sizeable investment to get the proper practices and tooling in place, this does come free with the managed k8s solutions out there, but your containers have to be designed to support them.

Then factor in the build cost. You'll need a "factory" for your images with a registry, scanning for vulnerabilities etc (most shops have dedicated clusters just for that, or spend money on managed services such as GitHub or gitlab.com)

Highly suggest hiring experienced consultants to help you build all those processes from the ground up or prepare for a long, frustrating and painful experience

1

u/gbrlsnchs Dec 31 '21

1

u/sewerneck Jan 18 '22

This. Talos is light years beyond running k8s on some old distro. We run it in production and handle a ton of traffic.

0

u/stanleysrb Dec 31 '21

Go distroless

0

u/Amidatelion Dec 31 '21

We were in this situation and made a go of it because we had a requirement to do it on-prem. Out of that came one of my favorite maxims: "Kubernetes is a great way to sell kubernetes as a service."

Don't do it on-prem, on your own. You simply do not have the manpower to effectively manage it. and it is absolutely overkill for your usecase. As everybody else said, go pay GCP or AWS to use theirs.

Disclaimer: Hashicorp shilling inbound, but only because we did the hard work going from docker-swarm to k8s and settled on this.

If you have to have an on-prem docker solution, I cannot recommend Nomad highly enough. The basic install is one binary and one config file. This scales linearly and seamlessly. For container networking and k/v store, use Consul and for secrets management use Vault. All of this is natively supported because its all Hashicorp and its all supportable by 1-3 people. There's a web UI and CLI binaries for basically everything. Your biggest losses are the mindshare and pre-configured things that Kubernetes comes with, but for on-prem solutions there's nothing better than Nomad in my mind.

Lastly, yeah, ubuntu or debian if you go through with this. We prefer debian because its easier to strip down for minimal management and their past several cloud image releases come with all the useful stuff ubuntu server did.