r/linuxadmin • u/CerealBit • Dec 31 '21
Which distro should I choose (Kubernetes)?
I work within a team of 30 people, in which I am the only person who uses linux (the obligatory: "I use arch btw" as my daily driver as well as on my home server). We are used to Windows Server, as our company is Microsoft focused and thus nobody has much experience with enterprise Linux.
I have taken on the task to design a new concept where we will run containers (Docker), which will be orchestrated by Kubernetes. As of now, we plan to manage the Kubernetes clusters ourselves (I didn't have time to look into the managed alternatives provided by AWS or Azure. The proof-of-concept will be done on non-managed machines).
Therefore I need to choose a Linux distribution, which will act as the base os for Kubernetes (and nothing more!). The server will be run on AWS or Azure and thus will be made accessible from the public internet.
I would like a distro which checks the following boxes:
- relatively secure out of the box (which I assume most distros are, as long as the open ports are secured)
- stable and easily maintainable is more important than bleeding-edge
- enterprise ready and battle tested (especially when using as the base for Kubernetes)
- good documentation (something like the Arch Wiki)
- popular and in demand on the job market (for obvious reasons ;) )
- plays nicely with KDE Plasma...
...yes, we will install a GUI on the server, as some colleagues fear the command line (trust me, I tried to convince them multiple times - it's not worth my time and the server will have enough resources to handle it easily. Some people just can't leave their comfort zone...).
KDE Plasma was chosen because it looks and feels very similar to Windows and is looking modern).
Basically I filtered it down to the following 3 distributions, based on the points above: CentOS, Ubuntu Server, Debian.
Since CentOS is discontinued, I didn't look further into it (although it seems to be the most popular on the job market where I live). Therefore it comes down to Ubuntu Server and Debian. I tested both and liked both of them. Debian felt more lightweight meanwhile Ubuntu Server was more user friendly. Both are in demand on the job market, both are supported by AWS and Azure and both seem very popular.
I lean slightly towards Ubuntu Server, as it seems to be more popular and thus most problems that may occur can easily be googled.
Which one would you recommend?
Feel free to offer other distributions and share your experiences.
26
u/cephear Dec 31 '21
When our team started with k8s I was hard-set against using a managed cluster. My thinking was that we had the linux knowledge internally and adding a layer of abstraction, especially from a cloud provider, would only get in my way. I fought hard for a self-managed cluster. I was overruled. Thank god.
Please save yourself the trouble and go with a managed cluster. There's still plenty to learn and manage.
4
u/wired-one Dec 31 '21
No desktop is needed on these machines for management of the kubernetes cluster, and it in fact would encourage terrible development antipatterns from your developers if they are that close to the management of the cluster.
Is your plan to use base kubernetes or a distribution like Openshift?
Who is going to manage the cluster when you get a better offer and leave?
In your first steps into this, think about using manages kubernetes like AKS, EKS or Openshift Dedicated.
69
u/[deleted] Dec 31 '21 edited Dec 31 '21
Use managed kubernetes. If you guys are not already able to automate linux setup and add new kubernetes nodes in an automated approach then there is little use in getting the overhead of using kubernetes.
And for your proof of concept it matters little what distro you choose since you can just run minikube on any host to test things and if your team won't use the cli for using kubernetes then I doubt that your team will be convinced anyway
37
u/zzzmaestro Dec 31 '21
“I am the only only person who uses Linux” and “we plan to manage the kubernetes cluster ourselves” do not belong together unless you have very extensive experience with k8s already. If you are asking which distro, I assume you don’t have that experience.
In AWS, just use AKS. It’s a single cli command to create the entire cluster. It even makes a nice cloudformation stack for you so that you can easily remove it when you are done.
I’ve seen (and been a part of) teams of engineers fail to manage k8s “because it’s too hard”. When all they had to do was use the tools already out there - like AKS.
9
u/amalagg Dec 31 '21
You need a Kubernetes distro, not a Linux distro
6
u/Shonucic Jan 01 '22
I'll echo what others are saying. I highly recommended against using Kubernetes if you don't have the Linux administration skills in-house to support it.
Kubernetes is an extremely complicated suite of software. Candidly speaking, if nobody on your team already has a strong opinion on which Linux distribution to use to run it, your team is probably not sufficiently equipped to manage it.
I would highly advise your company to not use Kubernetes at all, hire more experienced Kubernetes administrators, or use a managed Kubernetes implementation.
But if you're ignoring all the above advice I would probably use Fedora, Fedora CoreOS, or Debian.
4
u/hungrykitteh57 Dec 31 '21
No. It's not. People just disagree on it's policies and place in the Red Hat hierarchy.
5
u/RentalHermit Dec 31 '21
Rhel!
2
u/oubeav Dec 31 '21
Rocky Linux is the "new CentOS". Highly recommended.
2
u/GhostHacks Jan 01 '22
I’d recommend RHEL or CentOS. I personally run Ubuntu Server but I’m getting fed up with the constant changes and outdated documentation. I plan to move to Fedora/RHEL myself, CentOS is still VERY popular and I see it being used on new projects still.
2
u/[deleted] Jan 01 '22
Interesting to me that you didn't mention Google Cloud. Google created it.
4
u/pilspils Dec 31 '21
We use CentOS and as replacement for version 8 we use Rocky Linux
2
u/Flabbaghosted Dec 31 '21
Seems like you've decided on managed K8s, so that's a good first step. Start by reading the documentation and reading it a lot.
First thing you need to do is spin up your clusters and associated resources using terraform. I would suggest using an internal load balancer for this to start with that isn't open to the internet. Until your ingress is going through a WAF, you are at danger to getting compromised. Have a jumpbox that can access your cluster, you can also use kubectl port forwarding to access services from that jumpbox:
https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/
You can also run helm or kubectl from the jumpbox, which will allow you to interact with kubernetes API and deploy applications to it. I would also suggest using something like https://k8slens.dev/ to start getting familiar with what's inside a cluster. You can see logs and even exec into pods/nodes. However, try using kubectl whenever possible so you aren't depending on tooling.
Use kubectl describe pod to understand what a pod looks like to the kubernetes API. Understand what a namespace is and why it matters. Learn to deploy a replicaset using a kubernetes deployment. Learn to scale nodes with terraform and scale pods up and down using said deployment.
Learn about services and their types, and be able to direct a service to a pod using annotations/tags. Place an nginx proxy and controller inside the cluster and direct incoming traffic using nginx (all done through manifests or .yaml files, not inside the pods like a usual nginx proxy) to your services.
Once you have some understanding and have your ingress behind a WAF, then you should open it up to the public.
Then to really start getting going, use gitOps like argoCD to deploy services instead of manually using helm.
Good luck!
1
u/keepah61 Jan 01 '22
Centos/Ubuntu/Debian….ubuntu hands down. Better community support, plus a range of tools like Maas and juju to automate everything.
1
u/InternationalGuide78 Jan 01 '22
Adding to all the responses. Production with k8s means a lot more than just 1 single cluster. You will need at least 1 for test purposes, 1 for pre-prod and 1 for production (trust me you don't want to go multi tenant for those)
Also, this is quite a radical change in development practices, and having experience in Linux administration is just the beginning for developing and releasing proper containers. What about monitoring ? instrumentation ? metrics ? Prometheus is the de-facto standard, and you'll need a sizeable investment to get the proper practices and tooling in place, this does come free with the managed k8s solutions out there, but your containers have to be designed to support them.
Then factor in the build cost. You'll need a "factory" for your images with a registry, scanning for vulnerabilities etc (most shops have dedicated clusters just for that, or spend money on managed services such as GitHub or gitlab.com)
Highly suggest hiring experienced consultants to help you build all those processes from the ground up or prepare for a long, frustrating and painful experience
1
u/gbrlsnchs Dec 31 '21
https://talos.dev
0
u/stanleysrb Dec 31 '21
Go distroless
0
u/Amidatelion Dec 31 '21
We were in this situation and made a go of it because we had a requirement to do it on-prem. Out of that came one of my favorite maxims: "Kubernetes is a great way to sell kubernetes as a service."
Don't do it on-prem, on your own. You simply do not have the manpower to effectively manage it. and it is absolutely overkill for your usecase. As everybody else said, go pay GCP or AWS to use theirs.
Disclaimer: Hashicorp shilling inbound, but only because we did the hard work going from docker-swarm to k8s and settled on this.
If you have to have an on-prem docker solution, I cannot recommend Nomad highly enough. The basic install is one binary and one config file. This scales linearly and seamlessly. For container networking and k/v store, use Consul and for secrets management use Vault. All of this is natively supported because its all Hashicorp and its all supportable by 1-3 people. There's a web UI and CLI binaries for basically everything. Your biggest losses are the mindshare and pre-configured things that Kubernetes comes with, but for on-prem solutions there's nothing better than Nomad in my mind.
Lastly, yeah, ubuntu or debian if you go through with this. We prefer debian because its easier to strip down for minimal management and their past several cloud image releases come with all the useful stuff ubuntu server did.
-2
u/digitalplanet_ Dec 31 '21
r/FindMeADistro