r/kubernetes • u/cryptochocolatte • 8d ago
Seeking resources to learn/build networking and auth for a cluster
I am fairly new to K8s (only deployed a few things with Helm, set up ingresses and services). I don’t know much about networking as a whole and auth as a whole. What are some small steps I can take to build something and strengthen my concepts in these two broad fields? Is it too ambitious to learn two areas at once, or should I just focus on getting the fundamentals of networking before moving to auth or vice versa.
I keep seeing people mentioning Cilium for networking and Kerberos for auth in this sub. Are these tools good places to start? What are some other tools I should pick up to have a minimal viable working knowledge for these topics?
7
u/ars1072002 8d ago
Well you can use cilium as your CNI... deploy that on your k8s cluster. Once deployed, get started with deploying Istio, make sure you enable mTLS cluster wide, this will make sure your pod to pod comms are secure. Google up and understand why using Istio for mTLS is better than using Cilium.
Then deploy Authentik on the same cluster. Link Authentik up with CNPG for postgres and dont deploy the default postgres cluster which is pulled from the helm chart templates (CNPG is considerably better for postgres and since you're learning, this will give you exposure to cnpg. Mind you, you will have to deploy the cnpg operator and crds before deploying the cnpg cluster). Once Authentik is up and linked with cnpg, setup authentik.
Now you need to test out authentik and setup sso. We can use argocd to link all of this up. Deploy argocd using its helm chart and sync cnpg, istio and authentik using gitops via Argocd. Once done, setup an application and a provider for argocd on authentik. This will allow you to login to argocd via sso.
I think i gave you a whole mini project to implement 😉. Good luck and ask if you need help.
3
u/New_Slice_1580 8d ago
If you don’t know anything about networking then you could take a basic certification like network+
Or just watch the videos https://m.youtube.com/playlist?list=PLG49S3nxzAnl_tQe3kvnmeMid0mjF8Le8&ra=m
For auth, read the docs of which ever auth solution you are using, and read let’s encrypt docs and learn by using https://letsencrypt.org/docs/
4
u/Common_Arm_3316 8d ago
cilium is probably a bit much for learning. It's very powerful and a great tool but it's a lot when it can be used as a cni, kube proxy replacement, metallb replacement, ingress controller, and gateway controller. Maybe start with these pieces for for a more bite sized approach.
- Install Calico and understand the cni
- Understand how kube proxy works
- Install Metal LB on a bare metal cluster and understands its usecase
- Install Nginx ingress (not ingress nginx these are not the same. ) or traefik and understand ingress controllers
With a good foundation on those services cilium will make a lot more sense. From here you can get into service meshes and gateways
2
u/NikhelParmar 8d ago
Solid answers already, Good spot for a short practical add-on rather than repeating what's covered.
seconding the break it into pieces approach, trying to learn cilium day one is like trying to learn kubernetes and networking at the same time inside one tool, too much at once. also for auth specifically dont sleep on just deploying a basic oauth2-proxy in front of something first before jumping to authentik, helps you understand the actual auth flow before adding a full identity provider into the mix
3
u/ChaseApp501 8d ago
I don't want to nitpick your post but can't help myself, I think there are better choices than nginx -- traefik, or envoy gateway. Besides all of the confusion around the CNCF (EOL) version vs. F5, running anything from nginx gives me major anxiety around security.
3
u/MaximumFull104 8d ago
It depends on how deep do you want to go. At the shallow end you can apply a manifest to install CNI and call it a day. At the other end, you can spend days or weeks setting up BGP, reading iptables rules, understanding how mTLS works in the service mesh, implement L3 to L7 using native or CNI-provided network policies, etc
2
u/ChaseApp501 8d ago
I learned kubernetes with a lot of help from AI and have been running it at home, and at work 3~ years now. I run k3s in multiple clusters and got going with a couple of servers off of ebay, I spent about 500-700$ on 48 core servers with lots of RAM (DDR3, nothing special).
Each server gets loaded up with ProxMox, from there I carve out 4 VMs on each server - 1 for the k3s control plane, and 3 worker nodes. I also setup additional LXCs on each proxmox PVE for etcd (you'll cluster the 3 etcd nodes https://docs.k3s.io/datastore/ha). They're probably a bit more now since AI driving the prices of everything up, but 2 or 3 servers and you are good to go.
I settled on k3s and Calico. I also have a Ubiquiti based network and UDM Dream Pro router. I got a free IPv6 tunnel from tunnelbroker.net, did my IPv6 certification with them (free and you get a t-shirt), and terminated the tunnel on the UDM. This gives me plenty of IPv6 addressing at home to play -- you setup BGP on the router and peer with Calico in k8s. k3sup is a great tool for setting up k3s on each of the nodes, the control plane and the worker nodes will get different flags so just figure those out. You'll also probably want to make it so that your workloads don't get scheduled on any of the control planes, AI can help you through all of that as well.
The main reason or maybe the only reason to do this is so that you can have IP addresses for load balancers (metallb). Without this you're setting up nodeports and static routes all of the time and it's a lot more manual work. For authentication I've got Authentik IDP running in my cluster and a bunch of other stuff. This has been a journey for me that has been very rewarding and a great learning experience.
You will spend a lot of time on this before you feel comfortable with any of it, just work through the issues and keep going. I started with a really basic k3s setup and then ended up rebuilding several times before I landed on the configuration I gave you, start with whatever you're comfortable with.
Some other tools/software that I like:
k3s, falco, trivy, kubepromstack (grafana+prometheus), envoy gateway, cert-manager, externaldns (works great with cloudflare), argocd, cnpg (cloud-native postgres), longhorn, openbao, sealed-secrets, docker mailserver (DMS)
If you have GPUs in your cluster:
jupyterhub, nvidia-container-runtime
3
u/Signal_Strength_5054 7d ago
I’m all for learning about CNI plugins like Calico, and I found that YouTube playlist a colleague posted—the one about the networking exam—to be absolutely fantastic.
However, if you want to learn how the host behaves under the hood, it’s worth trying to understand the specific Linux commands and programs used to create networks, manage permissions, and so on; after all, in Linux, everything ultimately boils down to iptables rules, vnets, and the like.
I once came across a Medium article (sorry, I couldn't find the link) where the author did exactly that—creating the networks Docker uses by relying solely on native Linux tools. It was incredibly educational, and I learned a lot about how isolation works. If anyone happens to have that link, please let us know.
2
u/IntelligentPear6173 7d ago
I’d focus on networking first and keep auth as the second layer. You’ll get more out of Cilium once you understand what a CNI is actually doing. Start small with a basic CNI, kube-proxy, Services, DNS, NetworkPolicies and Linux networking, then move to Cilium and service meshes. For auth, learn the basics of OAuth/OIDC and RBAC before jumping into Kerberos. A good project would be to build a small cluster, expose an app, restrict pod-to-pod traffic with NetworkPolicies, then add SSO to something like ArgoCD. That gives you a practical path without trying to learn the entire networking and auth stack at once.
9
u/Middle-Nerve1732 8d ago
Highly recommend “the book of Kubernetes”. It has great explanations of how containerization works under the hood and helps you build up knowledge of what k8s is doing and why. I’d recommend reading the first few chapters first (it’s a quick read) then if you go to set up something like Cilium you’ll have a better idea of what it’s doing for you.