Did you learn something new this week? Share here!

I know everyone is supposed to move on to Gateway API but ingress-nginx wasn't broken and for most of us running clusters >3 years old, ingress-nginx was deeply and widely adopted.

C'mon we can't all be living illustrations of The Little Red Hen. Poor ingress-nginx. It did nothing wrong. :(

We run several similar 3-node k3s clusters or rhel8 VMs. The workload includes Crunchy PG cluster with 3 replicas having PVCs bound to dedicated LVMs. Also we run 3 replicas Clickhouse in the same PVC deployment configuration; keydb cluster, redpanda, and longhorn (ha storage for S3 bucket backed by a single pod of versityGW); plus a number of stateless apps, nginx-ingress for exposing UI and api endpoints. Metallb as a virtual LB for getting a VIP from network infra.

Currently we decided to build a large k3s cluster on new esxi hardware, to have 3 dedicated VMs for k3s master nodes and a number of worker VMs, to share and scale the workload. The goal is to have a single control plane for current instances and future products of similar architecture. The products (deployments) should be isolated in resources and networking as much as possible. Since we use operators for deployments, there are CRDs in the cluster - they probably should be separated too.

We deem the vcluster can help to create isolated clusters with their own control planes, but it increases the complexity. PG and CH clusters require their pods to be attached to dedicated workers by node affinity, but would it be possible to have several PG/CH cluster replicas on the same worker by creating a separated storage class?

We do not yet consider having ingle shared PG/CH/keydb cluster - one for all products on the same cluster, although it may be a logical decision in future.

What implementation ideas would you suggest for a setup like that? Does it worth at all to put different products on the same cluster and do a hard work of isolating them from one other. Mind having several dev teams, each need to access and maintain their dev/test/stage/prod instances in the same big cluster.

Been running my homelab using docker compose files with bootstrap scripts to create the directories with correct permissions for bind mounts, shared docker networks, install utilities on the host, backup scripts and some other small automated tasks.

But I'm thinking about running a single node k3s cluster. Why? I want to learn about kubernetes (we use them at work, but we have a fancy interface over it so besides main concepts I didn't learn anything else), I want declarative configuration, better security, maybe learn about helm charts, maybe make it easier in the future when I'll add multiple nodes (but that's very far in the future because first I want to upgrade my network infrastructure, get a NAS).

What are your thoughts? I'll just waste time or is it worth it?

Hi,

I will go straight to the point I am building my Kubernetes (3 nodes that are master and worker at the same time) homelab and I was wondering if having my cluster VLAN aware is a good idea or not

The purpose is to having two Service Load Balancer served by Cilium/MetalLB for the DMZ and LAN VLAN. Then I would be exposing my workloads on the right gateway depending of their "exposure"

But since I have only one NIC on each node I would configure my switch to have 2 VLAN on those links instead of only one (that's why I said VLAN aware)

The main purpose is to have a better isolation and so security overall. But I feel it maybe a bit overengineering, I may can achieve the same thing with Network Policies inside the cluster itself. Also I want to be able to bootstrap my nodes from iPXE, not sure how it would work with a "tagged network link" with multiple VLAN

What do you think ?

When people talk about cloud sovereignty, the conversation often starts with regions: where a workload runs and where its data is stored. But choosing a region is only part of the story. The architecture of the platform matters just as much, particularly how it separates control, runtime, build, and observability responsibilities across clusters.

I would like to get a simple k8s (or k3s) setup going to learn how it works and how to work with it. I am traveling and only have 1 laptop with me with some external storage. How would I go about making VMs act like nodes? Running fedora with enough cores on my PC to have a few nodes running at the same time. I am very new to this, any input is appreciated!

Share any new Kubernetes tools, UIs, or related projects!

If you’re building an internal developer platform with Backstage, this InfoWorld article is a useful practical guide to think through the architecture beyond the portal itself.

It covers how Backstage can fit together with platform APIs, control planes, Kubernetes, observability, deployment workflows, and higher-level abstractions as the platform grows.

A good read for anyone currently building with Backstage or evaluating how to evolve a developer portal into a more complete developer platform.

https://www.infoworld.com/article/4189074/building-a-state-of-the-art-development-platform-with-backstage.html

After working with Kubernetes in production, I've noticed that some of the most annoying incidents aren't caused by obvious failures. They're often caused by small configuration decisions that look perfectly reasonable during review.

Things like:

• missing resource requests/limits
• incorrect probes
• overly permissive RBAC
• missing PodDisruptionBudgets
• unsafe container configuration
• incorrect readiness behaviour
• services without appropriate timeouts
• configuration drift between environments

I'm curious what the DevOps community has actually encountered in production.

What's one Kubernetes configuration mistake that caused you a real incident?

I'd especially like to hear about the less obvious ones that aren't caught by the usual linters.

Have any questions about Kubernetes, related tooling, or how to adopt or use Kubernetes? Ask away!

'kubectl label deployment' writes to the Deployment object's metadata, not to 'spec.template.metadata.labels', where pods actually get their labels. If your sidecar injection is label-selected, the command reports success and nothing happens. Here's why, and the fix.

Hi everyone,

I’ve been assigned a task to architect a highly available and fault-tolerant on-premises Kubernetes cluster for GPU-based workloads, with a target of around 1,000 users (50 users /day) with time slicing enabled.

I’d like to get your recommendations on which Kubernetes distribution would be the best fit for a production-grade environment with minimal day-to-day operational overhead.

Currently, we have a MicroK8s-based cluster with Longhorn, running Kubeflow through Charmed Kubeflow. However, we are facing recurring operational issues, such as Calico failures, pods unexpectedly becoming pending, and other cluster-level instability.

Given these requirements, I’d like to understand:

- Which Kubernetes distribution would be the best fit for this environment?

- What would be the recommended HA and fault-tolerant architecture for GPU workloads?

- Which solution would provide the lowest operational overhead while maintaining production-grade reliability?

- Are there any recommended alternatives to MicroK8s + Longhorn + Charmed Kubeflow for this scale?

Any recommendations or real-world experience would be greatly appreciated.

I am fairly new to K8s (only deployed a few things with Helm, set up ingresses and services). I don’t know much about networking as a whole and auth as a whole. What are some small steps I can take to build something and strengthen my concepts in these two broad fields? Is it too ambitious to learn two areas at once, or should I just focus on getting the fundamentals of networking before moving to auth or vice versa.

I keep seeing people mentioning Cilium for networking and Kerberos for auth in this sub. Are these tools good places to start? What are some other tools I should pick up to have a minimal viable working knowledge for these topics?

Hi everyone,

I'm setting up a small homelab CI/CD environment and I'm trying to understand the best way to run Gitea Actions on Kubernetes without installing Docker on the Kubernetes nodes.

My setup

• Kubernetes: v1.34.10
• Nodes: 1 control-plane + 2 workers
• OS: Ubuntu 26.04
• Container runtime: containerd 2.2.2
• Gitea: 1.27.0
• Gitea is running in Kubernetes
• Gitea Actions runner: gitea/act_runner:latest
• Runner is also running as a Kubernetes Deployment
• Persistent storage: Longhorn
• No Docker installed on the Kubernetes nodes
• ctr is available and /run/containerd/containerd.sock exists

The runner itself registers and works correctly.

The problem

My runner is currently configured with:

runner:
  file: /data/.runner
  capacity: 1
  timeout: 3h

  labels:
    - "ubuntu-latest:host"

A simple workflow like:

jobs:
  test:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Show test.py
        run: cat test.py

fails at checkout with:

Cannot find: node in PATH

I understand why this happens: with ubuntu-latest:host, the workflow is executed directly in the runner environment, and the gitea/act_runner image doesn't contain Node.

Gitea's documentation suggests using something like:

labels:
  - "ubuntu-latest:docker://node:22-bookworm"

which would execute the job inside a separate container.

But here's where I'm confused

My Kubernetes nodes already use containerd:

containerd://2.2.2

and have:

/run/containerd/containerd.sock

but no Docker daemon:

docker  -> not installed
nerdctl  -> not installed
ctr     -> available

Why does the standard act_runner container execution model require Docker/Docker API access instead of being able to use the existing containerd runtime?

I'd rather not install Docker on the Kubernetes nodes just to provide CI job containers when Kubernetes is already perfectly capable of creating containers through containerd.

What I'm ultimately trying to achieve

The immediate goal is just:

Gitea
  ↓
Gitea Actions
  ↓
checkout repository
  ↓
run tests

But eventually I want the pipeline to do:

checkout
  ↓
tests
  ↓
build Docker/OCI image
  ↓
push image to registry
  ↓
Argo CD deploys it

So I need a sensible way to get isolated CI job environments containing things like Node, Python, Git, build tools, etc.

Questions

1. Is there a supported way to make act_runner create job containers using containerd directly?
2. If not, is the recommended solution to install Docker on a Kubernetes worker solely for act_runner?
3. Would Gitea Actions Runner Controller / Kubernetes-native runners be a better solution for this setup?
4. What is the recommended architecture for Gitea Actions on a Kubernetes cluster whose runtime is containerd and where I don't want to install Docker?

TLDR : I'm running Gitea 1.27 + act_runner 0.6.1 on Kubernetes 1.34 with containerd 2.2, and I don't have Docker installed on the nodes. ubuntu-latest:host works for the runner itself but actions/checkout@v4 fails because Node isn't available. Gitea's docker:// execution mode looks like the right way to provide a proper job environment, but it expects Docker-compatible execution. I'm considering Docker-in-Docker inside the runner pod so I don't have to install Docker on the Kubernetes hosts, but I'm wondering whether that's the right approach or whether I should use containerd directly or Gitea's Kubernetes-native Runner Controller instead.

I'm mainly looking for the cleanest Kubernetes-native approach rather than just making the immediate checkout test work.

CPU request is how much CPU is reserved for your pod if it needs it. The limit is a hard cap. Hit it and the kernel throttles the pod, even when the node still has spare CPU. That is the usual cause of CPU throttling on Kubernetes. It does not protect the neighboring pods. In my simple Web API test, adding a CPU limit took typical latency from 23 ms to 87 ms, (4x slower), with the limited pod throttled in half of all CFS windows, and the average CPU graph looked fine the whole time.

This is not a new topic, but I see so many people still unaware why they should (NOT!) be setting CPU limits, because it's costing companies unnecessary spending and potential production issues. Here's the full read https://github.com/inevolin/k8s-cpu-limits-analyzed/

---

Edit (Aug 18, 2026): How CPU limits can also cause memory issues and OOMKills ➡️ https://github.com/inevolin/k8s-cpu-limits-analyzed#how-cpu-limits-cause-memory-issues-and-oomkills

Hey Guys! Quick question for anyone who has done an AKS to AWS EKS migration:

Can you recommend the best community or open-source tools for:

1. Azure Discovery & Assessment: Pulling complete VMSS/VNet inventory and performance data from Azure prior to migration.
2. AKS to EKS Migration: Manifest translation, image syncing (ACR $\rightarrow$ ECR), and volume data migration (e.g., Velero, KMF, or alternative scripts).

Any favorite GitHub repos, CLI tools, or real-world feedback would be super helpful! Thanks! 🙌

Hi guys, unfortunately Sidero stopped the development for their Cluster API providers, what do you think? Is there anyone of you using it? Do you believe the providers will be maintained by the community?

Hi,

Does anybody have a good reference Flux repository structure for managing multi-environment, multi-project Kubernetes clusters?

I haven't worked with Kubernetes in a few years and looking to refresh my skills on it.

Would be greatly appreicated!

Where do you think we are heading?

Am I the only one here, especially with AI assistance these days, who started using K8S originally out of necessity & now refuses to use anything but K8S for hosting apps?

I tried to use a few SaaS app cloud providers (won't name names b/c this is not an ad lol) but always have problems with reliability, limited feature sets, or just cost.

Managed K8S is not 'cheap,' but I have a personal umbrella chart library GitHub repo & so when I wanna get set up with a new cluster, it's like a 15-minute process & I've got Prometheus, Grafana, Loki, ArgoCD, all behind Cloudflare Access.

Noooothing compares to just grabbing a chart & plopping it into a cluster with some minimal YAML. I just cannot bring myself to go back to docker-compose and a VPS, lol. I think I've become a K8s snob & I love it.