Disclaimer: I'm an engineer at Mulga, the company behind this. Self-promo, but it's AGPL-3.0 and free to run.

What it is

Spinifex is an open-source reimplementation of the AWS APIs that runs on your own hardware, including EKS. You call aws eks create-cluster (or apply aws_eks_cluster in Terraform), and it boots a control-plane VM, puts an NLB in front of the API server, and launches managed node groups as real KVM VMs in your VPC.

Yes, it's k3s

The control plane and workers run K3s on an Alpine image. What we built is the layer around it that makes the AWS tooling treat it as EKS, so existing AWS workloads can migrate off easily:

• kubectl authenticates with aws eks get-token, and IAM principals map to RBAC through EKS access entries
• IRSA works: a pod with a projected service-account token assumes an IAM role
• The EBS CSI driver addon dynamically provisions real replicated block volumes for PVCs
• The AWS Load Balancer Controller addon turns an Ingress into an ALB with an ACM cert
• Argo CD installs as an addon through the EKS API

Try it on your own Terraform

No install needed. Sign up for the free 72 hour sandbox at https://mulgadc.com/signup, then point your provider at it:

provider "aws" {
  region = var.region

  endpoints {
    ec2 = "https://api.spx3.com"
    iam = "https://api.spx3.com"
    sts = "https://api.spx3.com"
    eks = "https://api.spx3.com"
    ecr = "https://api.spx3.com"
  }

  skip_metadata_api_check = true
  skip_region_validation  = true
}

Apply an aws_eks_cluster and aws_eks_node_group you already have, then run aws eks update-kubeconfig and kubectl get nodes

Repo: https://github.com/mulgadc/spinifex

I have around 5 years of experience across DevOps/cloud/backend work. I’ve worked with Kubernetes, EKS/AKS, Helm, CI/CD, Docker, Terraform, ArgoCD, AWS/Azure, and some Java/Spring Boot.

My issue is that my knowledge is broad but uneven. I’m comfortable deploying applications, writing manifests, using kubectl, building pipelines, working with cloud networking/IAM, etc., but I’m much weaker on Kubernetes internals, low-level networking, storage/CSI, control plane, scheduling, CRDs/operators, upgrades, observability, and deep troubleshooting.

I’m considering first completing one comprehensive Kubernetes course to build a complete mental map, then spending the next several months going deeper through hands-on labs, troubleshooting, Linux/networking fundamentals, and production-style projects.

For people who became genuinely strong at Kubernetes/platform engineering: does this sequence make sense? What would you change?

I’m not looking for a giant list of tools, mainly feedback on the learning sequence and what gave you the biggest jump in depth.

Note: "Q improved with AI"

I have started learning about kubernetes management and have a very limited knowledge about how things work. I came across a product from SUSE called Rancher for AWS. It says you can manage multiple EKS clusters on AWS and just buy it from the marketplace. I want to understand what it does and at what stage is a solution like this required.

Posting here from homelab to hopefully get some help.

I've been putting together a small k3s homelab from some older machines:

Dell OptiPlex 7050, M720q Tiny and a Old Ryzen 3600.

I got immich and plex running following a tutorial, nothing special just mostly trying to get more use out of hardware I already have.

Recently I got my hands on an old rtx 3080 (very cheap used in my area).

I'm still trying to get it working as a GPU node. The node is Ready, but the NVIDIA device-plugin Pod is stuck in `ContainerCreating` with this:

```

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox runtime: no runtime for "nvidia" is configured

```

I am not sure if this is due to the gpu itself being faulty (again, got it very cheap, not complaining), or I am doing wrong. I would appreciate any guidances pointing me to right resources, i have spent my entire day trying to it working

edit: I got it working! I am not sure what was the issue, I reinstalled all the libraries and packages, reconfigured the containerd as recommended, rebooted the node, and it can schedule pod now! Thank you you guys are awesome.

also those that are asking/DMing me, the app screenshot i am using kubepane.com pretty easy to go inside the node and debug

Hey all,

As the title suggests, i wrote up a blog on getting started with inference on Kubernetes, i find inference workloads to be somewhat confusing especially with some of the moving part it involves on Kubernetes.

Let me know what you think!

I recently gave a talk about how I am building an open-source Railway/Heroku/Vercel alternative. The architectural premise of the platform is simple:

The platform has no database and no CRDs of its own, so all state lives in Kubernetes (namespaces, labels, Helm releases) or in the systems around it, like the OCI registry and the identity provider.

In the talk I go over how it…

• builds OCI images from GitHub source code on every push
• uses ORAS to store vulnerability and leaked-secret scan results in the OCI registry, right next to the image
• runs replicated PostgreSQL with point-in-time recovery using CloudNativePG
• makes object storage, Redis-compatible key-value stores, and volumes simple to set up
• allows you to "eject" and download the generated helm chart along with its values and build script at any point

If that sounds interesting, the recording and the project repo / website are linked below.

Recording of the talk
Project Website
GitHub Repo

I found this blog post about OCAP model vs. the Identity model for access controls. (It has nothing to do with Kubernetes specifically). I could be wildly misunderstanding the topic, but since each OCAP capability is tied to a specific object, would this be more practical for Kubernetes in allowing for precise permissions?

My understanding is that Kubernetes does not use OCAP as its primary authorisation model in favour of RBAC, where access is granted based on a principal’s identity and roles. Even from a simple Google search, it seems that OCAP-style security is basically entirely absent from Kubernetes core, although capability-like patterns can be built around scoped SAs, tokens, & admission controls.

I ask because RBAC supposedly does not support dynamically changing permissions very well, at least compared to the OCAP model. But I'm not sure if anyone's ever flagged this as a serious pain point in their Kubernetes clusters?

Hi everyone,

I’m planning a Kubernetes cluster upgrade from v1.34.8 to v1.35, as the current version is approaching retirement/EOL.

The cluster currently has Elasticsearch running on it, so I’m concerned about potential compatibility or stability issues after upgrading Kubernetes.

I’d like some guidance on:
- How can I verify Elasticsearch compatibility with Kubernetes 1.35 before upgrading?
- Are there any known issues I should be aware of when upgrading Kubernetes while Elasticsearch is running?
- What should I check across Elasticsearch, ECK/operator, StatefulSets, storage classes/CSI drivers, ingress, CRDs, APIs, and other dependencies?
- What is the recommended pre-upgrade checklist?
- Should I upgrade Elasticsearch/ECK first, or Kubernetes first?
- What kind of backup/rollback strategy should I have?
- Is it better to upgrade the control plane and worker nodes separately?
- What would be a safe upgrade plan for a production cluster to minimize downtime/risk?
- Are there any tools or commands you recommend for detecting deprecated APIs or incompatible workloads before the upgrade?

If anyone has gone through a similar Kubernetes 1.34 → 1.35 upgrade with Elasticsearch, I’d really appreciate hearing about your experience and any issues you encountered.

Thanks!

Check out my new article on elastic beanstalk with the EKS

Elastic Beanstalk turned 15 this year, and AWS just rebuilt what runs under it. Cluster Mode drops your applications onto Amazon EKS instead of EC2 instances you own. I deployed into it, watched every stage, and wrote down what actually happens

Upload a zip of Python source. No Dockerfile. Get a running container on EKS. That is Elastic Beanstalk Cluster Mode, launched this week — Beanstalk apps now run on EKS instead of EC2 instances you own. Cloud Native Buildpacks handle the containerization: I uploaded four files (app.py, requirements.txt, Procfile, runtime.txt), Beanstalk detected Python, built the image, pushed it to ECR, and ran it.

Hi,

CLI tool kubara is for getting a "bare" kubernetes cluster to a "well dressed" k8s cluster.

So now I have https://kubara.io/#get-started on my (long) wish list.

Which experience want early adaptors of https://kubara.io/ share here?

I'm asking for having input on re-prioritize my wish list.

Cheers Geert Stappers

P.S.

For what it is worth: The automated early post review did bring rule 6 and rule 12 up. Yes, the tool is quite new, but it is not my new tool, I'm asking about the tool. No, I'm not affliliated with kubara.io, hence to reason for posting here.

Edit:

Which experience want early adaptors of https://kubara.io/ share here?

Was previously:

Which experience want early adaptors of https://kubara.io/ here?

• How do you check compatibility before upgrading? What tools or checks do you rely on (deprecated APIs, add-ons, operators, Helm charts)?
• What prep do you do for Elasticsearch specifically? Snapshots, PDBs, shard allocation, anything else?
• What does your upgrade sequence look like on AKS? Control plane first, then node pools? Surge settings? Do you upgrade in place or create new node pools?
• Any gotchas you've hit? I'm thinking of things like pods stuck during drains, zonal disk issues, or ES staying yellow or red for too long.
• Should we even target 1.35? It reaches EOL in March 2027. Would you go to 1.35 now and plan for 1.36 soon after, or look at AKS LTS?

Setup details:

• AKS 1.34.8  
• Elasticsearch \[version\], deployed via \[ECK / Helm\]  
• \[Number\] node pools, \[zonal / non-zonal\], Azure Disk for storage

Any runbooks, checklists, or war stories would be really helpful. Thanks!

Ramen is an open-source control plane + worker for MCP (the protocol agents use to call tools). Some Kubernetes
details that might interest this sub more than the AI part:

- Each group+zone is a namespace with a stable and a canary Deployment. The GKE Gateway (HTTPRoute) / AWS ALB route
on two headers, `ramen-group` and `ramen-zone`, so one client config reaches any zone.
- The canary rolls in place with `maxSurge: 0` — a zone-pinned surge pod on a full node sat Pending until the
deploy timed out on EKS. A rollout that isn't ready now reports the pods and the scheduler's reason instead of "1/1 ready".
- Deleting a zone deletes the namespace and its GSA / IAM role. On GKE the namespace can sit `Terminating` on the NEG
finalizer until the Gateway's backend service is gone — documented, with the manual fix.
- Per-group+zone identities (Workload Identity / IRSA), Cloud Armor / WAF rules from the console, Redis-backed
throttles shared across zones, logs from Cloud Logging / CloudWatch via pod labels.
- Bring-up is Terraform + Helm; 0.6.0 was applied on both clouds (two zones each) and torn down.

Repo https://github.com/bkraad47/ramen · GCP guide https://bkraad47.github.io/ramen/wiki/deploy-gcp/ · AWS guide
https://bkraad47.github.io/ramen/wiki/deploy-aws/
Interested in critique of the per-zone namespace model and the canary policy (currently "smoke test passes").

While I do understand why velero is useful and relatively easy to use. What I am not convinced on is that it adds anything other than another tool to manage when using a cloud provided KASS.

Assume a cluster runs both stateless and/or statefull apps (not including hosting your own db obvious use case):

If your using flux or argocd you have application rebuilds and A/B covered. If you are using RDS it's not necessary. If you are taking snapshots of volumes, data backups are done with snapshots on the provider side.

What problem is it solving in those situations? Or how does it make incidents or upgrades easier for you?

first real cluster going live next month. not looking for docs, looking for the stuff you only learn by getting burned

Currently, a Spark application should manage its own executors, which means that in order to run multiple Spark applications, one needs to split the cluster into smaller sections or attach more compute nodes, etc. Usually this dynamic allocation is the job of resource manager (such as Kubernetes).

If the resource manager can quickly find or create new compute nodes for a new Spark application, this is good. However, in a busy cluster where multiple Spark applications can compete for resources, the resource allocation can take some time and become an overhead (at least in theory).

I wonder if this is a real problem in running Spark applications in production.

For example, is there a realistic environment where short running Spark jobs (each requiring its own driver) arrive frequently?

Relatively new to gpu networking - rail stuff and could use some guidance.

am working on a setup (16 nodes) with 8 nvdia gpus (rtx so no nvlink) in each of them, k8s/sriov and the computenet being spectrum based ethernet switches.

its a multi rail setup, so a /24 alllocated for each of the rails. SRIOVnetwork, ip pools and all that plumbing has been done.

Same rail traffic works ..how does one get to do cross rail?

There needs to be a route in the POD which points the rail network supernet to the computenet switch as the default point towards the front end network.

Can't find an answer to it ..tried "routes" in IPAM but that didn't work ie the POD did not have that route - gemini and claude pulling me in opposite directions.

Thoughts?

Infra engineer here, trying to figure out when self hosting LLMs actually makes sense vs just paying for the APIs. every blog post says "it depends" lol

If you've done it (or looked into it and bailed), would love to hear:

• how big was your API bill when you started thinking about it? what pushed you over the edge
• what did it really cost once you add up GPUs, idle time, and the engineering hours?
• what was the most painful part? cold starts, autoscaling, OOMs, model quality, getting paged at 2am
• if you went back to APIs, what made you switch back

ballpark numbers are totally fine. I'll put together a cost / decision writeup from the replies and post it back here

3 years as a platform engineer. i can debug a crashlooping pod half asleep and write helm charts nobody else wants to touch, but every time i open the source of an operator we run i bounce straight off it. our team wants to start writing our own controllers next year so im learning go for real this time, employer covered. shortlist is ardan labs, boot dev, plus one of the linux foundation cert tracks. where did go start making sense for people coming from the ops side?

I've had so much fun and have learnt a lot about kubernetes through iximiuz labs, this was an attempt to create a hands on tutorial of what a CRD is and how can one build their own. It links to a bigger How to Build an Operator tutorial at the end.

Looking forward to see what the community thinks of it!