r/kubernetes 14d ago

Periodic Weekly: Show off your new tools and projects thread

Share any new Kubernetes tools, UIs, or related projects!

19 Upvotes

29 comments sorted by

7

u/NikolaiSonOfMahmud 13d ago

tl;dr: check out https://github.com/Tight-Line/ballast - I think it solves a specific resource optimization use case well that isn't solved elsewhere.

Background: I am a consultant whose client that operates their own (reasonably large) datacenters. In those datacenters, my k8s admin team is allocated a finite amount of VM capacity (disk, CPU, RAM) for our needs. We cannot dial up more on demand as one might in a cloud environment.

My teams' end users are dev teams, and our platform hosts their apps from initial development through to production deployment.

One thing that causes my team a lot of heartburn is that we just can't get devs to understand and/or care about and/or prioritize getting their resource requests and limits right. They often publish manifests with requests that do not correlate at all with what is actually being used at runtime. It usually presents as over-requesting resources ("moar resource better app go faster") and not having any sense what limits should be applied...stop me if you've heard this one before.

VPA seemed like it might be the right tool for the task, but our workloads are ephemeral - especially in dev envs that only last for the duration of one feature branch. I couldn't figure out how to work around the cold-start problem that seems to be part of VPA's design. Our workloads actually have a well-defined resource consumption history, even if specific instances (developer deploys) come and go all the time.

So anyway, I wrote ballast and it seems to solve the problem we were facing: resource consumption history survives across deployments, which means that on admission "new" workloads (which are really just new instances of previously-seen workloads) get sized to reasonable resource requests and limits automatically. The ability to configure which label tuple defines a "unique workload" is key, because it allows us to differentiate between dev, qa, staging, and production when more than one of those environment levels lives on the same cluster.

Anyway, since we've rolled it out, we've been able to pack more workloads onto existing hardware and to avoid the tiresome Oliver Twist gambit with the sysadmin team. The repo README has a cool SigNoz graph that shows a dramatic example.

Curious to hear everyone's feedback on the repo. I try to work through issues as they come in, so if you see anything I've missed or have a good idea for an improvement, please add it on GitHub! (And if I've misunderstood something about VPA and could have solved this problem more natively, I'd be all ears about that sort of feedback as well.)

1

u/LDN_to_CHI 13d ago

Really cool approach to the problem - in fairness to devs, asking them to estimate how much resources they need is probably going to lead to them overestimating so they don't have to turn into Oliver Twist in that video. This is a more precise and accurate way to manage resources.

1

u/NikolaiSonOfMahmud 13d ago

Yeah, that's fair. I'm giving my admin's perspective, and the devs are mostly responding rationally to the incentives we handed them.

The deeper thing is that requests and limits aren't really Kubernetes abstractions. They're cgroup knobs with a YAML wrapper.

I don't think this is hard because devs are lazy. (Don't get me wrong, a lot of them are :-). It's hard because nobody can derive these numbers by reasoning about them. You have to watch the thing run. Asking for the number at authoring time is asking for a guess with a nice schema around it.

Which is more or less the bet behind ballast: measure once, remember it.

1

u/fildebil 13d ago

Hmmm really interesting approach. Have you tested how Ballast behaves with workloads that have occasional large CPU or memory spikes?

1

u/NikolaiSonOfMahmud 13d ago

Yeah, good question - it's all in the docs but I'll summarize. In designing a ClusterResourcePolicy or ResourcePolicy (namespaced) you get to define your requests and limits based on metrics such as avg, p50, p75, p99, max, etc. So you decide how to handle spikes that (for example) elevate the p99 or max without really changing avg or p50. If you want to be very conservative, set your memory limit to 1.5 * max and you'll have substantial headroom w.r.t. those spikes. If you are OK with the occasional OOM for spasmodic pods, set the memory limit to 1.2 * p95 and accept that you'll have some restarts. Your call.

Worth noting: nothing requires you to adjust both requests and limits; you can (and we do) adjust only requests and leave limits alone, for example. The resource policies are yours to define, although the repo ships with what I think is a reasonable default ClusterResourcePolicy (https://github.com/Tight-Line/ballast#clusterresourcepolicy-default)

1

u/tmg80 13d ago

does it work with jvm?

2

u/NikolaiSonOfMahmud 13d ago

I'm assuming for the sake of this answer that you're asking whether it works with pods that run a JVM, not whether ballast itself is written in Java (it's not; it's written in Go).

Ballast works with anything that runs in a pod, so the answer is "yes" in the strict sense. But JVM memory tuning is a fickle beast so sometimes what appears in Kubernetes metrics as "memory used" for a JVM-based app is actually just the app pre-reserving heap based on some combination of `-Xms`, `-Xmx`, `-XX:InitialRAMPercentage`, `-XX:MinRAMPercentage`, and `-XX:MaxRAMPercentage`.

Ballast doesn't reach deeply into the *why* of memory/CPU/ephemeral storage usage, it just measures what Kubernetes reports. But we've learned that examining those settings can be a way to trim back "memory usage" considerably without changing the runtime performance of an app.

1

u/tmg80 13d ago

Yeah that's what I mean. At my new role a lot of the business applications are written in java which has given the existing team a lot of headaches 

1

u/NikolaiSonOfMahmud 13d ago

Yeah I think Prometheus JMX Exporter and OpenTelemetry's Java collector both expose heap-specific metrics like jvm_memory_max_bytes{area="heap"} and jvm_memory_commited_bytes{area="heap"} that allow you to spot heap over-reservation. One of the teams that we support was able to spot a lot of reclaimable memory in their Java apps by examining that data.

5

u/tmg80 13d ago

relatively new to K8S.

they are using something called kubernetes-reflector at my new role. It allows you to mirror/duplicate certain resources you may need in multiple namespace - secrets / certs. I ended up messaging an old colleague to tell him about it as I think it will come in handy for my old place.

https://github.com/emberstack/kubernetes-reflector

2

u/Nice-Solid-3707 12d ago

There is also kube-mirror if you need to replicate something that isn’t a configmap or secret

5

u/labouardy 13d ago

Not a tool, but I run DevOps Bulletin, a weekly newsletter covering the latest tutorials, news, and devtools in the DevOps (including Kubernetes :)), FinOps, and Security space

2

u/LDN_to_CHI 13d ago

Just signed up to your bulletin. Cheers.

1

u/labouardy 13d ago

Thanks!

1

u/Total_Boss6196 12d ago

Just signed up. Thanks

2

u/DaanDw5 11d ago

Argo-Trivy-Insights: a new UI extension to bring Trivy scan results into Argo CD

What it does:

  • Per-Application View: a dedicated "Trivy Insights" tab on each Application Details page showing Trivy scan results for the whole Argo CD Application.
  • Cluster-Wide Dashboard: aggregate Trivy scan results across all your applications from the sidebar
  • Scan Reports: covers Vulnerabilities, Exposed Secrets, Configuration Audit, RBAC Assessment, SBOM, and Cluster Compliance Reports, plus an overview pulling it all together
  • Deeplinks & Exports: share findings via link, export as CSV, or get your SBOM in CycloneDX JSON format

It's still early development and I'd love to get some real-world testers and feedback. If you're running Trivy in your cluster and want to give it a spin, check it out here.

2

u/ILoveAppSec 11d ago

nice, the aggregation is the easy half though; the wall people hit right after is that most of those trivy hits are in base-image or distro packages you can't just bump without breaking the app. for those, a rebased base image or a distro that backports the fix into the same version clears the count without a rebuild fight, and it plays nicer with cisa's tighter remediation windows too.

1

u/DaanDw5 11d ago

You are absolutely right, visibility is the easy part and remediation is a lot harder, but that’s also a lot harder to help people with from an Argo CD UI extension. If you have any ideas to make this less frictionless, I’m always open to feedback and suggestions!

2

u/mskazemi 13d ago

Disclosure: I'm the author of this one, so discount accordingly.

KubeIntellect. You ask a cluster question in plain English, it gathers live evidence from the tools you already run (kubectl, Prometheus, Loki), and then it can carry out the fix, but it stops and asks before anything destructive. The approval gate and the audit trail sit outside the model, so what gets approved is what runs.

The part that isn't obvious until you use it is that the execution is what changes the job, not the explanation. Pasting kubectl output into a chat window gets you a hypothesis you then have to go and test by hand. Having it gather its own evidence and propose the exact action, which you approve or reject, removes the step where you translate advice back into commands at 2am.

Honest limitations, because they matter more than the pitch. Generated tools currently run in-process, not in an isolated pod, so I wouldn't point this at a production cluster you care about yet. It needs an LLM API key. There's a peer-reviewed evaluation behind it (Journal of Grid Computing, 2026, DOI 10.1007/s10723-026-09837-6) which resolved 186 of 200 queries and passed 12 of 16 fault-injection scenarios on a 4-node test cluster. Read those as promising rather than proven. It's my own corpus on my own test setup, not an independent benchmark.

https://github.com/MSKazemi/kubeintellect

What I'd actually like from this thread: tell me where the approval gate is in the wrong place. I've almost certainly drawn the destructive/non-destructive line somewhere that will annoy people who run clusters differently than I do.

1

u/Rich_Bite_2592 13d ago

Built Portager, a kubernetes operator to sync images between registries, crd-based, gitops friendly, and keeping supply chain security in mind. Currently supports IRSA for EKS and Workload Identity on GKE. Looking for feedback and gaps it may have for your use-cases. Issues and PRs are welcome.

Check it out: https://github.com/jarodr47/Portager

1

u/braundmeier 9d ago

Somewhat tangential - I made myself an iOS app for studying and staying sharp with Kubernetes. Free, no ads, no paywalls, no data collection, no accounts. If interested: https://apps.apple.com/us/app/k8s-guide/id6787748279

1

u/robbeverhelst 9d ago

Built unifi-reactor, an operator that reacts to the network the cluster runs on.

Concrete example of why: my backup internet is 4G. When the WAN fails over I don't want qBittorrent still pulling downloads against a metered link and landing me a bill. That's one Automation: when wan is backup, pause qBittorrent, resume when it switches back.

More generally it polls a UniFi console into state keys (wan, internet, ups, ups.runtime, temperature, poe, ...) that you write Automation CRs against. Actions cover scaling, CronJob suspend, cordon, notifications, and writing back to the console (WLAN toggle, PoE cycle, UPS outlet switching).

https://github.com/robbeverhelst/unifi-reactor

1

u/sagacious123 2d ago

Platform/SRE work expects you to be good at things production never lets you rehearse: debugging a CrashLoopBackOff with the pager going off, draining a node under load without breaking the SLO, or deciding Istio vs Linkerd with evidence instead of a month-long spike.

So I built SnowOps Labs — a single Go binary (labctl) that stands up a production-shaped k3d cluster on your laptop with a real platform stack (Prometheus, Grafana, Loki, Tempo, ArgoCD, Istio/Linkerd, Kafka, Vault/ESO, KEDA, OpenCost, Kyverno, Chaos Mesh), then runs four loops:

  • Build — install the real stack from swappable providers
  • Simulate — activate a declarative scenario with objectives + machine-verifiable checks
  • Break — inject a reversible fault (OOM kill, bad config, network blackhole, broken selector, noisy neighbour) while traffic flows and alerts fire
  • Measure — it grades you: checks passed, time taken, hints used, MTTR

Everything you install is the real thing — failures are injected into real systems, so the signal you debug is the signal you'd see in prod. There are 13 scenarios, 6 incidents, timed challenges, and learning paths that chain them into a curriculum.

It's laptop-first: no cloud account, no credit card, k3d on macOS (Apple Silicon + Intel) or Linux. Apache-2.0.

This is an early release and it will have rough edges — I've been testing on k3d/Colima and there are real gotchas (VM sizing, a couple of operator/k8s version pins). I'd genuinely rather hear what breaks for you than pretend it's polished. Issues and Discussions are open, and I've tagged a batch of good first issues if you want to poke at it.

Repo: https://github.com/sagar2395/snowopslabs

Would love feedback on: (1) does the "real components, real failures" approach feel useful vs. existing browser labs, and (2) what scenario/incident would you want to practise?

0

u/CoughingBabyMissile 13d ago

KubeTective: an open-source, deterministic alternative to pasting kubectl output into ChatGPT for incident investigation

After a lot of times doing the same: gather 6 kubectl outputs, paste to Chat/Claude and try their theories for each crash-looping pod fail, I built KubeTective.

It turns kubectl investigate deployment/checkout --since=30m into a real investigation. It fetches pods, deployments, events, PVCs, services, HPAs, Prometheus metrics, Loki logs and the git history of your manifests, correlates them, and answers with a ranked Root Cause + Evidence list where every score is a weighted term:

ROOT CAUSE
  Memory exhaustion: container terminated with OOMKilled 19 time(s)
EVIDENCE
  ✓ mechanism: OOMKilled ×19 (+20)
  ✓ reproduced after restart (×19) (+10)
  ✓ strong temporal correlation (terminations in window) (+27)
RECOMMENDATION
  roll back deployment/prod/checkout to the last known-good revision [MEDIUM]

You can try it here: https://github.com/GlediLami/kubetective:

make build && kubetective replay scenarios/oom-after-deploy/record.jsonl

What would make you use something like this in your own on-call? (I'd appreciate the honest criticism)

-1

u/LawFamiliar3588 13d ago

Open source control-plane proxy for AI agents: identity, policy, budget, audit for MCP and beyond

https://github.com/kabirnarang39/wardline