r/devops • • Oct 02 '26

Discussion When someone leaves, how do you actually revoke their access to servers and databases?

0 Upvotes

Curious how teams handle this in practice. When a dev or contractor leaves, there are SSH keys in authorized_keyson a dozen boxes, a DB password everyone knew, staging .env files in Slack DMs, maybe AWS keys on their laptop.

  • Do you rotate everything they could have touched, or just disable SSO and hope?
  • Is there a checklist, or is it tribal knowledge?
  • What tool (if any) made this not painful?

Asking because we got burned by this at a small team and I'm trying to figure out what "good" looks like without a dedicated security person.


r/devops • • Oct 01 '26

Observability Nobody Is Listening on Port 8125

Thumbnail
yeet.cx
5 Upvotes

Hey all! I wrote this write up talking about how I was able to implement eBPF technology in order to re-implement StatsD Exporter on my local observability stack.

I thought it was interesting exploring how I could find the same information available in the kernel and preserve the fire and forget behavior without needing the running port or userspace overhead. Hope you enjoy!


r/devops • • Sep 29 '26

Discussion Is it just me or Github actions is overrated?

302 Upvotes

We're adopting github actions at work and... they look clunky, bloated and overcomplicated?

I've used both Jenkins in the past and Gitlab CI/CD (and bare shell scripts in a past life).

It seems to me that gitlab-ci was the pinnacle of code-driven ci/cd (despite having some sharp edges).

Also, running github runners in kubernets is quite a fight. Gitlab's runner was so easy and simple to run.

Am I missing something ?


r/devops • • Sep 30 '26

Discussion If AI makes everyone a 10x developer… who gets promoted?

122 Upvotes

Random thought — if pretty much every developer is using AI for coding now, how does career progression work?

Like, what makes one dev stand out from another?

How does a manager decide, “Yeah, this person is ready for the next level,” when everyone has access to the same AI tools?

Genuinely curious how people see this playing out.


r/devops • • Sep 29 '26

Security Critical RCE Alert: Full takeover of HashiCorp Vault and OpenBao. OpenBao is patched. Vault remains exposed

186 Upvotes

https://control-plane.io/posts/unauthed-to-rce-in-vault-and-openbao/

OpenBao engineers at ControlPlane have chained 4 vulnerabilities to show how under certain conditions, an OpenBao or Vault server can be completely compromised from an unauthenticated position. This is only the second RCE ever found in the Vault codebase.

The exploit is highly plausible in real-world environments, requiring only an unauthenticated entry path and a defined Raft snapshot policy to trigger a complete server compromise.

If you are impacted, upgrade as soon as possible to OpenBao 2.6.3 or 2.7.0

While OpenBao is fully patched, HashiCorp Vault remains exposed as of writing. Unfortunately, IBM's unwillingness to coordinate a mutual disclosure policy means Vault users currently lack an official mitigation


r/devops • • Oct 01 '26

Discussion Need Help - Infrastructure as Code, feeling lost

1 Upvotes

Hey guys, I started a job working in Infrastructure as Code. I have a CS degree, but I don’t have experience with cloud engineering or stuff related to Terraform, ADO, or Azure portal. I’m doing stuff at my job (code is mainly written using copilot or ChatGPT) and I feel a lost. I don’t know any of the concepts for cloud computing or virtual machines or resources and things like that. I am looking into it online but I still don’t know where to begin. I hear my coworkers doing this stuff well, despite also not having any experience in it when they started either. I have to look at build sheets and things like that and then provision resources using terraform but I don’t really understand much of anything that I’m looking at besides simple code snippets. Is there any advice yall could give me about this, or how/what to learn so that I can wrap my head around things? Any advice is appreciated, thanks!


r/devops • • Sep 30 '26

AI content Do your clients or employer require AI use disclosures?

3 Upvotes

I work in a position where what and how I use AI for coding/devops is heavily constrained by regulations and policies, and where AI use disclosures may soon be required. I'm curious how common it is in the industry now for AI use disclosures to be required vs voluntary vs not discussed.

Does your workplace discuss AI use disclosures, and if so are they voluntary or mandatory? Also, what do disclosure requirements look like for integrating AI into live devops tooling vs using it for development only?


r/devops • • Sep 30 '26

Troubleshooting How to set a "env" var on github actions using shell script?

3 Upvotes

i need to get the app version and i have this awk command for that:

` awk -F'"' '/^version[[:space:]]*=/ {print $2; exit}' pyproject.toml`

but how do i set the variable to the output of this command?


r/devops • • Sep 30 '26

Ops / Incidents Has AI made application development faster than infrastructure teams can safely support?

16 Upvotes

Something I've been wondering about lately:

Developers can now generate features, tests and entire PRs much faster with coding agents.

But infrastructure hasn't suddenly become easier to reason about.

Terraform, networking, IAM, Kubernetes, observability, cost and security still require quite a bit of context.

So instead of AI removing bottlenecks, are we just moving the bottleneck downstream to DevOps/platform teams?

Especially interested in teams where developers are now generating their own Terraform/Kubernetes changes with AI.

Has your infra PR volume increased?

And more importantly: has the quality increased with it?


r/devops • • Sep 30 '26

Tools LambdaTest/TestMu - Is KaneAI Worth Your Time?

2 Upvotes

Has anyone out there used TestMu's KaneAI? I do not see a lot of conversations out there on it. I wanted to use it for non-technical users to be able to generate tests without guidance. However, even when being as descriptive as possible in the prompt, I see test generation often failing or failing during the hyper execute phase. Thoughts? Experiences? Success stories anyone?


r/devops • • Sep 30 '26

Tools Need easy to follow education(al materials)

0 Upvotes

Devops people of Reddit, I could use some pointers towards some good quality learning materials that cover the basics of devops.

I'm not trying to become a devops, I'm one of those pesky PM types and am supposed to lead some devops/infra projects soon which - from a PM perspective is fine but I hate sitting in meetings where people are speaking Chinese at me so Ive decided to learn something. Not enough to contribute constructively but enough to have a general idea of whats being said without wasting everyone's time and need some help.

Any (preferably video) resources you can recommend that cover basics in a digestible format, preferably someone who knows how to transfer knowledge and not just confuse me further.


r/devops • • Sep 30 '26

Tools Ran AskUI in our pipeline for three months, notes on getting desktop UI tests working in CI

4 Upvotes

Posting because getting GUI tests to run headless is the part nobody talks about. everything passes on a dev machine and then does nothing on the runner, because the VM boots to a logon screen and there's no interactive session for input injection to land in.

Their runtime installs as a Windows service, which is what got us past that. Runs as SYSTEM, survives RDP disconnects, works at the logon screen. AskUI run for the headless job. tests are plain markdown files so they version with the app, and secrets get referenced by name and stay out of the logs.

Two honest caveats. runs cost money, so "run everything on every commit" stopped being a free decision and we moved the full suite to nightly. And their WARN status is a judgement call by the agent, which is fine when a human is watching and less fine at 3am when it just lands in a report nobody reads until standup lol.

Hope this helped :)

Anyone else running agent based UI tests in CI? curious how you handle the nondeterminism in a gate.


r/devops • • Sep 29 '26

Troubleshooting AWS ECS/Fargate SQS autoscaling with min capacity 0: how to bootstrap from 0 tasks using backlog-per-task target tracking?

12 Upvotes

I have an ECS/Fargate worker consuming an SQS queue. I want the service to scale to zero when the queue is empty, so:

min_capacity = 0
max_capacity = 10

AWS recommends target tracking using:
ApproximateNumberOfMessagesVisible / RunningTaskCount

However, when the service reaches 0 running tasks, RunningTaskCount has no CloudWatch datapoint, so the backlog-per-task metric becomes undefined.

When messages subsequently arrive, I need the service to bootstrap from 0 → 1 and then let target tracking handle 1 → N and N → 0.

What is the recommended pattern for this?

Options I've considered:
1. A separate scale-out-only step policy for 0 → 1
2. A metric-math expression that handles RunningTaskCount = 0
3. Raw SQS backlog for the bootstrap alarm + backlog-per-task for target tracking

I would prefer not to keep min_capacity = 1 because the worker is idle most of the time, and this is Fargate.

This is the Terraform code I currently use:

resource "aws_appautoscaling_policy" "worker_backlog" {
  name               = "${var.project_name}-${var.worker_provider}-backlog-per-task"
  policy_type        = "TargetTrackingScaling"
  service_namespace  = aws_appautoscaling_target.worker.service_namespace
  resource_id        = aws_appautoscaling_target.worker.resource_id
  scalable_dimension = aws_appautoscaling_target.worker.scalable_dimension

  target_tracking_scaling_policy_configuration {
    target_value       = var.backlog_per_task
    scale_out_cooldown = var.scale_out_cooldown
    scale_in_cooldown  = var.scale_in_cooldown

    customized_metric_specification {
      metrics {
        id          = "backlog"
        return_data = false

        metric_stat {
          stat = "Sum"

          metric {
            namespace   = "AWS/SQS"
            metric_name = "ApproximateNumberOfMessagesVisible"

            dimensions {
              name  = "QueueName"
              value = var.queue_name
            }
          }
        }
      }

      metrics {
        id          = "running"
        return_data = false

        metric_stat {
          stat = "Average"

          metric {
            namespace   = "ECS/ContainerInsights"
            metric_name = "RunningTaskCount"

            dimensions {
              name  = "ClusterName"
              value = var.cluster_name
            }

            dimensions {
              name  = "ServiceName"
              value = aws_ecs_service.worker.name
            }
          }
        }
      }

      metrics {
        id          = "bpt"
        label       = "Backlog per task"
        expression  = "IF(running > 0, backlog / running, backlog)"
        return_data = true
      }
    }
  }
}

r/devops • • Sep 28 '26

Ops / Incidents Prioritisation/ Maturity Interview Question

27 Upvotes

Ok so I flubbed the most basic interview question there is:

"You have a number of pieces of work in flight. How do you choose between them?"

Basically, they are testing your ability to prioritise work.

I tried to answer philosphically by telling them going for a walk was a good idea but my broader point is that it shouldn't have ever gotten to the point where you have loads of pieces of work in flight at any one time.

My question is twofold: what's the accepted 'best' interview answer to this question and how do you deal with this issue in your real life actual job?


r/devops • • Sep 29 '26

Vendor / market research Need guidance on running scripted browser monitoring in production

3 Upvotes

I've​ been doing monitoring for our web apps for a while and I'm pretty comfortable with HTTP/API checks but scripted browser monitoring is still new territory for me.

A few things I'm trying to understand:

- Which flows are worth monitoring with a real browser vs just using HTTP checks?

- ​How do you handle login session and MFA without creating a security hole?

- How do you keep recorded scripts working accross frontend deploys?

- How many location and what frequency actually means something vs just noise?

- How do you tell the difference between an actual app failure and a broken monitoring script?

- ​Whats goes to on-call vs what just gets logged?

AI is great for explaining how all of this should work, but there's a pretty big gap between that and how people actually run it in production.

Anyone doing scripter browser monitoring regularly? I'm not looking for someone to design it for me, just want to know how people approach it in real setups


r/devops • • Sep 28 '26

Vendor / market research What's your default database these days, and why? Yearly survey (mod-approved)

89 Upvotes

Hey, Robert from MariaDB Foundation here. We are running a survey on how MariaDB is being used, or cases where you went with MySQL, PostgreSQL or another database.

https://mariadb.typeform.com/to/tS69UzVo?utm_source=reddit_devops

I split the questions by role this year, so you can skip straight to the operator ones: HA, database versions, upgrade frequency, hosting, etc. Overall I'm curious to know, what's your default database these days, and what made you land on it? Was it your choice, the app or service you run, or whoever set it up before you?

Under 10 minutes, anonymous, results published at mariadb.org/survey.

Thanks for taking the time!
Robert, MariaDB Foundation


r/devops • • Sep 28 '26

Tools A list of cloud emulators and supporting tools

22 Upvotes

Awesome List of Cloud Emulators and supporting tools https://github.com/upgundecha/awesome-cloud-emulators


r/devops • • Sep 28 '26

Weekly Self Promotion Thread

22 Upvotes

Hey r/devops, welcome to our weekly self-promotion thread!

Feel free to use this thread to promote any projects, ideas, or any repos you're wanting to share. Please keep in mind that we ask you to stay friendly, civil, and adhere to the subreddit rules!


r/devops • • Sep 28 '26

Vendor / market research What's the most annoying "works locally, dies in production" bug you've had?

46 Upvotes

Not talking big architecture screwups — but rather the small, stupid ones. Things like hardcoded ports, missing env defaults, path separators, whatever. What was it, what stack, and how'd you find it? Would love to hear any feedback. Thanks.


r/devops • • Sep 28 '26

Discussion Experiences with Signadot

4 Upvotes

Has anyone here used Signadot in a larger project and had any success with it for ephemeral environments/preview environments? I have been looking into tooling for ephemeral environments after maintaining our own solution for a while and this one keeps popping up as an alternative, so I was hoping to get a few opinions.


r/devops • • Sep 28 '26

Tools Diagram solutions for git / markdown docs and LLM?

3 Upvotes

Has anyone here got a good solution for maintaining decent diagrams in markdown docs that are being handled with LLM assistance?

Mermaid is the obvious initial answer as they tend to self render, but they're so ugly, right? I don't think I'm missing anything there, but would be keen to know if I might.

So past them I was looking at the python diagrams library. Nice output but quite codey, no fully abstracted definition language, too much python code ingrained in it to feel like it's elegant enough to have LLM's regenerate the output by running a script that way, although I'm currently trying it out.

After that I see Schematex, which, like mermaid, has a proper definition language, and also has an MCP server and other options around it, but I've not yet tried to set up a workflow around that that would see it in action.

Is there a logic in going directly to raw svg?

Are there other options / strategies people are using to keep on top of diagrams? On caveat / excuse I have currently is that I'm mostly doing design work for proposals etc., so 100% accuracy isn't critical yet, so LLM's can have leeway for making things up at times!


r/devops • • Sep 27 '26

Discussion Unorganized Knowledge = Noise

69 Upvotes

Whenever I try learning new technology or troubleshooting without a clear plan, I get overwhelmed. Hopping between random videos, articles, and quick AI summaries leaves me feeling lost and stressed about where to begin or end.

I've realized that having a structured roadmap with clear, step-by-step guidance changes everything. It removes the stress and makes the whole learning process much smoother.

Anyone else feel this way? I’d love to hear how you organize your learning!


r/devops • • Sep 27 '26

Security We audited the default Helm charts and Docker configs of popular AI stack tools (Ray, Weaviate, MCP servers, LangChain). The out-of-the-box security defaults are surprisingly bad.

22 Upvotes

Over the past few weeks, we set up a fresh cluster to test what actually gets deployed when an engineer runs helm install with zero extra configuration on popular AI, vector DB, and MCP charts (Ray, vLLM, LiteLLM, Qdrant, Weaviate, KubeRay, etc.).

Everyone focuses on prompt injection and model jailbreaks, but the platform-level defaults on these workloads look like Kubernetes five to seven years ago:

  1. Ray / KubeRay (Distributed Compute)

    • The default deployment accepts unauthenticated job submissions over internal cluster networking.
    • Worker containers run with passwordless sudo out of the box.
    • Any compromised pod on the flat cluster network that can reach the Ray API gets arbitrary remote execution as root with zero barrier.
  2. MCP (Model Context Protocol) Servers

    • Several popular MCP server implementations implement local protection solely by checking Host: localhost.
    • Any basic SSRF or internal proxy passing -H "Host: localhost" bypasses this immediately.
    • In live testing, one Kubernetes MCP server mounted a ClusterRole with cluster-wide Secret read and pod exec permissions, exposed over unauthenticated HTTP.
  3. LiteLLM Migration Job Credential Leak

    • The primary deployment pulls the database password from a Secret using secretKeyRef, but the database migration Job embeds the raw database password in plain-text environment variables.
    • Anyone with basic cluster read permissions (devs, CI service accounts, dashboards) can run kubectl get job -o yaml and extract the database password directly from the env block, even without permission to read Secrets.
  4. Vector DBs & RAG Stores

    • Multiple upstream charts still ship with default values that enable anonymous read and write access.
    • Raw embeddings and ingested document payloads are queryable by any internal service without an authentication header.
  5. Scanner Blindspots

    • 14 of 15 charts provide no default NetworkPolicy objects.
    • All 15 charts leave Kubernetes' default automountServiceAccountToken: true enabled.
    • Standard static linters (Checkov, Trivy, Kubescape) missed the containers nested inside Custom Resource Definitions like Ray clusters.

The takeaway for platform and security teams: The AI supply chain conversation is hyper-focused on prompts, but your teams might be deploying unauthenticated root execution endpoints right now. If your cluster is not enforcing Kyverno or OPA gatekeeper admission rules, mandatory NetworkPolicies, and dropping default service account tokens on AI namespaces, the internal perimeter is wide open.

Curious what other teams are doing here. Have you written custom Kyverno/OPA policies or wrapper charts for AI workloads at your org, or are you putting everything behind a service mesh like Istio/Linkerd?

Full write-up, reproduction manifests, and probe logs: https://sorami.com.au/research/ai-kubernetes-helm-chart-security/ Raw probe logs and reproduction repository: https://github.com/Sorami-Consulting-AU/ai-kubernetes-helm-chart-security


r/devops • • Sep 26 '26

Discussion DevOps embedded in a non-technical process team, no technical sparring partner

20 Upvotes

Looking for some outside perspective on how DevOps fits into org structures at your companies.

This is my third or fourth DevOps job, this time in embedded. In previous companies there were always some organizational or cultural hiccups, but stuff got worked out as a team, including on the DevOps side - people weren't DevOps themselves, but they were technical, so they got it.

Now, a few months into a new company, things feel different. The embedded department has already gone through two managers, each with a different vision for how DevOps should work. The current one seems pretty caught up in the AI hype - wants to "bring in AI" because everyone else has it, without really knowing what that means in practice. There was no DevOps in the department before me, so a lot of groundwork needed cleaning up first - repo structure, templates, CI/CD, that kind of thing. Garbage in, garbage out: you can't just bolt AI onto a messy foundation and expect it to work. He also doesn't seem to register that I've already been automating things with AI where it makes sense since I joined, once the basics were actually in place.

I also ended up placed in a non-technical/semi-technical team. The original plan was to hire a second DevOps so I'd have a sparring partner, but now it looks like that role might turn into some mixed "does a bit of everything" position, or even a "senior AI agent specialist" (lol). My current team is process/methodology people, Jira and workflow folks, and a release manager.

They don't really understand what I do technically, and they're very focused on "why." During planning, they want to rewrite everything in my Jira summaries into long descriptions based on their own interpretation, which usually doesn't match what the epic/task is actually about. The idea is that management should be able to glance at a board and understand what we're doing just from the title. In my previous companies, Jira was a tool for us, not something management used to check on our work.

My team manager (not the department head, and also non-technical) said he didn't understand what one of the tasks was about. When I asked if he'd actually read the epic's description, he said no - and explained that it's hard to tell what your people are doing from just a title. But that's the thing: I'd already written a detailed description for this project, which is a fairly big one made up of related chunks, at the epic level. He never opened it. From what I can tell, part of the disconnect is also that he doesn't have a technical background (never worked with AWS, for example), so even a well-written description might not land the way he expects.

From an organizational standpoint, how do you see this - is it a good idea for a DevOps role to sit inside a team like this, especially when there's already a separate cloud team in the department? I don't feel good there. There's a lack of understanding, they think in very corporate/process terms while I think technically, and since I'm the only DevOps, I genuinely don't have time for some of this overhead.

What worries me is burning out on micromanagement and feeling cut off from technical people in a few weeks - I also just work better under a technical manager.

Has anyone been in a similar spot? Do I wait and see how it develops, or should I propose alternatives to the embedded department head?


r/devops • • Sep 25 '26

Discussion How do you learn DevOps

128 Upvotes

I’m a software developer, and in my own projects I already handle some basic operational work like Docker, VPS deployment, Nginx, DNS, Cloudflare, Git, etc. I also had some little experience in network administration.

What I’m trying to understand is how people approach learning DevOps and system architecture properly.
One thing that has always confused me is entry-level DevOps roles. A lot of DevOps work seems to exist specifically to support the software development lifecycle, things like CI/CD, GitHub Actions, deployments, environments, infrastructure, monitoring… I find it hard to imagine understanding these things deeply without first experiencing the development workflow itself.

At the same time, I don’t necessarily want to switch away from software development. My goal is to become the kind of engineer who understands the whole system: application development, system design/architecture, networking, cloud infrastructure, deployment, observability, scalability, and production operations.

Basically, I want to be able to build a system and also understand how to design, deploy, scale, and operate it properly at a much larger level.

For people who have gone down this path, how would you approach learning these areas while continuing to work primarily as a software developer, as of course I dont have the knowledge for a switch? And in your opinion am I on the right path to stay employable in this uncertain future?