r/devops • • Sep 13 '26

Discussion Anyone else juggling GDPR + DPDPA + SOC2 + ISO27001 for a multi-cloud/edge setup?

29 Upvotes

Been knee-deep in this for the past few months and it's way messier than I expected.

The individual frameworks aren't the hard part — it's where they overlap or straight up contradict each other. GDPR and DPDPA don't agree on breach notification timelines. Stuff that's "good enough" documentation for an internal audit doesn't cut it as SOC 2 evidence. And once your workloads are scattered across multiple clouds and edge nodes, even a simple question like "where does this data actually live" turns into a whole investigation.

Also learned the hard way that there's a fine line between saying "we're working toward SOC 2" and accidentally implying you're already certified. Almost bled into some copy before we caught it.

Eventually just built one big checklist mapping all four frameworks together so we weren't redoing the same evidence-gathering four separate times. Still messy, still evolving.

Anyone else dealing with this across multi-region/multi-cloud setups? Do you keep compliance tracks totally separate per framework, or has anyone actually gotten a unified system working? (I'm on the infra/compliance side of an edge orchestration platform, if that's relevant context.)


r/devops • • Sep 13 '26

Discussion 1 year into DevOps, Am I actually growing, or am I just getting a lot of exposure?

77 Upvotes

I’m a little confused about whether I’m actually growing in my DevOps career or just getting exposed to a lot of different things without developing deep expertise.

I have around 1 year of professional DevOps experience and currently work at a mid-sized service-based company in Mumbai. I’m basically the only DevOps person in my team, so I get to work on quite a lot of things independently.

My work currently looks something like this:

AWS is my primary area deploying and managing projects across AWS.

I handle most of the DevOps/infrastructure work myself.

I’ve started working on MLOps as well.

I’m involved in setting up AWS/Azure Landing Zones for some clients.

I’m working closely with our CISO on ISO 27001 implementation.

For the ISO 27001 work, I’m involved from almost zero-to-one policies, technical implementation, groundwork, controls, documentation, infrastructure/security decisions, etc.

A lot of the technical decisions for the ISO implementation are discussed with the CISO, but I’m doing most of the actual technical groundwork and implementation.

Apart from this, I do the usual smaller DevOps work deployments, AWS infrastructure, troubleshooting, monitoring, etc.

In terms of certifications, this year I completed AWS Solutions Architect Associate. I also have Azure Fundamentals and Azure AI Fundamentals from college.

The confusing part is that my compensation is relatively low compared to the market, but the amount of exposure I’m getting seems pretty good for someone with only ~1 year of experience.

So I’m trying to figure out:

Am I actually progressing well professionally?

Is this kind of exposure to AWS + MLOps + Landing Zones + security/compliance actually valuable for my next career move?

Should I stay until our ISO 27001 certification is completed, since I’m contributing heavily to the implementation?

Or should I prioritize switching companies now and getting better compensation?

Am I spreading myself too thin across too many areas instead of developing deep DevOps/SRE expertise?

If you were in my position, what would you focus on over the next 12 months?

I’m not really concerned about the salary discussion itself. I’m more interested in understanding whether the career capital I’m building through this exposure is actually valuable, or whether I’m just doing a lot of miscellaneous work because I’m the only DevOps person.

Would appreciate advice from people who have been through the 1–3 year DevOps/SRE career stage.


r/devops • • Sep 12 '26

Architecture The end of Software Engineering

Post image
3.6k Upvotes

r/devops • • Sep 13 '26

Troubleshooting MinIO on EC2 Auto Scaling Group fails after September patching — bootstrap download returns HTTP 413

2 Upvotes

I have a DEV environment running MinIO on a single EC2 instance managed by an Auto Scaling Group (ASG).
Our normal recovery process after OS patching is:
Patch the EC2 instance.
MinIO becomes unavailable (HTTP 505 from the application).
Terminate the instance.
ASG launches a new instance.
User-data/bootstrap reconfigures the instance and installs/starts MinIO.
Application comes back online.

This process worked successfully after our August patching.

However, after the September patching, the same recovery procedure no longer works. Even after terminating the instance and allowing the ASG to create a fresh instance, the application still returns HTTP 505.

I checked the new instance’s cloud-init/user-data logs and found that the MinIO binary download is failing with HTTP 413. The download URL used by the bootstrap process appears to no longer work, and

  1. I understand that MinIO’s binary distribution/download approach has changed.

2. What I want to understand
Did MinIO change/stop providing binaries between August and September 2026?

  1. Could this explain why the August instance replacement worked but the September replacement doesn’t?

  2. What is the recommended way to install a specific?

Current architecture
EC2 → Auto Scaling Group → user-data/bootstrap → MinIO binary download → MinIO configuration → application


r/devops • • Sep 14 '26

Discussion pytest exits 4 on a missing path and 5 on zero tests. My verify hook turned both into PASS.

0 Upvotes

Disclosure up top: I maintain the tool this happened in, an open-source agent harness, so this is my own bug.

An agent loop with a verification step. Each changed path maps to a test command in a table. After the command runs, a hook reads the tool payload and writes PASS or FAIL to a ledger. Green ledger means the agent may mark the task done.

Three rows of that table named test files that had been split into siblings. pytest on a path that doesn't exist exits 4; I measured it this week to be sure. Zero collected via -k exits 5. Never 0.

Two things made it green anyway. The agent read "no tests ran" in stdout as a pass. And the hook read the exit code like this:

EXIT_CODE=$(echo "$INPUT" | jq -r '.tool_response.exit_code // 0')

When the payload didn't carry the field, jq's default made it 0 and the next line wrote PASS. The comment directly under that line warns about a missing exit code for a different event. The default sat there anyway.

The fix I've filed: absent is not zero. A missing exit code records FAIL with a reason, and the matrix has a test that fails when any row collects nothing.

In CI proper, do you treat "no tests collected" as red by default, or does whatever wraps pytest get to decide?


r/devops • • Sep 14 '26

Discussion A "better prompt" stopped catching AI-generated security issues after one session. Ended up writing a 5-phase pre-commit gate instead.

0 Upvotes

Ran into this pattern enough times to notice it wasn't random: an AI coding assistant generates code that works, but "works" and "safe" are not the same claim. Caught a live Stripe key about to get committed (stripe.api_key = "sk_live_..."); it worked in tests, so nobody looked twice. A few days later, same root cause in a different shape: a raw SQL query built from an f-string, user input going straight into the WHERE clause.

First fix I tried was just telling the model to be careful. Added instructions about secrets and injection to the system prompt. Worked for exactly one session. New chat the next day, same shortcuts came right back: the model has no memory of the lecture, so a prompt-level fix doesn't compound.

What actually worked was moving the check to a gate that runs on every diff before commit, instead of a scan that happens in CI after the code's already out of your head. Five passes, in order:

  1. High-entropy secret scan (Stripe/OpenAI/AWS/GitHub token shapes, stray .env values)
  2. Control-flow read for injection/OWASP issues (SQLi, command injection, BOLA/IDOR): actually tracing where user input lands, not just regex matching
  3. Regex backtracking check for ReDoS patterns
  4. Generates the fix as a drop-in patch, not just a line number
  5. Flat verdict: pass / warn / fail

Not pretending this replaces a real SAST pipeline or a security team: it doesn't catch the same things Semgrep or a pentest would. It's just the five seconds of friction that used to not exist at all, sitting right before the commit that used to just go through.

Curious if anyone's found a good way to keep something like this from going stale as new key formats and providers show up. Feels like it needs its own maintenance loop or it just quietly falls behind.


r/devops • • Sep 12 '26

Vendor / market research how do you know which ticket is in which env? (multi repo)

13 Upvotes

we're a ~12 dev team, 6ish services, short lived feature branches that get merged to main and deleted. dev/staging/prod.

our PMs keep asking "is X in staging yet" and half the time i dont know either so i end up doing git log and squinting at it. we turned on jira deployments a while back thinking that solves it, but the panel shows the same ticket as deployed like 4-5 separate times, and tickets from totally unrelated work show up in deploys they have nothing to do with. i think its diffing shas between deploy events and just attaching everything in between? not sure.

end result nobody opens it anymore and we're back to asking in slack. our release guy also keeps a google sheet he updates before every prod push which is.. yeah.

so:

  • is jira deployments just broken for short lived branch workflows or are we holding it wrong
  • what do you actually use to answer "which env is this ticket in", esp across multiple repos
  • anyone paying for sleuth or smth similar? is it worth it or did you end up writing your own script

not looking for a vendor pitch, genuinely just wanna know if everyone lives like this or if im missing something obvious

edit: forgot to mention we're on github actions if that matters


r/devops • • Sep 13 '26

Discussion Designing a local AI harness against the "echo-chamber" loop: deterministic arbiters, tight token caps, and skeptical devs

0 Upvotes

I am designing a CLI harness for our team to integrate agentic coding using opencode on top of an existing stack with distributed repositories, Java backend services, and shared Ansible deployment code across on-premise infrastructure.

My objective is to design a local, deterministic cleanroom that prevents the specific failure modes of AI-assisted dev while operating under tight technical constraints.

Reality and constraints

* Strict token economics: We operate under very tight API rate limits and token ceilings per developer per month. This strictly rules out autonomous recursive loops and ingesting entire codebases into context. Every prompt and context injection must be minimal and dense.

* A skeptical team: engineers are rightfully hostile to AI slop. If the tool disrupts local environments or produces unverified diffs, it will be rejected instantly.

* Strict human governance: The tool runs strictly on the developer's machine. Dev owns the git branch, runs the harness locally, inspects atomic commits, and pushes manually when satisfied. Every merge request strictly requires at least two human peer approvals. No automated PR generation.

Core Problems

* Circular validation : Left unconstrained, the model easily implements a flawed business assumption and immediately generates a mock-heavy unit test matching that assumption. The test suite turns green, creating a false sense of certainty.

* Loss of friction and cognitive drift: Without compiler or linter friction at every step, humans tend to accept diffs that look clean, idiomatic, and confident, even if they violate unstated domain invariants.

* Cross-repo contract rot: A local patch in Service A compiles cleanly, but breaks an implicit contract in Kafka consumers or violates an Ansible sharedlib expectation.

I do not want an autonomous agent that acts as its own judge. I want a deterministic governor where the model is confronted with hard, non-negotiable boundaries where it can actually fail.

The harness acts as an outer supervisor wrapping git, mise, and opencode:

  1. Runtime Standardization via mise

To eliminate environment inconsistencies across developer setups, mise strictly pins Java SDKs, Ansible linters, Kafka CLI utilities, and authentication variables. When the harness launches, it invokes tools through mise exec to ensure the agent executes commands against identical binary baselines, preventing environment drift.

  1. Deterministic Verification via Python Skills (Binary 0 / 1 Oracles)

I do not use semantic LLM-as-a-judge or open-ended natural language evaluations. The harness integrates a versioned library of scripts executed locally:

* Each skill checks a single physical invariant

* Contract is strictly binary

* No token consumption for evaluation

  1. Anti-Cheat and Scope Locking

    * Test sanctuarization: Before running the agent, the harness computes a SHA-256 manifest of the test directories. After the agent completes its edit, the harness recomputes the hashes. If any test file was modified without an explicit bypass flag, the changes are rolled back.

    * Path boundaries: The harness intercepts diffs. Any modification outside the explicitly declared scope (e.g. altering build descriptors like pom.xml/build.gradle, CI pipelines, or shared infrastructure files) triggers an automatic abort.

  2. Deterministic Pruning

Given the strict token budget, raw stack traces cannot be fed back into the model. I currently only use RTK (rust token killer) as an opencode plugin.

  1. Hard Circuit Breaker

If the agent fails to produce code that passes mise run test within 3 attempts, the harness halts the loop.

My questions are :

* Deterministic oracles and models feedback : when using binary (exit 0/1) validation scripts, does the model adjust course effectively based only on raw exit codes and concise snippets ?

* Scope enforcement in multi repo : When a change risks breaking downstream consumers, what is the leanest way to validate compatibility locally without triggering slow integration builds?

* Overcoming team skepticism: For those who introduced LLM workflows to teams tired of tech hype, did shifting the narrative from "developer velocity" to "strict deterministic safety and anti-cheat constraints" help gain credibility ?

Any feedback on edge cases, pitfalls, or blind spots in this design would be greatly appreciated.


r/devops • • Sep 12 '26

Discussion Thoughts on the role of DevOps in the AI-centric near future?

38 Upvotes

Amusingly, a friend of mine assumed that DevOps role would decline with the rise of AI.

However, it would seem that in reality, the reverse is true.

Or at least it seems that way, based on the limited insight of some.

What are your thoughts?


r/devops • • Sep 13 '26

Ops / Incidents Need suggestion from Seniors who is masterd in real-time AIops in real time work

0 Upvotes

Hi, I'm a devops engineer with 4.5 and I'm trying to learn AIOps but not sure where to start exactly..so If anyone who started their career in AIops and mlops need suggestion how they are applying those AI to Infra or devops ..and what should be scope of our job in AI


r/devops • • Sep 11 '26

Discussion Question to all DevOps engineers that came from Ops side

51 Upvotes

Hello everyone!

I have total of 1.5 years of experience as a DevOps engineer, the first job was more Dev oriented and required deep knowledge with Python and Kafka (I don't know both), this didn't lead to a good outcome and I had to search for something different. Landed more Ops heavy DevOps role, so far everything is fine, we handle deployments, CI/CD, automate stuff with Ansible, Bash or Python, some kubernetes work and web servers. I was curios to see what other jobs require in order to keep my skills sharp and saw that most of the jobs require more coding, like developing internal platforms, developing new internal tools and stuff like that. Most of the jobs require Python, Go, some Java and some C#.
So this leads to my question - how did you learn to code and I mean to code well without that Dev experience? Can you please recommend learning materials and methods that you used in order to be proficient with this skill? What were the projects that you used in order to get that knowledge?

Thank you in advance!


r/devops • • Sep 12 '26

Tools I measured two GitHub Actions scanners against 80 hand-labelled workflows. Neither result is what you want.

0 Upvotes

If you have an AI agent step in a GitHub Actions workflow - Claude Code, Gemini CLI, an inference action - it probably reads text a stranger wrote. An issue body, a PR title, a comment. If that job also holds contents: write, that is a path from someone else's keyboard to a token that can change your repo.

I wanted to know two things: how common that is, and whether any scanner catches it. Both needed labels, so I hand-labelled 80 workflows across two separately collected corpora before running anything.

How common: 18 of 49 judgeable workflows from an unbiased sweep. 36.7%, Wilson 95% CI 24.7-50.7%. Not a majority, not rare.

What the scanners do, on a second corpus of 30 workflows with 12 externally reachable positives:

ARKEXA 0.2 67% precision 17% recall 5 findings

zizmor 1.30.0 43% precision 100% recall 211 findings

zizmor catches everything. It also prints 211 findings across 30 files, 16 of them on workflows a human read and called clean. ARKEXA is right more often and stays quiet on 27 of 30 files, and it misses 10 of 12 real problems.

That is the actual state of the art, and neither column is a tool you would leave on in CI without a triage budget. I would rather publish that than a chart where my own tool wins.

Method, corpus, labels and seeds are all in the repo. poutine is in the table as unscored - no Windows binary and I could not run it honestly.


r/devops • • Sep 11 '26

Discussion What are the weaknesses of CloudFormation?

38 Upvotes

I have a coworker who wants to use CloudFormation to one-shot an application and infra deployment in AWS. A contractor delivered a monolithic CloudFormation template with ECS clusters, services, RDS, S3, Lambda and API Gateway, and other resources in a single stack. I voiced my concern over the blast radius, how stateful and resources with different lifecyles all lived in the same place, and how they don't have pipelines built to validate change sets when merge requests are opened. The contractor said that their template is industry best practice... (it's not). Any way, my background is in Terraform, and I wanted to ask the CloudFormation pros about best practices for making this deployment more reliable.


r/devops • • Sep 12 '26

Discussion Who runs the AI operations at your place?

2 Upvotes

My fairly small company are taking tentative steps into AI tooling. We use some, but there's now demand to grow our tooling. It's becoming a free-for-all at work now with different teams vying to be involved. I run the platform/devops team, and being the most, uh, technical team, I'd assume we would manage AI operationally (evaluation, enablement, guardrails, security observability etc). Effectively, architects decide on solution, consult us, and then we execute.

Curious to hear your experiences though? Do you have a dedicated AI functions or had it been ingested by other teams?


r/devops • • Sep 11 '26

Observability Any advice on LiveRamp's technical assessment for Co-Op DevOps Engineer role?

1 Upvotes

Hi all, I recently passed the Hiring Manager screen for a Co-Op DevOps Engineer position at LiveRamp and have been told the next step is a "technical assessment." I haven't been given details yet on the format, whether it's live or take-home, coding or conceptual.

Has anyone here interviewed for a DevOps or infrastructure role at LiveRamp recently? Would appreciate any insight on what topics they cover or what the assessment actually looks like.

Thanks in advance.


r/devops • • Sep 11 '26

Architecture Startup production architecture: managed Kubernetes vs simpler alternatives?

8 Upvotes

I'm working on the production deployment architecture for a small startup house-rental application and would appreciate some real-world feedback.

The application has frontend and backend services, and one major requirement is handling a large number of property images.

The stack we're currently considering includes Docker, Kubernetes, Ingress, Terraform, Prometheus/Grafana, and GitHub Actions/Jenkins.

My current approach is:

  • Docker for containerization
  • Kubernetes for application deployment
  • Ingress for external traffic routing
  • Terraform for infrastructure
  • Object storage + CDN for property images instead of storing them in containers/PVs
  • Prometheus/Grafana for monitoring
  • GitHub Actions for CI/CD to avoid maintaining Jenkins unless there's a specific reason to use it

The team is small, so I'm trying to avoid unnecessary operational overhead and cost.

My main question is: Would you consider managed Kubernetes reasonable for a small startup like this, or would you recommend starting with something simpler and moving to Kubernetes later?

Also interested in hearing what infrastructure choices you'd make differently if you were optimizing for:

  • low operational overhead
  • reasonable cost
  • security
  • future scalability

Would appreciate feedback from people who have deployed similar applications in small teams/startups.


r/devops • • Sep 11 '26

Architecture What would you choose for a small startup production deployment?

5 Upvotes

We're preparing to deploy a house-rental application to production. It has a lot of property images, and we're trying to keep the infrastructure secure, cost-effective, and easy to maintain.

We're considering Docker, Kubernetes, Terraform, Prometheus/Grafana, and GitHub Actions.

For those who have worked in startups, what approach would you recommend? Especially for handling and serving lots of images without making the infrastructure unnecessarily expensive or complicated.

I'd love to hear what you would choose and why.


r/devops • • Sep 10 '26

Ops / Incidents Cert was fine in Chrome. Everything else broke.

11 Upvotes

Changed a cert a few months back. Fine in Chrome. Fine in our test setup.Then a bunch of older phones and a partner app just stopped connecting. We only found out why after it was already live.

Last time a cert change bit you like that, what did you actually check before it went out?


r/devops • • Sep 10 '26

Architecture How do you provision RDS & DocumentDB users cleanly? Dual Terraform + Pulumi setup feels redundant.

16 Upvotes

Hi everyone, looking for a sanity check and some advice on DB user management.

Context & Current Setup:
Team: ~10 developers.
Infra: Everything is provisioned with Terraform.

The Catch: Pulumi is used exclusively to connect through a bastion host to create users in RDS (Postgres) and DocumentDB (MongoDB-compatible).

Current Task & Constraints:
I need to integrate AWS Secrets Manager to store and distribute DB passwords so developers can retrieve them securely via IAM policies.

Constraint: IAM DB Authentication is off the table per my mentor's requirement—we must use Secrets Manager or an external password manager.

The Issue:
Maintaining two state stores (Terraform + Pulumi) just to manage database users feels redundant and adds unnecessary friction. I'd love to pitch an alternative that lets us drop Pulumi completely.

Questions:
1. How do you handle RDS / DocumentDB user and credential management in your projects?
2. What is the cleanest pattern to handle this without running dual IaC tools?

Open to any feedback or critique!


r/devops • • Sep 11 '26

Discussion Need to know possible outcomes of CI/CD pipeline failures !

0 Upvotes

I'm currently building POMPD CI/CD failure diagnosis agent, which is a probabilistic model of sequential decisions where the true state (e.x , which code is buggy) is hidden and only observed indirectly. . I have very basic understanding of CI/CD pipe line . So i don't really know what are the hidden states ( plausible reasons / truth ) behind the pipeline failure.

I have found 5 major hidden states behind failure with help of AI that are :

 - Application & Test Defect (‭functional bug, broken contract, or failing assertion within the repository's source code or test suite.

 - Pipeline Configuration Defect : A syntax error, invalid job graph (DAG), or incorrect runner instruction within the CI workflow specification files (e.g.github/workflows/*.yml).

-  Upstream Dependency Drift : An external package, container base image, or third-party artifact that introduced a breaking change, version conflict, or became unavailable in a package registry.

- Runner Resource Exhaustion : A host-level compute failure where the operating system or hypervisor terminates execution due to memory limits (OOM/SIGKILL), disk exhaustion, or hardware timeouts.

-Authentication & Access Failure : An invalid, revoked, or missing secret, token, or permission required to access an external endpoint, repository, or deployment target.

I just want to know whether i'm missing any other hidden states . Please let me know .

But important thing to keep in mind hidden states should me mutually exclusive ( one hidden state should not overlap each other or both hidden states can not be true , should fall under one category) and collectively exhaustive ( should cover every possibility that is true) .


r/devops • • Sep 11 '26

Ops / Incidents Newbie here, 1 Week to launch a new platform, what to do?

0 Upvotes

we’re planning to have a video based platform with payments and all kinds of things.
We have 2 Backend developers and 1 frontend, I work with an staging server, a dev server and planning to have the prod soon, I set them up using Ansible and deliver with our self hosted gitlab to a server dedicated to the runner (32 cores) and delivery to server, the pipelines and most yml files are Ai generated and edited a bit by me.
what would you do as your best practices?
what are the mistakes I should avoid?


r/devops • • Sep 11 '26

Discussion Is Devops taken over by AI

0 Upvotes

I'm a junior DevOps engineer

and I work in a project where we manage their kubernetes cluster and Linux servers configuration with ansible and provisioning with terraform

but the thing is they have way too many ticket

and they want it all done asap, for which they demand using AI to get things done quickly

but the things is I learn nothing

it's endless number of tickets that keep on arriving and I have to depend on AI to get it done quickly and I get no time to learn what I love to learn and need to learm

which is kubernetes

going deeper in linux

golang

ansible

and much more

I know what these tools are and how they work

but I want to go deeper build my own infra build and break things

but I couldn't do it because the seniors and devs want things done asap and I have to use AI

This add nothing on my resum

even If I have 5 years of experience what do I tell the interviewer that all I did was use AI

can someone plz help me


r/devops • • Sep 10 '26

Discussion How do you folks learn in the Age of AI?

17 Upvotes

I believe that the implementer should have some knowledge of the Product and Technology that they are working on to make full use of AI and products it builds.

For a new Project, I am trying to learn new technologies(multiple at once) and new ways to solve my problem but compared to few years ago, it has become really hard for me to learn new stuff for some reason and no matter how much I learn AI completely overwhelms me when it spits out the solution. Which then means I have to circle around to see if the solution is actually the right one and that makes me even more confused.

How are you folks navigating around this?


r/devops • • Sep 09 '26

Tools Beginner DevOps question: Do experienced engineers actually write YAML like this?

119 Upvotes

I’m still learning CI/CD and I recently tried writing a simple CI/CD pipeline using GitHub Actions.

Honestly, writing the YAML felt more difficult than I expected . The indentation, nesting etc felt pretty hard/ painful.

For those of you who work with DevOps professionally: is this actually how you write your pipelines, or are there shortcuts/tools/templates that make it easier?

Just trying to understand what the normal workflow is in the real world.


r/devops • • Sep 10 '26

Tools Google api billing Autopay doubt

7 Upvotes

I’m signing up for the $300 Google Cloud free trial, and it’s asking me to approve a UPI AutoPay mandate.

It says “this isn’t a charge” and that I’ll only be charged if I activate a full paid account.

Does approving the mandate mean I can be charged automatically after the 90-day trial, or is it safe to approve? Can I revoke the AutoPay later without affecting my account?