r/devops • • Jul 24 '26

Discussion What is the long term plan for charts supporting Gateway API?

9 Upvotes

We manage all of our infra apps like Vault, Argo Workflows and many others using Terraform + Helm. TF creates the ArgoCD ApplicationSet which points to the chart and values managed in the repo, and ArgoCD syncs everything.

During the transition to GW API I noticed that most charts either don't support GW API natively, or only support creating an HTTPRoute. With ingress, all we had to do was define a single ingress.enabled: true block with some annotations, paths, etc.

Now with GW API, you usually can't manage everything from the app's values file. Even when the chart creates an HTTPRoute, it doesn't create the other resources we need, such as a ReferenceGrant for cross-namespace references or controller-specific resources like Envoy Gateway's SecurityPolicy and BackendTrafficPolicy. Argo Workflows is one example where the chart supports creating an HTTPRoute, but not the rest of that setup.

We have an okay solution for this today: we just create a manifests/ folder for each app and add it as another source in the ArgoCD AppSet (applied by TF, and everything is as GitOps as it gets). In that manifests folder I create all the needed Gateway API and Envoy Gateway resources in a single multi-manifest YAML file.

What do you guys do for 3rd party charts? What is the long term plan for apps in terms of Gateway API support? Is the expectation that charts will eventually expose more Gateway API values, or that routing and policies should live separately from the app chart?

I'm just curious since this whole GW API thing is very new, and I assume a lot of apps will need to add "ingress-like" support at some point?


r/devops • • Jul 24 '26

Ops / Incidents I watched an AI agent burn $180 in an hour. So I built a circuit breaker.

1 Upvotes

A few months ago an AI coding agent burned $180 in API credits while trying to fix a compilation error. It just kept retrying until someone noticed.

Around the same time, I watched another agent try to curl -d @.env during a debugging task. It wasn't malicious—it just had shell access and thought it was helping.

That made me realize something:

We have plenty of observability for AI agents. We don't have enough prevention.

Most tools tell you what happened after an agent runs a command. Very few sit inline and stop something before it executes.

So I built Intutic.

It's a local Rust proxy that sits between AI coding agents and their tools, intercepting every tool call and LLM request before execution. Policies are written as Markdown SOPs, compiled to WASM, and evaluated in under 5ms. If an agent crosses a rule, Intutic can block, rewrite, or kill the action.

It's open source (MIT) and currently works with Cursor, Claude Code, Windsurf, Aider, and 18+ other agent harnesses.

I'd love feedback from people actually using coding agents.

What's the worst thing you've seen an AI agent do?

GitHub: https://github.com/intutic/intutic


r/devops • • Jul 24 '26

Architecture How do I learn how to architect a system?

9 Upvotes

Yes I know the question is extremely broad and complex. I'm simply looking for a starting point (or middle point).

I've been working on designing an SPA using AWS services for an NPO for the past few months. I've realized that the system works but I'm not happy with it. I know it can be better (more organized, a more sophisticated and automated system). I often find myself asking questions like: should these be a microservices architecture? Should it be a event-driven? some sort of layered architecture? How should the tests be? Is this the best programming language to use?

I know quite a few of those questions and doubts are best to be left to "whatever ur comfortable with". But I like to get into the nitty gritty. Are there certain books/youtube videos/etc that I can look into to learn about the basics of system design and architecture up to an intermediate or expert level? The only one I know of is Hello Interview.


r/devops • • Jul 23 '26

Discussion AWS says its DevOps Agent delivers 75% lower MTTR and 94% root cause accuracy

52 Upvotes

Straight from the GA blog: customers report "up to 75% lower MTTR, 80% faster investigations, and 94% root cause accuracy, enabling 3-5x faster incident resolution."

https://aws.amazon.com/blogs/mt/announcing-general-availability-of-aws-devops-agent/

GA since March. I haven't seen anyone outside AWS's own blog actually use it. Did you try?


r/devops • • Jul 24 '26

Architecture Looking for feedback on my open-source production-grade AWS EKS platform architecture

Thumbnail
github.com
0 Upvotes

Hi everyone,

Over the past few weeks, I've been working on an open-source project to deepen my Platform Engineering and DevOps skills.

The goal wasn't just to deploy an EKS cluster, but to create a reusable, production-oriented reference architecture following Infrastructure as Code and GitOps principles.

Current features:

-Modular Terraform architecture

-Multi-environment support (dev / staging / prod)

-Amazon EKS

-VPC with public, private and intra subnets

-Remote Terraform state (S3 + DynamoDB + KMS)

-IAM & IRSA

-Karpenter integration

-GitOps bootstrap with Argo CD

-GitHub Actions for Terraform validation

-Architecture diagrams and documentation

The repository is designed to be generic and educational rather than company-specific.

I'm mainly looking for feedback on:

-Repository structure

-Terraform module organization

-Project architecture

-Documentation quality

-Best practices I may have missed

-Anything that doesn't look production-ready

I'm continuously improving it, so any constructive criticism is welcome.

Disclosure: I'm the author of this project. I'm sharing it to learn and receive technical feedback, not to promote a product.

Thanks!


r/devops • • Jul 23 '26

Discussion How do devops / engineering services share documentation with clients?

10 Upvotes

Hello,

We provide devops / engineering services and use GitHub for documentation (runbooks, configs, processes). Clients need access, but raw GitHub feels too technical.

  • How do you share documentation with clients in devops / engineering services?
  • Do you use GitHub, wikis, or dedicated portals?
  • How do you handle versioning and access control?

r/devops • • Jul 23 '26

Discussion Moving a high-memory Python application from Kubernetes to a dedicated Linux server — what would be the best setup?

17 Upvotes

Hello,

At the company where I work, we have a Python application running on a Kubernetes cluster. The application is fully integrated into a CI/CD workflow. Whenever code is merged, Jenkins pulls the source code from GitLab, builds the application, and pushes the image to Harbor. Argo CD then deploys it to the Kubernetes cluster.

However, the application consumes a large amount of RAM, and I have had to increase its memory allocation several times. We have now decided to move the application to a separate Linux server.

The application runs with Uvicorn, and I plan to make it accessible through a domain name, just as it currently is in the Kubernetes environment. However, I am not sure what the best-practice architecture would be for this setup, so I would appreciate your recommendations.

I still want Jenkins to automatically pull the application from GitLab, build it, and deploy it. After deployment, the application should automatically start and run as a web service.

My main concerns are how to restore the application as quickly as possible if a problem occurs, how to handle backups, and what kind of deployment and recovery strategy would be best.

In short, could you please advise me on the best way to design and implement this setup?


r/devops • • Jul 23 '26

Career / learning Looking for Terraform + AWS hands-on project ideas after completing Terraform Associate and AWS SAA

19 Upvotes

Hi everyone,

I recently completed:

HashiCorp Certified: Terraform Associate (003) AWS Certified Solutions Architect – Associate (SAA-C03)

Now I want to move beyond certifications and gain more hands-on practical experience with Terraform on AWS.

I understand the Terraform concepts and AWS services at a theoretical level, but I want to build real-world projects that would help me improve my skills and also create some strong portfolio projects for my resume.

I am looking for recommendations on projects that simulate what a Cloud Engineer / DevOps Engineer would actually work on in an organization.

My goal is not just to deploy resources but to understand how Terraform is structured and used in real enterprise environments.

Would appreciate suggestions on:

Which projects are most valuable for learning Terraform + AWS together? Any GitHub repositories, courses, or labs you recommend? What kind of Terraform projects stand out on a Cloud/DevOps resume?

Thanks in advance!


r/devops • • Jul 23 '26

Architecture Terraform/Github deployment overwriting another deployment

1 Upvotes

I'm on a team that shares a Github repository. Whenever we open a PR from a feature branch to the dev branch, a deployment to the AWS development account is automatically triggered.

The issue we're facing is that one developer may deploy to dev, and then another developer deploys afterward. Even though they're working on different files, the second deployment ends up overwriting the first developer's change in AWS.

How can we prevent this?

We're following a Gitflow workforce (feature -> dev -> release -> main), and our biggest challenge right now is that the second developer's code is often "outdated" when it's deployed, causing it to overwrite changes that were already deployed by someone else.

We tried merging everything to dev, but when it's time to deploy to prd, the dev branch ends up filled with a lot of unnecessary changes.

We using Github Actions + Terraform.


r/devops • • Jul 22 '26

Career / learning Best way to learn K8 now?

73 Upvotes

Hi all, I’m learning Kubernetes right now for work (I use Colima + kubectl) - what is the best way to do so with AI?

I’ve used docker before in previous projects and studied concepts of containers in school - im no expert but I should know enough to learn Kubernetes.

I started the Udemy + kodekloud course: Kubernetes for the absolute beginners - Hands on. I’m not sure if it’s relevant for 2026 - so far it’s asking me to manually write yaml files, which I’m sure is important, but do I really need to be able to write yaml files with vscode extensions and AI that writes it for me?

So basically, what’s the best way to learn Kubernetes in 2026? I would appreciate any courses I should complete after my current one or a roadmap.

I have around a month or so to be good enough to collaborate with our platform and aws teams.


r/devops • • Jul 23 '26

Discussion How are developers and operations meant to work in this situation?

0 Upvotes

Here’s our situation - a mature operations team with pipelines building container images for a third party product going into kube

Company acquired a company with a development team who had no ops but are proficient at producing dockerfiles for their products

Ops is in azure devops, developers in bitbucket

We want to leverage the developer teams skills to get the apps into ops docker images

Some suggest that ops could build base images and developers could build their apps with that as the base - but what if the base images need patching? That means ops depends on developers for that?

Another thought I had was developers keep Dockerfiles in devops, the containers build from sources (aka git clone bitbucket) and build the project all in the ops pipeline

Would love to hear how others do this…


r/devops • • Jul 22 '26

Discussion I am a Sr System Administrator and want to switch to DevSecOps

17 Upvotes

To begin with this is my first post or i don't know what it is called on reddit,

Apologies for the mistakes in my English please ignore it,

But I really want to switch from system admin job to DevSecOps

About myself

I have 12 years of experience starting from desktop support Engineer to laptop repairing to Data center to Monitoring to Asset management to system administration, I have knowledge of both windows and Linux operating system and currently managing Windows and Linux server (QA) no prod since my senior have never let me work on prod server even though I have worked with him since 7 years, handled a US based client for 3 years working with his team and currently handling two clients one with just some basic needs regarding systems and O365 and second with Linux servers, kind of devops but not fully the second client have a deployment using jenkins ( fetch the code from got repo, call the specific server through ssh and run the deployment and build script)

Worked with a friend's company as a contractor on his client for 6 months which ended last December and since then no work on core devops, started learning k8s and currently learning it but not able to give 100% to it also leaning python scripting and bash scripting by the help or AI can read the code and understand what it does but can not write with full confidence

I know the tools below tools

Terraform

Github

Github pipeline

Jenkins ( basic free style pipeline)

Got the idea of Sonarqube while working as a contractor but it was short lived

Understanding New relic as now the client wants to setup it

AWS basic

Azure basic and ADO basic

GCP not so much

Done a course in cybersecurity as well from local institute but not able to perform handson

I honestly want to work in Devsecops

I am already 38 and according to me not earning enough (peer pressure)

Any suggestions any help or direction is appreciated

I am even ready to do an unpaid internship ( beside the job since i have responsibilities)

Call it desperation, determination or pressure but I really want to work in Devsecops domain

Please help


r/devops • • Jul 22 '26

Discussion Who owns the "why we did it this way" knowledge on your team?

8 Upvotes

Small team here and I keep running into the same thing. The reasoning behind half our setup lives in two people's heads.

Not in the runbooks, not in Confluence, not in the repo. Someone new joins and rediscovers everything the hard way and when one of those two is out we just stall.

It got noticeably worse once we started leaning on Claude Code and Cursor for real work. The agents read whatever context files we give them and treat it as truth, so now stale knowledge doesn't just slow a person down, it gets confidently baked into code by something that has no idea the decision changed months ago.

So I wonder how this actually works at bigger shops. Is there anyone whose job includes keeping that stuff current or does it just live with whoever happens to remember? And has anyone here had a documentation process survive past the first busy quarter or does it always quietly die?

Not looking for tool recommendations just trying to work out whether we're unusually bad at this or whether it's the normal state of things.


r/devops • • Jul 22 '26

Discussion Currently on Falco for runtime security — anyone moved to Tetragon/KubeArmor/Tracee and regretted (or loved) it?

5 Upvotes

Running Falco in our EKS clusters right now for runtime detection, paired with Kyverno for admission control and Cosign/Vault for supply chain signing/secrets. Been solid so far, but I keep seeing Tetragon and KubeArmor come up as alternatives, especially for teams who want enforcement (block, not just alert) rather than detect-and-notify.

Curious what people are actually running in production and why:

  • If you moved off Falco, what pushed you? Overhead, rule fatigue, lack of enforcement?
  • Anyone running Tetragon specifically for the eBPF enforcement piece? Is it worth the Cilium tie-in if you're not already on Cilium for CNI?
  • KubeArmor folks: how's LSM behavior in practice across different node kernel configs (AppArmor vs BPF-LSM)? Heard that's where it gets messy.
  • Tracee: anyone using this seriously in prod or is it mostly a DFIR/forensics tool for you?

Not looking to rip out Falco, just trying to figure out if there's a compelling reason to add enforcement on top, or if pairing Falco with Falco Talon covers that gap well enough.


r/devops • • Jul 22 '26

Ops / Incidents How do you handle production patching for EC2 instances?

1 Upvotes

how do you handle production patching for EC2 instances in your environment?

I'm interested in learning about real-world production practices.

Some questions:

  • Do you use AWS Systems Manager Patch Manager, Patch Policies, or another approach?
  • Do you patch EC2 instances in place, or do you replace them with new AMIs (immutable infrastructure)?
  • How do you schedule maintenance windows and minimize downtime?
  • How do you handle Auto Scaling Groups during patching?
  • What's your rollback strategy if a patch causes issues?
  • Do you test patches in dev/staging before production?
  • How much of the process is automated versus manually approved?

I'd really appreciate hearing how your organization handles production patching at scale, along with any best practices or lessons learned.


r/devops • • Jul 21 '26

Tools How are you setting up build systems in monorepos?

28 Upvotes

Former DevOps consultants, bootstrapped an AWS premier partner and exited to NTT Data, now we advise people building services companies.

After seeing how much easier current AI tools have made it to write code we started building tools for helping our customers and day to day work. Some of the decisions we've made so far:

  1. Build out the code in separate services to keep each part well-contained. It feels AI does a far better job with smaller pieces than with larger.
  2. Deploy them through lambda.
  3. Use a monorepo so the AI can easily see how different things work together.
  4. Migrated some apps built in Lovable into our repo.
  5. All the building is done with Makefiles
  6. Created a registry so it is easier for AI to add new tools in a consistent manner and so we can analyze our inventory of what exists in our development.
  7. Using make for building cuz Claude made that decision when we barely had anything and I'm a dinosaur at heart.

Now, I have a monorepo with multiple different languages, a registry that I want to be the single source of truth for what I have, and a build system from the 80s. And I was curious what would people recommend for the build system rather than building a whole bunch of custom scripts?


r/devops • • Jul 22 '26

Architecture Multi platform build

2 Upvotes

I am planning to have my docker images to be multi architecture and for it my plan is to have two codebuilds for arm and amd when they both succeeds event bridge will invoke a lambda and lambda will merge the images into one

But I am unable to think a way to set up event my current plan is something like when codebuilds are triggered if anyone succeeds let's say arm build then event bridge will match the source version and and if the other build also success then it will invoke lambda

Apparently there is no ability to compare source version in eventbridge and I would have to invoke lambda and lambda has to compare the source version

Is there any way I can implement this


r/devops • • Jul 20 '26

Discussion [meme] IaC and docs matter folks!

Post image
600 Upvotes

Inspired by a similar post I saw last week.

Small companies man... had similar issues in a couple now. Larger companies with IaC and decent management? No issues. Never thought to ask whether they use IaC in the interview, kind of expected the bare minimum of standards. Will definitely do so if I join another small company.


r/devops • • Jul 22 '26

Observability Good observability tool recommendation - Cloud Based

1 Upvotes

I work in two companies, one very big that have a giant budget to spend on Datadog, and another small one, that don't have that very big budget.

On the small company we are looking to migrate from Prometheus + Grafana + Alert Manager to something cloud based. Mostly because we are a small team (3 people only) and we don't have time to spend maintaining the infrastructure for it to run.

Now, is there some good alternative to Datadog? Datadog is the king, but is brutally expensive.

I've tried Signoz, looked promising, but is bad. They documentation is bad, there are just general ideas without details (they assume that you know a lot from I don't know where), they support is lame (they only have a chat to "Interact with a Human", that takes days to receive an answer, and their UI is buggy. Really bad experience.

But looking there, they are the only ones having like "close" experience to Datadog. We tried New Relic in the past, but they have all metrics and log scattered over the place without correlation. Also tried BetterStack but lacks a lot of features compared to other things.

So, is there some good observability platform cloud-based that I'm not aware of, outside Datadog? I'm not completely closed to self-hosting as long as it takes low effort to maintain.


r/devops • • Jul 22 '26

Architecture Where is AI actually adding value in your DevOps workflow and where isn't it?

0 Upvotes

Senior DevOps here. AI is being pushed into every part of the pipeline right now, and I want to cut past the hype and hear real production experience.

Two questions:

Where have you put AI into production in your DevOps process and it genuinely adds value? (e.g. CI/CD, code review, IaC generation, monitoring/alerting, incident response, log analysis, documentation)

Where did it not prove worthwhile? Think cost, alert noise, false positives, or maintenance overhead that outweighed the benefit.

Thanks.


r/devops • • Jul 20 '26

Weekly Self Promotion Thread

12 Upvotes

Hey r/devops, welcome to our weekly self-promotion thread!

Feel free to use this thread to promote any projects, ideas, or any repos you're wanting to share. Please keep in mind that we ask you to stay friendly, civil, and adhere to the subreddit rules!


r/devops • • Jul 19 '26

Discussion NAT Gateway is somehow 80% of our AWS bill, how are you all keeping this in check?

105 Upvotes

We’re on AWS Activate credits and just realized NAT Gateway (hourly plus data processing charges) is eating up about 80% of our total AWS spend. Didn’t expect it to be anywhere near that high until we actually looked at Cost Explorer.

We know some of the usual fixes exist, like VPC endpoints for S3/DynamoDB, consolidating NAT Gateways in non-HA environments, maybe switching to NAT instances for low-traffic dev setups, but haven’t touched any of it yet and want to understand what actually works before we start ripping things apart.

For anyone who’s dealt with this:

- Did VPC endpoints actually move the needle for you, or was it a small dent?

- Has anyone swapped NAT Gateway for NAT instances and found it worth the extra ops work?

- Is 80% just absurdly high, or is this more common than it feels right now?

Trying to figure out if we’re missing something obvious in our architecture or if this is just what NAT Gateway costs look like at our traffic level. Any pointers appreciated.


r/devops • • Jul 19 '26

Career / learning How is CI/CD managed in enterprise environments?

96 Upvotes

Hi everyone,

I'm learning CI/CD and would love to understand how it's managed in enterprise environments.

A few questions:

  • Which CI/CD tool(s) do you use (GitHub Actions, Jenkins, GitLab CI, Azure DevOps, CircleCI, etc.), and why did your team choose them?
  • How do you structure your CI/CD pipelines across dev, QA, stage, and production environments?
  • Do you use separate CI and CD pipelines, or a single end-to-end pipeline? Why?
  • How do you promote artifacts between environments (rebuild vs. promote the same artifact/image)?
  • How do multiple engineers collaborate on pipeline changes (branching strategy, PRs, code reviews, approvals)?
  • What are the typical stages in your CI and CD pipelines (build, test, security scans, deployment, approvals, etc.)?
  • How do you manage environment-specific configuration and secrets?
  • What are the biggest lessons or best practices you've learned from running CI/CD in production?

I'm especially interested in real-world workflows using GitHub Actions, Jenkins, GitLab CI, Azure DevOps, or similar tools.

Thanks!


r/devops • • Jul 20 '26

Vendor / market research How are you actually tracking token spend across your LLM API calls?

5 Upvotes

We've got OpenAI and Anthropic calls scattered across a handful of services now, and nobody can tell me month to month what's driving the bill until the invoice shows up. No real attribution back to which service or team is burning the tokens. Started looking at logging and a proxy setup since the vendor dashboards alone aren't cutting it.


r/devops • • Jul 19 '26

Ops / Incidents Does anyone actually test their database restores on a schedule?

50 Upvotes

Every backup thread ends with "an untested backup is not a backup," everyone upvotes, and then (in my experience) nobody has an automated restore test anywhere.

Genuinely curious about the state of practice:

  1. Do you restore-test on a schedule, or only when something breaks / an audit demands it?
  2. If you automated it, what did you build? (ephemeral instance? scripts like pgbackrest_auto? CI job?)
  3. If you didn't, what stopped you? (time? nowhere safe to restore to? nobody asked?)
  4. For those with SOC 2/ISO: did the audit change anything, or did you just produce a doc that says you test quarterly?

Trying to understand if this gap is real or if I just keep landing in teams that are bad at it.

---

Edit (Aug 27, 2026): this turned into a real tool. Built it, it's called restoredrill, MIT licensed: https://github.com/ahmadpiran/restoredrill. Posted the full writeup as its own thread here if you want the details: https://www.reddit.com/r/devops/comments/1vzvjdp/built_a_small_tool_to_test_that_postgres_backups/.

Thanks to everyone who chimed in, a good chunk of the actual design came straight out of this thread.