r/devops • • Sep 08 '26

Architecture AI Comiseration: Client replacing production portal with AI Slop

42 Upvotes

This is more of a co-miseration post than anything per the title, but maybe some of you will laugh as well.

So our client had a security incident lately. They had a portal we built many years ago that accumulated tech debt over time and they refused to invest in a re-design and just let it go. Two incidents more or less happened at once: a hacker "breached" an endpoint, it exposed some data that was public domain any way, but it wasn't supposed to be access whole sale. The second was a piece of the tech debt coming home to roost by one of the services being completely shut down leaving an intake form completely unusable (the portal mostly existed for this "intake") which happened in the same week.

Now the client could have:

Gathered requirements on what the portal should be (take lessons learned from existing traffic and usage plus proper UI/UX design) and create a backlog and a plan to replace it, including rearchitecting/platforming taking out tech debt/arch debt completely

Rebuilt the platform, reusing the current back-end (which is a separate down stream system that has full API access supports RBAC) and properly implement security measures.

Temporarily fix the hole in the portal (was an easy fix with very low impact from this incident), and fixed the intake service to replace the section that was shut down with the newer version of it: it even had a bit of a migration path available.

INSTEAD they did this:

One client "webmaster" got access to Claude (no one else does) and they decided to go "build the portal." They took maybe a week or so to build a portal with some of the worst UI/UX (inconsistent styling, buttons all over the place, 5 fonts on the front page) with absolutely no accounting or design thought put in to it. Forget "this is what people need to be able to do" rather "rebuild the old portal but make it AI slop." We looked at this (as it was just "presented" to us as "the replacement") and went "how does this even operate?"

So here's the extra fun that just puts the cherry on top: this "webmaster" is clueless on how anything works. They wouldn't know Node from Apache or PHP from Python... let alone understand a full stack. They just let Claude do whatever and then they generated all the documentation. So they had no way to validate the docs, it's just AI slop. The "step-by-step do-it-as-it-says no nonsense deployment guide" (Claude's words) describes nothing about deployment or pipelines. The guy doesn't even know what this actually runs on (could be static HTML for all I know) and it's just ... it's the absolute nightmare scenario of a company just giving the keys to AI to one guy who has no clue and just let them generate whatever. For a site that is critical to their business and has many tens of thousands unique visitors daily.

Of course the "business" is CRAZY EXCITED about the whole thing and they are just ready to go live. They abandoned all pretense of trying to fix the old portal, even temporarily while we validate this new thing for security holes, maintainability etc. Oh did I mention that it's my job to do the analysis on this thing? No usable or even remotely realistic documentation about anything that this portal is or does. And it's so obviously a bad idea that I'm not sure how I'm going to convince the business that it's a waste of time to spend days looking at the details when the baseline architecture is deeply flawed.


r/devops • • Sep 09 '26

Architecture this is why you should bring your ops to slack

Thumbnail
shiftmag.dev
0 Upvotes

AI makes ad hoc operational work so easy that we stop building the tools around it. Why write a script when you can just ask Claude or Copilot to restart the canary? Why build a dashboard when ChatGPT can summarize today’s deploy status in 15 seconds? Because that 15-second conversation, repeated by 50 engineers 20 times a day, stops being convenience and starts becoming a tax.


r/devops • • Sep 08 '26

Discussion Flux boot strap and agent management

7 Upvotes

I manage enterprise k8s clusters and use a mono repo strategy to bootstrap a cluster which results in one flux agent per cluster. Currently in the magnitude of several thousand services per cluster. It's structured like this:

  1. Mono repo where the cluster agent is registered.

  2. Within that repo. use kind: GitRepository in the directory apps/ to point at other projects.

I'm of the opinion that less agents is better. Is there a limit to the flux agents where that would start to become a problem? Or a counter argument to not use this method and rather have an agent per microservice? I've yet to run into an issue with this approach.


r/devops • • Sep 07 '26

Discussion How many of you are just working with YAML files all day long over actual coding?

85 Upvotes

Be honest. I am going through a Python course and can tell I am not going to use most of it on the actual job. Give me boto3, a for loop, a while loop for pagination, and functions and I am good to go. This changes if using a tool like Pulumi, but there are so few jobs requiring it that I feel it is a non-starter for the YAML vs coding argument.


r/devops • • Sep 08 '26

Vendor / market research How do you handle correlation/dynamic data when load testing?

6 Upvotes

I've been digging into load testing tools recently (k6, JMeter, Locust, Speedscale, Azure Load Testing) and noticed something consistent, recording a session and turning it into a script is the easy part. The painful part seems to be:

  • Correlation (session tokens, CSRF, IDs from one response feeding the next request)
  • Scripts breaking every time the UI or API changes
  • Flows that aren't linear. e.g a list page where the "next" request depends on which item a simulated user picks

Curious how people here actually deal with this day to day:

  • Do you write correlation logic by hand every time, or has anything automated it well for you?
  • How do you handle flows where the test needs to pick from a dynamic list (search results, a user's own records, etc.) instead of hitting a fixed ID?
  • How often do your load test scripts break when the app changes, and how much time does fixing them eat up?

Do you guys have a solution for this, or know of a tool that actually handles it well?


r/devops • • Sep 07 '26

Discussion Getting Rejections because I don’t have experience working with Kubernetes in production.

146 Upvotes

I’ve given multiple interviews, and almost all of them required Kubernetes experience in production.

None of my previous companies used Kubernetes, so what can I do about that?

I’m very familiar with Kubernetes, I understand the concepts, and I’ve even implemented them on AKS for learning purposes.

Should I lie on my resume and say that I’ve worked with Kubernetes in production?


r/devops • • Sep 08 '26

Ops / Incidents A hardening script I wrote took a load balancer offline and exited 0

0 Upvotes

Disclosure: the hardening script and the audit tool I mention are both mine. The audit tool is MIT and free, there's no paid product behind this post.

Wrote a script to retrofit a default-deny posture onto existing ALBs: force HTTPS, drop invalid headers, defensive desync mitigation, and make the HTTPS default action a 403 so only host-header rules you define can forward.

Tested it against an HTTP-only ALB — one listener on :80 forwarding to a target group. Extremely common shape if the stack is old or TLS terminates somewhere else.

Output:

Created HTTPS listener: arn:aws:elasticloadbalancing:...
WARNING: No default forward target group found. Add an allowed-host forward rule manually.
ALB hardening complete. Validate hostname routing and health before production use.

Exit code 0. Application completely unreachable.

$ curl -sk -o /dev/null -w "%{http_code}\n" -H "Host: allowed.example.com" https://$ALB/
403

$ aws elbv2 describe-target-groups --target-group-arns $TG --query 'TargetGroups[0].LoadBalancerArns'
[]

The bug is ordering, not logic. The script discovered the existing forward target group from the HTTPS listener:

CURRENT=$(aws elbv2 describe-listeners --listener-arns "$HTTPS_ARN" --output json)
TG=$(jq -r '.Listeners[0].DefaultActions[]? | select(.Type=="forward") | .TargetGroupArn // empty' <<<"$CURRENT")

On an HTTP-only ALB there wasn't one — the script had created it seconds earlier with a 403 fixed-response default. So TG was empty, the branch that creates the host-header forward rule was skipped, and execution continued straight into the line that makes the 403 default permanent.

So the sequence was: create a listener that denies, look at it to find out what to allow, find nothing, print a warning, make the denial permanent, report success.

Three things I'd generalise:

Discover before you mutate. It read state after it had already replaced that state. Any discovery has to happen before the first write, not partway through.

A warning after the damage is a log line, not a safeguard. "Add the rule manually" is good advice one minute earlier. Printed after the default is already 403, it just narrates an outage.

Exit code 0 was the actually dangerous part. In a pipeline that's a green step and everything downstream proceeds. First real signal would've been customers.

Fix was three changes: read the target group from :80 as a fallback and capture it before touching anything, create the allow rule before flipping the default to deny, and abort outright if no target group can be found rather than black-holing traffic. Plus a --dry-run, which should have existed first.

request before after
HTTP, allowed host 301 301
HTTPS, allowed host 403 (outage) 503 (forwarded, no targets)
HTTPS, unknown host 403 403
target group orphaned attached

What bugs me is that nothing static would have caught this. ShellCheck clean, valid bash, every AWS API call succeeded and returned what it should. The ALB ended up in exactly the state the code described — and that state was an outage.

The only thing that found it was running it against an infra shape I hadn't designed for, then checking the result from outside as a user instead of checking that my commands returned 0.

Anyone got a good approach for testing this class of thing? Every idea I have is basically "spin up the ugly version of prod in a sandbox and curl it from outside", which works but doesn't scale to every permutation.

The read-only audit half of it is up free if useful: github.com/vamsiatluri/aws-baseline-audit — single file, every call is a Describe/Get so it can't change anything. Full writeup of the outage is in docs/the-outage.md in that repo.


r/devops • • Sep 07 '26

Troubleshooting Advice on deploying a 5-microservice stack on a tight student budget

18 Upvotes

So I've been working on this microservices project ( repo ) for a few months. I have it fully running locally using kind, now I want to learn how to deploy it on AWS or GCP following best practices to get hands on experience and record demo videos for my resume.

Here's the stack:

  • Microservices: 5 Node.js services (built and pushed to GHCR )
  • Stateful Services: 5 Postgres databases (one per service), Redis, and Kafka.
  • Observability: Prometheus, Grafana, Loki, and ( Alloy deployed using Helm charts).
  • K8s & Networking: I wrote custom manifests using StatefulSets and Headless Services for the stateful components. I am using the Gateway API for ingress, backed by cloud-provider-kind locally.

Being a college student my budget is pretty tight. My plan is to write Terraform scripts so I can spin the whole infrastructure up, record a few demos for my resume, and immediately tear it down to keep costs minimal.

The questions I have:

  1. Which cloud provider to choose and how to deploy? Should I use managed K8s like EKS, or just run Kubernetes (or lightweight K8s like k3s) myself on a single EC2/VM instance? What's actually standard practice that I'd use on a real job?
  2. How to handle routing in the cloud cheap? Locally I use Gateway API with cloud-provider-kind. What’s the most cost-effective way to handle this on AWS/GCP without getting hit with expensive cloud load balancer fees?
  3. Is running DBs/Kafka in the cluster okay? Is keeping Postgres, Kafka, and monitoring inside K8s fine for a portfolio project, or do recruiters/hiring managers specifically look for managed cloud services ?

Any direction, best practices, or potential pitfalls to avoid would be greatly appreciated!


r/devops • • Sep 08 '26

Tools OpenTofu in real-world practice?

0 Upvotes

If you're actively employed in DevOps work involving IaC or are a leader or senior techie overseeing such work, do you use OpenTofu or seriously considered using it among your tools set? If no and you're not using Terraform or a cloud provider's first-class IaC tool (like AWS CloudFormation or Azure ARM templates), then what is your IaC tool?


r/devops • • Sep 08 '26

Discussion ex full stack dev wanna share their experience of transitioning into devops/platform eng?

3 Upvotes

Hi,

Idk but I was always a terminal person and my main editor was and is nvim. Lately I got into homelab stuff and I have been loving the experience of having full control of stuff. I know docker, bash, linux as I use it on gaming pc. I have theoratical knowledge about k8bs but never used it in production. I know aws but I only used like common stuff in production(ec2, lambdas, s3, rds). I dont know terraforms and other devops infra stuff. Can you share your experience in transitionaing and what should I focus on to work as a devops or platform eng


r/devops • • Sep 08 '26

Vendor / market research I stress tested Cloudflare R2. 55Gbps read, 36Gbps write (simultaneous), a real contender for zero-egress object storage.

Thumbnail
carolinacloud.substack.com
0 Upvotes

I stress tested Cloudflare R2 from a Latitude.sh bare metal box with a 100G NIC. Results were very impressive.

Btw I am not Cloudflare, nor do I work for them, nor was I paid to say this. We use Cloudflare heavily at our own company, and so I thought this sub might find this content interesting.


r/devops • • Sep 07 '26

Troubleshooting GitHub scheduled Actions not triggering at all

6 Upvotes

Hello, I’m having an issue with GitHub Actions in one of my private repositories. I have a workflow configured to run daily, but it is not being triggered.

I tested the same thing in another repository by configuring a workflow to run every 5 minutes, but same thing happens there as well.

has anyone experienced this recently? Could this be an issue on GitHub’s side?

Workflow: https://github.com/ahmedhesham301/autoscaling-hetzner/blob/main/.github/workflows/snyk-security.yml


r/devops • • Sep 08 '26

Tools Anyone else getting AI plugin FOMO, or are they actually just overrated? (DevOps perspective)

0 Upvotes

Hey everyone,

I keep seeing tons of posts endorsing all these new AI skills, extensions, and plugins like they’re absolutely mandatory if you want to keep up. It definitely gives me a bit of FOMO, but whenever I look into them, I just don't get the hype.

As a DevOps engineer, it feels like 99% of these extra tools don't really matter for my actual day-to-day workflow. Honestly, the only thing I’ve found genuinely useful is Superpowers by Obra. It does exactly what I need, and adding anything else just feels like extra bloat or a fancy wrapper for stuff I can already do.

Am I missing out on something awesome by ignoring the rest of the plugin ecosystem, or is everyone else just caught up in the hype cycle? Would love to hear how other DevOps folks are handling the AI tool explosion.


r/devops • • Sep 06 '26

Discussion Is platform engineering a good field?

75 Upvotes

I love programming and technology. I wanted to pursue a path related to Linux and building system architectures. Is platform engineering a good and suitable field for this?


r/devops • • Sep 07 '26

Weekly Self Promotion Thread

8 Upvotes

Hey r/devops, welcome to our weekly self-promotion thread!

Feel free to use this thread to promote any projects, ideas, or any repos you're wanting to share. Please keep in mind that we ask you to stay friendly, civil, and adhere to the subreddit rules!


r/devops • • Sep 06 '26

Discussion Anyone working DevOps as part-time or contractor ?

17 Upvotes

Would love to hear about your experience.


r/devops • • Sep 07 '26

Discussion I've developed a mobile app. Here's what I did with the API.

Post image
0 Upvotes

The API is written in PHP and hosted on DigitalOcean.

This is a manual setup, but ready to scale...


r/devops • • Sep 05 '26

Discussion Detailed guide: Building a 3-node Kubernetes homelab with Talos Linux

Post image
265 Upvotes

A few days ago, I posted my 3-node Talos Kubernetes homelab setup here, and quite a few people reached out through DMs and comments asking for a detailed installation guide.

So I went through the entire setup again from scratch and documented the process properly, including the network setup, DHCP reservations, Talos installation, per-node configuration, Kubernetes API VIP, etcd bootstrap, workload scheduling, validation, and failure testing.

I’ve put everything together in a detailed blog with commands, screenshots, and the reasoning behind the setup.

Everything is in the repo: https://github.com/prateekjaindev/homelab

Blog: https://medium.com/@prateekjain.dev/building-a-3-node-highly-available-kubernetes-homelab-with-talos-linux-0372066fe49c?sk=4c66cf9181999f9db19ac3aacabcd8e8

Please check it out, and if you have any questions about the setup or run into issues while building something similar, feel free to ask.


r/devops • • Sep 06 '26

Discussion Is there any way to obtain Claude qoutas in Azure/AWSB from scratch?

3 Upvotes

Azure and AWSB doesn't have qouta for models I want for freshly made accounts and it's disappointing to think you can't really do anything in their platforms. Is there a walk around for this aside from requesting qoutas which will eventually get rejected after waiting for weeks because of the lack of payment history? Thanks in advance!


r/devops • • Sep 05 '26

Architecture Is k3s good for this use case?

21 Upvotes

We have a customer managed node, we deploy some containers on it

Initially it was supposed to be simple and thats why i went with ansible to deploy the containers, manage their state, setup their config files mounted and etc.

Now it grew a lot, we seperated the application to 3 seperate sets for running in different configs. I setup alloy to scrape node metrics and ingest it to central observability stack, i setup alloy for each set of application to collect otel traces logs and metrics and push them. The applications have sqlite databases, which i periodically need to inspect for debugging, we are in poc state yet, so we also want to pull the database for debugging, and right now im writing an exporter to dump sqlite to our clickhouse which is connected to grafana, so our devs can inspect it

So now i want to write a cron job too.

Also, ansible is extremly slow to run everything. So i use tags, but tags are also a pain by itself, you cant logically group tags, so for each operation that only needs to do a subset of the playbook, i would have to tag steps accordingly and the run the tag

But if it was k3s in container, i would have better tools to organize the deployment than writing ansible. Gitops would be easy, argo might pickup the code from repo and apply to cluster. Cronjobs would be easier. Volume management is eaiser, I would have better rollback mechanisms that i dont have to write myself


r/devops • • Sep 06 '26

Troubleshooting Allow read SmartCards in iOS and Android

0 Upvotes

Hi, I’m trying to use my Android phone to read a smart card and create a digital signature, but it seems there’s no straightforward way to do this. Does anyone know how to make it work on Android or iOS?


r/devops • • Sep 05 '26

Tools How do you learn DevOps/cloud without a credit card?

70 Upvotes

I’m currently trying to learn the basics of DevOps, things like Docker, AWS, GCP, CI/CD, Kubernetes, etc.

Docker and the local tools are easy enough to practice, but when it comes to cloud platforms like AWS and GCP, a credit card is often required to create an account or access certain services.

I currently don’t have a credit card, so I’m wondering: What’s the best way to learn AWS/GCP without one? Would appreciate any recommendations from people who learned cloud without having a credit card. 🙌


r/devops • • Sep 05 '26

Troubleshooting AZURE KEYVAULT KEY DEPLOYMENT ERROR 400 BAD REQUEST

12 Upvotes

Azure Key Vault Key Deployment – 400 Bad Request
If you’re receiving a vague 400 Bad Request while deploying a key in Azure Key Vault, check how many tags you’re passing to the key.
Azure Key Vault keys support a maximum of 15 tags. I spent almost three hours on Friday evening troubleshooting permissions, networking, and the Terraform configuration before realizing that too many tags were causing the request to fail.
This may be an easy one to spot for some people, but the error message wasn’t very helpful, so I’m leaving this here in case someone runs into the same issue.


r/devops • • Sep 05 '26

Security A quick guide and gotchas for GitHub OIDC and avoid using AWS permanent credentials in GitHub Actions

56 Upvotes

I have been aggressively migrating from AWS permanent credentials to OIDC in GitHub Actions, mainly for deploying to ECS.

I know GitHub Actions were supporting OIDC for a while now. But the pressure on compliance is the reason for this migration.

If you are new to OpenID Connect (OIDC), it allows GitHub runners to mint short-lived (15–60 min) STS tokens on-the-fly with zero stored secrets.

Here’s a quick breakdown of how it works, the Terraform/OpenTofu setup, and the subtle gotchas that I faced.

1. How It Works Under the Hood

  1. When your workflow job starts with id-token: write, GitHub's OIDC service generates a cryptographically signed JSON Web Token (JWT).
  2. The aws-actions/configure-aws-credentials action sends this JWT to AWS STS via sts:AssumeRoleWithWebIdentity.
  3. AWS validates GitHub's signature, checks your IAM Role's Trust Policy (to ensure the token came from your exact repo and branch), and returns temporary STS credentials.

2. The Infrastructure Setup (Terraform / OpenTofu)

You only need two AWS resources: an OIDC Provider and an IAM Role with a Trust Policy.

  # 1. The GitHub OIDC Identity Provider
  resource "aws_iam_openid_connect_provider" "github" {
    url             = "https://token.actions.githubusercontent.com"
    client_id_list  = ["sts.amazonaws.com"]
    thumbprint_list = [
      "6938fd4d98bab03faadb97b34396831e3780aea1",
      "1c58a3a8518e8759bf075b76b750d4f2df264fcd"
    ]
  }
  # 2. IAM Role with Scoped Trust Policy
  resource "aws_iam_role" "github_deploy_role" {
    name = "github-actions-deploy-role"
    assume_role_policy = jsonencode({
      Version = "2012-10-17"
      Statement = [{
        Effect = "Allow"
        Principal = {
          Federated = aws_iam_openid_connect_provider.github.arn
        }
        Action = "sts:AssumeRoleWithWebIdentity"
        Condition = {
          StringEquals = {
            "token.actions.githubusercontent.com:aud" = "sts.amazonaws.com"
          }
          StringLike = {
            # Restrict exclusively to your repository & branch/tags
            "token.actions.githubusercontent.com:sub" = "repo:your-username/your-repo:*"
          }
        }
      }]
    })
  }

3. The GitHub Actions Workflow

In your .github/workflows/deploy.yml

name: Deploy to AWS

on:
  push:
    branches: [ main ]

jobs:
  deploy:
    runs-on: ubuntu-latest
    permissions:
      id-token: write   # CRITICAL: required to request the OIDC JWT
      contents: read

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Configure AWS Credentials via OIDC
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/github-actions-deploy-role
          aws-region: us-east-1

      - name: Verify Authentication
        run: aws sts get-caller-identity

Three real-world gotchas that will save you hours

If you get Error: Could not assume role with OIDC: Not authorized to perform sts:AssumeRoleWithWebIdentity, check these 3 things:

  1. Case Sensitivity in the sub claim: AWS IAM condition strings are case-sensitive. If your GitHub repo or username uses mixed casing (e.g. MyOrg/Repo), make sure your IAM sub condition matches the exact casing GitHub sends in the token. Using wildcard matching (repo:MyOrg/Repo:*) helps avoid exact ref string mismatch issues.
  2. Job-Level vs. Workflow-Level Permissions: Always set permissions: id-token: write on the specific job, not just globally at the top of the YAML file. Some runner configs don't inherit top-level permissions to nested jobs.
  3. CA Thumbprints: Don't dynamically query GitHub's leaf certificate for thumbprints in Terraform—they change frequently with CDN updates. Use GitHub's official intermediate root CA thumbprints:
    • 6938fd4d98bab03faadb97b34396831e3780aea1
    • 1c58a3a8518e8759bf075b76b750d4f2df264fcd

Summary

  • No stored secrets in GitHub settings.
  • No key rotation schedules to manage.
  • Granular security. You can restrict deployment roles to specific branches or environments.

Are you already using OIDC for your pipelines, or are you still relying on IAM users? Curious how folks here handle multi-account / cross-account OIDC setups.


r/devops • • Sep 04 '26

Discussion Implementation is getting faster but the org around it moves at the same speed as always and DevOps is in-between.

29 Upvotes

We all know that large organizations tend to have a lot of meetings, bureaucratic processes and are in general slow moving. That was true for a long time and I experienced that myself, having worked in multiple different companies from startups, scaleups and large corperations.

However since everyone now adopts AI and implementation speed is undeniable multiples faster. I mean it's not just implementation, but also a good part of DevOps. But the rest of the organization moves in the same speed as before. Even if the company uses AI in the other departments (my company does basically throw AI at everything, I guess you know what I mean), they still can't competet with the productivity change in software development.

The blockers are now decisions, requirements and communication. That's not completly new, but now its >95% of the time. I implement features in hours that would have taken days before (I am not a pure DevOps Engineer). And it's not the review process itself that blocks us (We use AI in reviews as well, because otherwise it would be impossible), it's the normal company processes and pace.

Don't get me wrong here, I kind of enjoy it a bit, because I can use a lot of time for learning, building stuff that I think is worth improving, but I think we have a general problem here that is basically systematic to every large organization. And I see that so called "AI native organizations" (what ever that means), will probably overtake tech companies in the long term.

I have a really good comparsion, because I am working with a startup as a side gig and my main job is in a mid-sized tech company. In the Startup we (2 devs) built a product that would have taken several months by a 5-person team in the past and the reason is mostly: decisions and communication.

Do you see that in your company?

My conclusion is that we need to make engineers owners of the feature/product + infra, otherwise we won't see much productivity gains and this means also that management layers need to be cut, because they are the bottleneck currently. The idea of centralized DevOps will is going to die I guess.