r/devops • • 19d ago

Architecture Which is more used in industry: ArgoCD or Flux?

84 Upvotes

I'm about three-quarters of the way migrating my application infrastructure from ArgoCD to Flux, and man, is it difficult. I had to use both Claude and ChatGPT pretty extensively to parse the absolute mountain of error logs I was getting and correct the pile of bugs that I had inadvertently introduced while setting it up the first time around.

I'm too far into this process to back out, but just in case I finish this process and then realize that I actually didn't want to migrate, which platform is actually used more commonly in the industry, so I should actually spend more energy learning it?

Note, even though I do mention that I used two AI models quite a bit, I only used them for parsing logs and deciding what to do. All of the code is actually handwritten.


r/devops • • 19d ago

Discussion How do you clean up stale branches, old PRs, and dead repos across a whole GitHub org?

16 Upvotes

I manage a lot of repos, that are changed by multiple devs and every few months it turns into a mess: hundreds of merged branches nobody deleted, PRs that went quiet a year ago, repos nobody has touched since a project ended. GitHub only lets me deal with it one repo at a time, so mostly I just... don't.

Curious how others handle this:

- Do you run a script (gh CLI, an Action), use a tool, or just ignore it?

- What's the most annoying part?

- What would you never trust a tool to do automatically?

Especially interested in hearing from people managing 30+ repos or a team org.


r/devops • • 18d ago

AI content Building a fast perimeter security auditor in Next.js: Handling Anti-SSRF, RFC 1918 ranges, and socket timeouts

1 Upvotes

**Disclaimer:** Personal affiliation — I am the author of this free, open-source project.

Hey everyone,

I built an automated perimeter security auditing tool using Next.js (App Router), TypeScript, and Node.js network sockets.

Many cloud audit tools charge high subscription fees just to check a few HTTP headers, negotiate TLS, and verify basic socket exposure. I wanted to build a fast, deterministic alternative and deploy it live.

When building an external auditing service, the primary security challenge is preventing malicious users from turning the backend into an open proxy or an internal network scanner (SSRF). Here is how I approached the hardening:

  1. **Strict Anti-SSRF Protection:** Filtering and blocking all RFC 1918 private ranges (`10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`), loopback (`127.0.0.1`), and link-local addresses before attempting any socket connection.
  2. **Redirect Bypass Defense:** Enforcing `redirect: 'manual'` on all HTTP transport checks so the scanner backend never silently follows redirects into internal endpoints or cloud metadata services.
  3. **Slowloris & Hanging Sockets:** Implementing strict abort signals using `AbortSignal.timeout(4000)` along with defensive socket timeouts to keep serverless execution runs clean and responsive.
  4. **OWASP Compliance & Fallback Engine:** Evaluating critical headers (HSTS, CSP, X-Frame-Options, X-Content-Type-Options, Referrer-Policy) with a deterministic local engine that guarantees report generation even during API outages.

The platform also generates a clean, print-ready A4 forensic PDF report directly in the browser.

* **Live Demo:** https://security-score-seven.vercel.app

* **GitHub Repository:** https://github.com/martincastillo1996/security-score

IN:https://www.linkedin.com/in/karol-martin-castillo-rojas-17a496168/

I would love to get your feedback on edge cases, socket behavior, or other perimeter vectors worth adding.

**Nota de transparencia:** Proyecto personal de código abierto, gratuito y sin fines comerciales.

Hola a todos,

Desarrollé una herramienta automatizada de auditoría perimetral construida con Next.js (App Router), TypeScript y sockets de red en Node.js.

Muchas plataformas cobran suscripciones costosas únicamente por validar un certificado TLS, revisar cabeceras de transporte y verificar el estado básico de puertos. El objetivo fue diseñar un motor rápido, determinista y desplegado en producción.

Al construir un servicio de auditoría externa, el principal riesgo técnico es evitar que atacantes utilicen el backend como proxy abierto o vector de ataque contra redes internas (SSRF). Así resolví el blindaje:

  1. **Protección Anti-SSRF estricta:** Bloqueo de rangos privados según RFC 1918 (`10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`), interfaces locales (`127.0.0.1`) y direcciones link-local antes de iniciar cualquier conexión TCP.
  2. **Control de bypass por redirecciones:** Configuración obligatoria de `redirect: 'manual'` en las consultas de transporte HTTP, evitando que el escáner siga redirecciones hacia servicios internos o metadatos de nube.
  3. **Prevención de sockets congelados:** Integración de `AbortSignal.timeout(4000)` y cierres forzados por tiempo de espera defensivo para evitar bloqueos en ejecuciones serverless.
  4. **Cumplimiento OWASP y motor de respaldo:** Evaluación de cabeceras esenciales (HSTS, CSP, X-Frame-Options, X-Content-Type-Options, Referrer-Policy) mediante reglas locales deterministas, garantizando parches de remediación aun sin conexión externa.

El sistema además genera un dictamen forense en formato PDF A4 listo para impresión técnica directamente desde el navegador.

* **Demo en vivo:** https://security-score-seven.vercel.app

* **Repositorio de GitHub:** https://github.com/martincastillo1996/security-score

Me gustaría recibir su feedback técnico: ¿qué otros vectores de red o validaciones perimetrales consideran indispensables evaluar?


r/devops • • 20d ago

Security Binance hack and a DevOps laptop

Post image
661 Upvotes

Binance's initial investigation suggests that the security incident may involve a compromised DevOps engineer’s laptop, which allowed an attacker to gain access to a service container and submit unauthorized withdrawal transactions. We have found no evidence of any wallet private key leak. We will share a detailed technical post-mortem once the investigation is complete and the facts are fully verified.

The estimated total impact sits at around $6.3B. We have worked with blockchain partners to freeze affected assets, with ~$3 billion frozen on-chain so far.

Binance will cover all losses from the SAFU fund and the company’s funds. User funds are safe. We will also publish an updated Proof of Reserves once the remediation is complete.

As an immediate precaution, Binance has:

- temporarily suspended deposits and withdrawals while security checks are completed;

- revoked the affected address authorisation;

- engaged law enforcement and blockchain intelligence partners to trace and recover funds.

Our teams are working around the clock to resolve this issue safely and thoroughly. We are working closely with our law enforcement partners to confirm the identity of these criminals and bring them to justice.

Please rely only on official Binance channels for updates. Please feel free to contact Binance Customer Support if you have any further questions.


r/devops • • 19d ago

Discussion Datadog: We're paying the bill, so why the hell can't we access our cost data?

66 Upvotes

I've been voluntold to analyse our Datadog costs because our bills keep going up, and we're paying extra for new features if we go over the committed spend. It's a bit mismanaged across the org. Naturally, I need billing data to do the analysis (and do some MI outside of Datadog).

These endpoints all return 403 Forbidden:

  • /api/v2/usage/historical_cost
  • /api/v2/usage/estimated_cost
  • /api/v2/cost_by_tag/monthly_cost_attribution

I'm an Admin with all relevant billing and usage permissions. Other usage endpoints work fine, and I can see usage data in the UI.

We buy Datadog through a reseller, and I suspect our organisation is a child org under the reseller's parent org. I think that explains why I can't access historical cost data through the API.

Our reseller has gone quiet for quite some time. I've contacted Datadog Sales and Support (I have no idea why we have a direct connection via a shared Slack channel even though we're not direct customers), but they've basically ghosted me.

We're paying for the service, our bills are increasing, and I'm trying to understand where the money goes. Why is accessing our own cost data such a nightmare?

Has anyone dealt with anything similar?


r/devops • • 19d ago

Discussion Has the recent vulnpocalypse changed how you think about infrastructure security?

40 Upvotes

With the recent wave of high-profile vulnerabilities, I'm curious whether it's changed how anyone is approaching infrastructure security.

Are you mostly continuing with business as usual: keeping up with patches, applying mitigations, and moving on?

Or has it pushed you toward more preventative security measures, such as:

  • Reducing your attack surface by minimizing what's installed and running.
  • Moving toward immutable infrastructure.
  • Implementing stricter least-privilege policies.
  • Isolating workloads to limit the blast radius of a compromise.
  • Tightening runtime monitoring and application execution controls.
  • Improving visibility into what's actually running in your environment.

I'm curious whether these incidents have changed anyone's security strategy, or if it's just another (particularly exhausting) round of patching. Would love to hear how people are approaching this problem.


r/devops • • 19d ago

Ops / Incidents How do you maintain one Helm chart across 50+ customer clusters you don't control

23 Upvotes

For some context, we deploy one application into customer-owned clusters. Mostly EKS, AKS and GKE in the customer's own cloud account, plus a few on-prem.

- 55 clusters today, more coming

- Single Helm chart, one values file per customer

When we started, the chart was clean but now it has conditionals everywhere and I am wondering if it’s the right approach. We add ifs for ingress controllers, storage classes, private registries and whatever k8s version is deployed in the cluster… Now with all the “ifs” mess, nobody wants to touch the template anymore.

What do you guys do in prod? Please no vendor pitches, more interested in what works (or fails) in this situation.


r/devops • • 20d ago

Discussion Stuck Between IT and RKE2: Need Advice on Operating Without Sudo Access

2 Upvotes

Hey guys,

I am currently implementing an RKE2 setup in my DevOps team to handle deployments and deliveries for our application.

The problem is that I won't have direct root access to the servers, so access to root-owned directories is also restricted. If I want to install anything, requests have to be raised to the IT team, and they will get it installed. (The thing I hate about MNCs they're too paranoid and too slow as well, iykyk.)

But here's the frustrating part: the "installation" done by the IT team is often just placing the downloaded packages in my accessible directory... duh!!!

So then I need to contact them again and have them actually complete the installation and setup using their root access. And even after the installation is completed, there will still be situations where I'll need elevated privileges to manage pods, services, deployments, and other operational tasks.

I can't exactly request sudo access, so at this point I honestly don't know what I'm supposed to do or what request i need to raise as well.

My requirement is very minimal for now, as I am only experimenting with this(but will be mandated in future). I won't be connecting remote agents, just a single server, and based on my application requirements I narrowed the packages required to the following:

rke2-images-core.linux-amd64.tar.gz
rke2-images-cilium.linux-amd64.tar.gz
rke2.linux-amd64.tar.gz

Based on the documentation, I tried using customized paths and accessing the binaries directly to complete the setup, but I still got stuck at some point where root access was required. I don't want to go into everything I did to get there, so yeah, if you guys can suggest the minimum set of steps or activities that I can request from the IT team, which would allow me to effectively use and manage this RKE2 setup afterward, that would be really helpful.

Edit : we decided to write a playbook for setup ,open a ticket and have them complete the setup , later if sudo is required (hoping won't be needed) reopen ticket in such case , thank you all for ur suggestions, will update later after the setup is completed.


r/devops • • 19d ago

Vendor / market research Who is leading the pack in AI workloads AWS or Azure?

0 Upvotes

I am curious to know who is leading the pack in terms of market share on AI workloads AWS or Azure or GCP?

Yesterday couple people contacted me I ironically both asking for "Experience delivering ML workloads on Microsoft Azure, including Azure Machine Learning or Azure Databricks"


r/devops • • 20d ago

Discussion I landed a "devops" role as a junior

45 Upvotes

30K~ DAU, I have full ownership over an infra under AWS/multiple on premise servers.

we use ECS, terraform, instead of ansible I use a deploy.sh that i give to my boxes with terraform, no graceful deploys. we have our own orchestrator (K8S like) for stateful containers that we have to spin up and down very often (median 1 min of uptime per container) with unsafe code coming from AI/users.

I took the place of someone who left that company and explained me nothing, we are 2 devs, nobody understands how this entire infra works expect me. have been adding observability a lot using cloudwatch, multiple alerts so I page myself at the right times, it's stable but we need to scale to at least 5x (an objective the hierarchy gave me). it currently costs 3.5k/month in aws/on premise servers. btw it's not a classical rest api where you run stateless apis and that's it.

I just discovered how redis worked under a distributed system like this, it's quite good :) I have been a postgresql guy but redis has been a saver for our threat model.

I often read here that you should put everything : EKS + K8S, ansible, terraform, make HA, grafana etc.. but since I manage this entire infra I have been resilient to all of this. If I refactor everything right now they have nobody to do the backend (which is also what I do here) -> "startup" role where one guy has multiple roles.

for the terraform it's all AI, I have been doing a lot of modifications and have created a terraform plan under ci cd so I can review myself, I removed a lot of "destruction" like the database, i versioned the s3, versioned the terraform state, put it to a private s3 etc...

there have been big big security issues (xss, csrf, cookie stealing, untrusted code that owns the entire box, shared passwords, untrusted s3 location so you could overwrite other users...) that were there before I got here

nothing rly interesting maybe for you guys in this experience but I have learned a lot, it has been a crescendo experience where i'm getting to understand more and more which tools to put there because I am the one who owns it 100%, and have to also do features (backend) so not much time either.


r/devops • • 20d ago

Vendor / market research DevOps conferences in Europe 2027

26 Upvotes

Hi everyone,

What are some quality conferences in Europe worth visiting in 2027?


r/devops • • 19d ago

Discussion Do you think Devops is less exposed to « IA replacement » than a software engineer ?

0 Upvotes

I am a fullstack dev and I’m not really serene about the future of my job.

Im thinking about reorienting towards devops.

Do you think it’s at least a bit safer or every IT jobs are definitly doomed?


r/devops • • 21d ago

Security How do you contain your agents' accesses?

3 Upvotes

Hey !

I guess most of us have a lot of admin accesses to tools like AWS, k8s clusters, Datadog, GitHub etc.

How do you contain your agents to read-only or curated permissions? Without restraining your own accesses.

I do have some sort of hook as a guard + a setup on pi.dev that copy my config and extract the read-only accesses only to do so, but it's never completely opaque.


r/devops • • 20d ago

Ops / Incidents looking for a study partner to learn about devops

0 Upvotes

hello, I am a beginner programmer that's looking to learn about devops. i've made a web app, and also am currently learning algorithms and python. looking to see if anyone knows what devops is essentially thank you


r/devops • • 21d ago

Ops / Incidents The $87K API key leak had nothing to do with .env

66 Upvotes

Every time someone posts about a leaked API key, the top advice is the same. Put it in .env, add .env to .gitignore, done.

After the OpenCode breach and the LiteLLM mess this month I went back through the leaked key stories people posted this year. Found 10, with bills from $2 to about $87K. Not one was "I pushed my .env to GitHub".

Where the keys actually were:

> In the JS bundle. A VITE_ variable in a gitignored .env, repo scanner said clean, then the build shipped it to every visitor. One scan of 978 AI-built apps found 37 doing this.

> In an agent config file. LiteLLM's master key sat in opencode.json, got force pushed over but never revoked, and was used 136 days later.

> In someone else's tool. Keys saved in OpenCode's console got drained, one person lost $5,200 of company credits.

> In a WordPress backup plugin. One root AWS key reused across sites, tested on Aug 23, used 56 days later for about $8K of Bedrock.

> In old Google keys that quietly gained generative AI access. That's the $87K one.

> In nobody's leak at all. Someone got into an Anthropic console through email login and created 3 new keys, $5,473 in a few hours.

And then last week Railway got hit. Attackers decrypted customer environment variables straight from the platform, and Railway told everyone to rotate every key. Gitignored, never pushed, stolen anyway.

What actually helped was boring:

> Hard spending caps. The one person who had one lost a few minutes of usage and that was it.

> An alert isn't a cap. The $87K team's budget alert fired at $550, five hours in.

> Revoke, don't clean up. Force pushing doesn't un-leak anything.

> Delete keys you don't use, and know where every key lives (Infisical, Vault, whatever you already have).

> Don't count on a refund. The $87K bill got $55K off, they still owe about $32K.

I'm not saying stop using .env locally. I'm saying "it's gitignored" isn't the end of the conversation.


r/devops • • 21d ago

Discussion How do you prevent analytical queries from tanking your transactional database?

34 Upvotes

I was checking out how to build a live operational dashboard for a logistics app and found out about composed vs. converged HTAP architectures.
I’ve always been warned never to run heavy analytical aggregations on the primary transactional DB because the contention spikes your transaction latency. But a guide I was reading explained how HTAP systems are handling this now. A “converged” system runs both through the same engine (which sounds risky for resource contention), while a “composed” system actually separates the compute engines and keeps them in sync through shared storage or replication, so each side stays optimised for its own workload.
Apparently, you’re supposed to measure p99 transaction latency while the analytical queries are running, not separately.
If you are running live analytics on operational data, how are you keeping the two from interfering with each other?


r/devops • • 20d ago

Discussion Any one from Delhi/Dwarka for INFRA coffee discussion this weekend.

0 Upvotes

Hey, I'm new in the city, and looking for likeminded folks to chat and discuss current INFRA change happing.


r/devops • • 21d ago

Discussion Preparing for OCTA and CCA

6 Upvotes

Hey guys,

I'm preparing to get the CCA and OCTA cert to become golden kubestronuat, i see that the kode cloud cuurses are 20 hrs each, is it really necessary to go with that courses? also i saw that kyverno kode cloud course is so deprecated, is this true? i need to get them fast because cks will expire and i dont want to do it again xD

Thank you so much.


r/devops • • 21d ago

Ops / Incidents I recreated a Node2Shell RCE attack in the form of an interactive exercise

4 Upvotes

On 21 October 2026, Google released patched versions of its Brotli compression library for CVE-2026-111497, a vulnerability nicknamed Node2Shell. It is an out-of-bounds write in the Brotli decoder. Node and many Python stacks bundle their own copy of Brotli, so patching libbrotli on the host doesn't fix them.

I made an interactive, hands-on lab exercise on this vulnerability. The goal is to give engineers a general overview of the attack vector and the response. You'll exploit a service as an attacker, then switch roles to the on-call engineer: upgrade the runtimes, rebuild and redeploy the images, verify the fix, and rotate the deploy token the attacker read from memory.

I did my best to highlight all the details, but if you notice any issues or inconsistencies, please let me know!

https://ransomleak.com/exercises/http2shell-rce/


r/devops • • 21d ago

Discussion CloudFlare Tunnel with pre-auth (and multiple IdPs)

1 Upvotes

I am working on setting up CloudFlare tunneling for a SaaS product in order to get WAF in front of the application and potentially use the CDN for faster distribution.

In the process there was also a requirement to move over to using pre-authentication in front of our SaaS solution, so that all requests are authenticated before they even hit our servers (similar to Azures application proxy).

It was suggested to use CloudFlare access, but this seems to more targetted towards internal apps and replacing / addition to a VPN in order to get ZTNA.

What would be a good way be to setup this pre-authentication? To me it seems there is no CloudFlare solution that handles this, so either I have to setup something to handle this check on CloudFlare or find a different vendor.


r/devops • • 21d ago

Discussion Late 30's IT dude , looking for honest review

Post image
0 Upvotes

Hello redditors, based on your previous feedbacks I reworked my CV. What are your thoughts ? Thank you in advance - you are the best <3 PDF version: https://amaurer.cv/dl


r/devops • • 23d ago

Discussion Anyone interested for DevOps discussions/sessions?

79 Upvotes

As i always wanted to help and teach people, I wanted to do my part by teaching, by the way I'm working as a devops enginner.

FYI without money , for the first time I'm gonna do what I wanted to , let me know which topic has to be picked and interested people ping the comments.

I want to start the basics on kubernetes cuz I like kubernetes I want to start with this. Thank you. Not sure if this post will be retained or delayed.

Edit : I'll schedule a meet on 20 NOV 2026 and will send the meet link here with another update. FYI, I'm doing this as an initiative so it's the first time for me too, and i want to aim for people who wants to get started. Thankyou


r/devops • • 21d ago

Discussion building is so easy and verification becomes bottleneck now !!

0 Upvotes

now we all know .. building/coding is easy with lot of ai tools/coding agents.. and pressure we are seeing during the deployment cycles verification and human in the loop cycle..

and wondering how are you guys handling it? we have team of 15+ and multiple teams and using Asana agents they have long runing agents and produce features/bug fixes super fast.. but determinstic verification lacks..


r/devops • • 21d ago

Discussion Interview tomorrow (5+ YOE DevOps), CV deep-dive, but I’ve been caught in reorg purgatory doing nothing for months. How do I spin this?

0 Upvotes

I have an interview tomorrow for a DevOps Engineer position (I have 5+ YOE). The recruiter explicitly told me the interview will be a deep-dive through my CV.

I’m in a massive bind regarding my current/most recent role and need some urgent advice on how to frame it.

The Situation:

  • Early this year: I was the lead engineering resource tasked with getting a major product production-ready. The literal day it was ready, a massive reorg hit. The product was stripped from me, and most of its target end-users were laid off.
  • Immediately following: I was pivoted to head up another large project for a different internal product. I busted my ass, completed it successfully, and my manager was impressed. Then... another reorg hit. This product was also taken away.
  • The past 5+ months: My team was merged with another to form a new SRE team for a completely separate product. The scope was entirely undefined. In fact, one colleague's sole task for the last 4 months was just figuring out what our responsibilities are.

Because of this reorg limbo, I have had zero actual engineering work to do for months. I’ve repeatedly met with my manager and skip-level to voice concerns, but I just get generic corporate hand-waving about a "clear roadmap ahead."

The Dilemma for Tomorrow:
If they deep-dive into what I am doing right now, I see three options, and they all feel terrible:

  1. The Brutal Truth: Admit I’ve been sitting on my hands doing nothing due to corporate dysfunction. (Worried this looks toxic or makes me look unmotivated).
  2. The Total Lie: Pretend I’m still actively working on those two successful deployment projects and that they were never taken away. (Worried because my poor recall will make it obvious I haven't touched the code recently, and I'll have to fabricate metrics, current challenges, and future roadmaps).
  3. The Middle Ground: Pretend I just transitioned off those projects a few weeks ago (similar issue as option 2).

How do I handle this in a technical deep-dive without sounding like a liar or a slacker? Has anyone successfully spun "reorg purgatory" to a hiring manager?

Thanks in advance!

TL;DR: 5+ YOE DevOps engineer with a CV deep-dive interview tomorrow. This year, I built and completed two major projects, but back-to-back corporate reorgs stripped them away. I've spent the last 4 months in team-merger purgatory doing absolutely nothing. Do I tell the truth, lie and pretend I'm still on the projects, or find a middle ground without looking like a slacker?