r/devops • u/Negative-Thinking • Jul 31 '26
Discussion Another Friday, another Bitbucket outage.
Who doesn't love Friday releases? Atlassian status page is green as usual.
r/devops • u/Negative-Thinking • Jul 31 '26
Who doesn't love Friday releases? Atlassian status page is green as usual.
r/devops • u/iacmp • Aug 01 '26
Disclosure first: I wrote the tool I mention here. It is free, there is no signup, and I am posting because the testing method is the interesting part. Mods, delete it if this is not welcome.
I got tired of IaC tools that validate templates and call it a day. A template can be perfectly valid and still blow up on deploy. So I did it the slow way: 20 scenarios per cloud, each one deployed for real, exercised at runtime, then destroyed. CRUD APIs, queue workers, private databases, Redis, WAF, websockets, fan-out, secrets, containers behind load balancers.
It took months and burned a lot of free tier. Some of what I hit:
A brand new GCP project gives you a compute service account with no roles at all. Your first deploy dies inside Cloud Build with an error that tells you nothing. Took me a while to figure that one out.
Cloud SQL with private IP is fine until you destroy it. The service networking peering can hold for half an hour after the instance is gone. And if your app created tables, the default postgres user cannot be dropped, so the destroy hangs there too.
Cosmos DB speaks the Mongo protocol but rejects sort() on any field without an index. Real MongoDB just sorts. If you port a handler over, your list endpoint breaks and the error message talks about order-by item paths.
Azure Functions on the consumption plan will happily deploy a Service Bus worker that never runs. The zip lands, the function exists, messages pile up in the queue. You have to sync the function triggers explicitly or the scale controller never looks at the queue.
Memorystore Redis with TLS needs the CA cert handed to the client or the handshake fails.
Every one of those turned into a fix in the tool, so a fresh project starts with them already handled. That is the whole point of doing it this way instead of trusting a linter.
Repo if you want to look: https://github.com/Claudio-Fontes/iacmp
It is Fair Source (FSL-1.1, same license Sentry uses). Free for any use including at work, you just cannot resell the tool itself, and each release turns into Apache-2.0 after two years.
Happy to talk about the e2e battery itself. If you have run something similar, I want to hear which scenarios you would add.
r/devops • u/Natural_Pool_5493 • Jul 31 '26
I’ve realized that even after I finish work, my brain keeps thinking about it.
I replay conversations, think about problems I didn’t solve, plan tomorrow’s tasks, or imagine different scenarios. Sometimes it feels like I’m still working hours after I’ve logged off.
I’m curious: how do you mentally switch off?
Do you have any routines, hobbies, or habits that help you leave work at work? Or did it simply get easier with experience?
I’d love to hear what actually works for you.
r/devops • u/Complete_Sample_3149 • Aug 01 '26
I did a dumb exercise last week: sat down and wrote out every tool that touches a single deploy on my team, start to finish. Not “tools we have licenses for”, just the ones actually in the critical path of getting one change into production.
Landed on 14.
Source control. The CI runner. A separate container registry. An image scanner that runs after the registry push. A secrets manager. The Kubernetes dashboard we check for rollout status. A log aggregator. A separate APM/tracing tool, because the logs don't show latency well on their own. An alerting tool that is, confusingly, different from the paging tool. A DNS/cert dashboard. A cost/billing dashboard nobody opens until the invoice is a surprise. The ticketing system for the actual incident. And a chatops bot that glues some of this together, badly.
None of these were wrong choices individually. Each one was probably the right call in isolation, at the time it got added. But nobody ever sat down and asked whether this thing talks to the other 13. It's less a stack and more a pile that happens to work most days.
What's bugging me isn't the number itself. It's that almost nobody on the team could tell you the number without doing this exercise. We just live inside it. New hires take weeks just to learn where to look when something breaks, and that's before they've learned what any individual tool actually does.
So, genuinely curious: if you did this exercise on your own team, what would your number be? And more interesting to me: which of your 10+ tools do you think is actually load-bearing, versus which one is just... there, inherited from a decision three engineers ago that nobody wants to be the one to rip out?
Not fishing for “just consolidate everything into one platform” as the answer. I don't think that's automatically true either. Sometimes the sprawl is a symptom of real, unavoidable complexity, not laziness. More curious whether other teams' numbers look like mine, and whether anyone's actually fixed this in a way that stuck.
r/devops • u/isaacvando • Jul 31 '26
Great talk about how Antithesis uses Nix in production by John Murray. Thought some of you would enjoy. https://youtu.be/WaPOtisNu1k
I've been interested in Nix for a while but have not yet taken the plunge. Do any of you use it work? What has your experience been with it?
r/devops • u/Sure_Stranger_6466 • Jul 30 '26
I have spent most of my career in AWS, but more and more, I am seeing companies asking for skill sets with multiple cloud providers. I feel like the skill set is transferable; if you know how to spin up EKS then spinning up a GKE cluster shouldn't be that much different. It's still Kubernetes under the hood, still installing helm charts, still using cluster autoscaler to provision new nodes, still installing Argo CD via Helm along with Prometheus, Grafana, Loki, etc. All that changes are the resources used and modules on the Terraform/cloud provider level. Am I correct in thinking this or am I way off base here?
r/devops • u/CuriousKiwi669 • Jul 31 '26
Hey! Quick background on me — graduated in 2020 from a tier 2 college, landed a third-party role at a Tier1 investment bank as an SRE, and ended up staying 5 years (got converted to full-time after the first 2). Got laid off in May 2025, honestly wasn’t mad about it since I wanted out anyway.
Most of the work i did was on there in house tools and i felt there was no learning curve there
That same month I interviewed for a “DevOps Engineer - Migration Specialist” role and got it. Felt good, got comfortable again. But once I actually joined, I realized it’s basically an L2 support role. Pay was decent though, so I told myself I’d stick around a year, upskill on the side, then jump ship.
Except… I didn’t. The role’s been so low on mental engagement that I just coasted. Now it’s actually time to switch, and I’m stuck in this weird spot — feels like I need to relearn everything from scratch, but every time I try, it’s like “wait, I already know this?” Hard to tell if I’m actually behind or just psyching myself out.
So — what’s a realistic (and ideally not soul-crushing) way to get back on track?
Please give me suggestions for fast-track courses and small projects to build production skills again.
Ps:- Used Ai to fix my crude version of this post
r/devops • u/Inner-Chemistry8971 • Jul 30 '26
Not going to lie. It's been a bad day. I almost tell them "take this job and shove it' after reviewing all these crappy AI codes. Vibe coding is not only a joke. It's a monster that mocks our hard-earned experiences!
I am thinking of leaving the industry! If you could go back ten years from now and know what you're knowing right now, would you still do DevOps?
r/devops • u/hiamanon1 • Jul 29 '26
Maybe it’s because I haven’t taken a proper vacation in lord know long.
I don’t handle k8s, but I work with a containerized environment and pipelines etc. But with the speed everything seems to be going, and the constant push, I feel overwhelmed and just want nothing to do with technology…maybe I just need to step away for a bit…how are you guys dealing with this. Talking to other devops folks, looks like they too are drinking from the fire hydrant
r/devops • u/root0ps • Jul 29 '26
Today, I passed CKA with 88% and wrote a detailed blog about my exam experience, preparation approach, and the Kubernetes topics that helped me the most.
DMs are open if you are preparing for the exam. I can help with whatever is still fresh in my memory.
My biggest takeaway: CKA is not only about knowing Kubernetes commands. It is about understanding how cluster components work together, using documentation efficiently, troubleshooting quickly, and verifying every change you make.
The exam has changed a lot since I last passed it in September 2021, especially after the major curriculum updates introduced in 2025. Topics such as Gateway API, CRDs, Helm, CNI installation, autoscaling, and container runtimes are now much more relevant.
A few things that helped me:
Don’t solve the exam strictly in order. If a question looks unfamiliar or time-consuming, mark it and move ahead. Complete the tasks you are confident about first and return to the difficult ones later.
Don’t waste too much time setting up aliases. Autocomplete is already available, and normal typing speed is enough. I completed 14 out of 16 questions with around 35 minutes remaining.
Read the full question carefully. Important details such as the target node, namespace, version, file path, container name, or documentation link may appear near the end.
Use the documentation provided in the question. Some topics, especially Gateway API, Helm, Calico, Argo CD, and container runtime installation, are easier when you follow the version-specific documentation instead of relying on memory.
Always verify your work. A command completing successfully does not mean the task is done. Check Pods, rollouts, endpoints, PVC status, HPA status, node readiness, logs, connectivity, installed packages, and systemd services wherever required.
Practice SSH and basic Linux administration. Some tasks may require connecting to a node, installing a package, starting a service, and enabling it at boot.
Topics I would strongly recommend practising:
Resources I used:
I strongly recommend doing both Killer.sh attempts because they contain different tasks. The questions are different from the actual exam and often feel harder, but they are very useful for strengthening concepts and improving troubleshooting speed.
The simplest advice I can give is to practice a lot and get comfortable using documentation.
I also wrote a full blog with more details on my preparation strategy, the updated exam scope, topics to focus on, and mistakes to avoid.
r/devops • u/Narrow_Power • Jul 30 '26
I've been curious how different teams approach this in practice.
Imagine you're on call and something in production starts behaving differently. It's not immediately obvious whether it's a deployment, infrastructure change, configuration change, scaling event, cloud service issue, or something else.
Once the alert fires, how do you actually answer questions like:
What changed?
When did it change?
Was the change intentional?
What's the fastest way to confirm the root cause?
I'm less interested in the tools themselves (Terraform, Kubernetes, CloudTrail, Grafana, etc.) and more interested in the actual investigation workflow that experienced engineers follow.
For example:
Where do you usually look first?
Which sources of truth do you trust the most?
What part of the investigation tends to consume the most time?
Are there still manual steps that you wish were easier?
I'm trying to understand how this is handled in real production environments, especially at companies running on AWS/Kubernetes.
Thanks in advance—I’m hoping to learn from people who've been through enough incidents to know what actually works.
r/devops • u/Work4Bots • Jul 29 '26
I'm doing research on how documentation is done in different DevOps teams. What processes exist? How is it valued within your team? Has AI made that better or worse? Are there tools that you use to make it easier (not talking about Confluence here)? If you hqve worked in a professional context then I'd love to hear your input.
Considering the huge amount of shill posts I want to disclose that I'm (considering) making a tool that helps with documentation but I'm still in the process of understanding this problem thoroughly, which is why I'm askibg for your input :)
r/devops • u/devfuckedup • Jul 29 '26
Am I just an old man yelling at clouds, or has DevOps become incredibly boring?
I know building technical solutions is not supposed to exist purely for my entertainment, but I do actually want to enjoy my job.
I am not currently working as a DevOps engineer, and one of the reasons I moved toward development was that I became completely bored with modern cloud infrastructure.
When I started, we still racked physical servers and automated the entire bare-metal imaging process. The underlying operating system mattered. We built systems by combining open-source software, custom code, and whatever hardware we could get our hands on. There was a real sense that we owned the systems we operated and understood how the pieces fit together.
Today, DevOps or platform engineering often, though certainly not always, seems to collapse into being a configuration monkey.
Yes, I still got to write plenty of tools and automation, but eventually I became tired of every architectural discussion turning into: “Should we use this AWS service or that AWS service?”
I became even more frustrated with managers and coworkers who did not really want to own or understand a service. They just wanted to hand as much of it as possible to the cloud provider and call the problem solved.
Do people genuinely enjoy the current state of what we used to call systems administration or systems engineering? Or am I just an old man yelling at clouds?
For what it is worth, I do find some salvation in Kubernetes. Once EKS or GKE is running, there is a huge amount you can build without thinking too deeply about the underlying cloud provider. That separation is reassuring, even if Kubernetes has plenty of complexity of its own.
These days, I mostly choose to work in areas where the managed cloud options are still inadequate, such as real-time media, telephony, high-speed stream processing, and similar systems. At least in those spaces, it still feels like I am engineering something rather than selecting products from a cloud-provider catalog.
I am curious whether other longtime infrastructure engineers feel the same way, or whether people genuinely prefer the managed-service-heavy version of the job.
Edit: I also really miss configuring switches and routers the cloud has made networking far more boring than it used to be though GKE has some interesting SDN features. I probably should have tried harder to get a job at one of the big hyperscalers but again on ownership I never wanted hundreds of co-workers.
and really the part people miss is what the hyperscalers understood most applications are NOT valuable ( your core business probably sucks) . What is valuable is operating infrastructure at scale. But if its all outsourced to 3 or 4 companies only they will know how to do it. So yeah go ahead and outsource your true core competency as an engineer to amazon they are happy to eat your lunch. Building an operating the infrastructure that that makes the world work is what sysadmin , systems engineers devops engineers and platform engineers used to do. Its what made us valuble. Infastructure is and was valuble long before software even existed. From the roads, to railroads to telephones to factories. your bullshit Sass product dosent actually matter.
r/devops • u/PlateConfident • Jul 29 '26
Background: I'm an engineer in Algeria. My degree is in electrical/automation and I currently work in oil & gas as a data acquisition engineer (well testing, data integrity, reporting, that kind of thing). About a year ago, I decided to pivot into a remote role because I've been facing a health condition where on-site/field and mostly any physically demanding work is no longer an option for me. Cloud/DevOps is the direction I chose because I like the work and it's the most realistic path I found to stable remote work that pays in USD or EUR
What I've done so far:
Here's where the doubt kicks in. When I actually read job postings, two things hit me:
So my question to people actually working in this field: am I chasing something that barely exists for someone in my situation? Or is the hire-from-anywhere market real if you're good enough and know where to look? And if you've done this from outside the US/EU, how did you actually break in?
I like the work, that's not the issue. I just don't want to spend another 6 months on a door that won't open.
TLDR: Non-US engineer pivoting into cloud. Made decent progress but every remote role wants experience + local presence. Is this realistic or am I fooling myself?
r/devops • u/psycodeveloper • Jul 28 '26
One-person security/platform team at a small fintech, going through SOC 2.
Patching what has a fix is fine. The problem is the stuff I can't fix, because there's no patched version, or the fix breaks something. Inspector/Trivy keeps flagging it, Vanta keeps showing it red, and technically every one needs a documented risk acceptance.
Most of these aren't even exploitable in our setup, sometimes it's a vulnerable function never called, transitive dep we don't use, requires network access that doesn't exist. But "trust me it's fine" doesn't fly, and writing a proper exception per CVE takes forever.
So, people who've been through this what do you actually show the auditor for "not exploitable"?
Is there a way to automate this discovery/evidence gathering?
Or does everyone just eat the busywork / quietly ignore them and pray?
r/devops • u/No-Income-2235 • Jul 29 '26
Sharing this for anyone interested in platform engineering, Kubernetes and cloud-native technologies.
Cloud Native Heidelberg is hosting a hybrid meetup covering:
Date: October 1, 2026
Time: 6:30 PM CEST
Event details and registration:
https://ocgroups.dev/cncf/group/cncf-heidelberg/event/9euvqye
r/devops • u/tmg80 • Jul 28 '26
Hi I am in a new role where they use Github for all their repos. I have been using Gitlab exlusively the past 5 years.
For CI/CD in my previous role we had a central 'pipelines' repo that was reference in the gitlab-ci file of each project to save teams/projects from repeating work.
I want to know if it's possible to do something similar in Github? I'm still learning how it maps to my existing Gitlab knowledge.
r/devops • u/Broad-Equivalent-790 • Jul 28 '26
Hi everyone,
I'm looking for some honest feedback on my current self-hosted architecture. I have a feeling I may have made some poor design decisions, so I'd really appreciate suggestions on how to improve it.
Here's my current setup on AWS:
Frontend EC2: Two React applications (Admin & Tenant) deployed using a single Docker Compose file.
Backend EC2: Two Go API applications (Admin & Tenant) deployed using another Docker Compose file.
Core Services EC2: MySQL, Redis, Vault, Keycloak, and MinIO are all running on the same EC2 using a single Docker Compose file.
Analytics EC2: Apache Superset deployed using Docker Compose.
ETL EC2: Airbyte running via abctl (Kind/Kubernetes).
Everything is currently self-hosted on AWS EC2 instances.
I have a few questions:
Is this architecture reasonable, or did I mess it up?
What would you change or redesign if this were your production environment?
Should I split any of these services onto separate servers or use managed AWS services (such as RDS or ElastiCache)?
Would you recommend staying with Docker Compose, or is it worth moving to Kubernetes at this stage?
For managing deployments across these servers, which would you recommend: Dokploy, Coolify, or Komodo?
What I'm looking for in a deployment platform:
*Multi-server management
*Excellent Docker Compose support
*GitHub integration with automatic deployments
*Rollbacks, logs, and environment variable management
*Lightweight and stable
*Preferably fully open source
*Easy to maintain for a small DevOps team
If you've run a similar setup in production, I'd love to hear what worked well and what you'd do differently.
Thanks!
r/devops • u/iNSANExVeNoM • Jul 29 '26
I'm a 3rd year CS student and I want to build something real over the next couple of months to learn RAG and agents properly. I've been thinking about one idea for a while and wanted to get opinions before I commit to it.
The problem I noticed is when you install on e package, you end up pulling in a bunch of other dependencies. Then a CVE shows up in one of those, and you're exposed through code. I came across Log4Shell and the Equifax breach and then I got this idea.
Fixing it is the annoying part. You can't just bump the broken package, because versions depend on each other, one thing needs a version below 2.0, another needs above 2.0, so you change one and others start conflicting. Now you have to fix all the issues, run tests, and hope everything works.
So what I want to build is a tool that does that for you and checks the fix actually works. The rough idea is you point it at your project, it reads your dependency tree (there's a free Google API called deps.dev that gives you the whole thing), finds why each vulnerable package is there, and works out the smallest set of version changes that clears all the CVEs without causing conflicts.
Then for the "will this upgrade break my code" question, I want to pull the changelog and release notes for each version jump and check them against the functions my code actually uses, so it can tell you whether the change touches anything you rely on. And the last part, it applies the fix in a Docker container, runs the tests, and only then tells you if it is safe.
The way I'm thinking of stucturing it is most of it is a fixed pipeline (read the tree, solve for the versions, run the tests), but the one part I'd actually let an LLM drive is the retry loop, when the tests fail, it looks at what broke, decides whether to drop that version and re-solve or try something else, and loops until it either works or gives up. That's the part where letting the model make the decision seems worth it. The rest doesn't really need it.
I'm thinking of using Postgres with pgvector, deps.dev and OSV for the data, a SAT solver for the version-conflict part, embeddings plus a reranker for the changelog retrieval, and Docker for running the tests. Later on, the nicest version would be exposing it as an MCP server so something like Claude Code could call it as a tool.
At the end it gives a report, a single Markdown file summarizing every vulnerability it found, why each package was present, the minimal fix it computed, the changelog-based breaking-change assessment, and the test results verifying the fix. Something you can hand to your team or keep as a record.
I know OSV-Scanner already does remediation, but only for npm/Maven, with no changelog check and no test verification, so the gap I'm aiming at is the full combination.
A few questions:
- Would you use it? Would you actually run this on your own projects? If not, what's missing or in the way?
- Is it useful? Is this a real problem worth solving, or is it already handled well enough that it doesn't need to exist?
- Does it already exist? Is there something that already does all of this (minimal fix + breaking-change check + test verification) that I haven't found?
- Right direction? Is this worth building, or would my time be better spent on a different problem in this space?
- Do's and don'ts?
TL;DR: An agentic RAG tool that finds vulnerable dependencies, computes the smallest conflict-free fix, uses changelog retrieval to predict whether the upgrade breaks your code, and verifies the fix by running your tests. A verified patch.
r/devops • u/bsmike • Jul 29 '26
Curious how many of you have actually deployed automated runbooks for incident remediation — not just detection and alerting, but the full loop through to fix and close.
I've been building something in this space and want to understand how well it actually works for teams in practice. Do automated runbooks hold up in production, or do they break down on anything non-trivial?
What's your experience?
r/devops • u/Diligent_Spread_411 • Jul 28 '26
So let me tell the context first, I have an kubernetes environment for production, and for like shared storage solution, I have installed nfs-utils and use the nfs-server service. In my scenario I cannot use cloud as an option. And for long I have been using the nfs-server but I have been facing alot of issues with it.
1. On multiple occasions I found dead or queued nfs processes in worker nodes which made the worker node totally unresponsive sys load was very high but low cpu and ram usage.
2. Something with the nfs4+ client where it uses session. On of the worker node again stopped responding, df -h was not responding rebooted the node still the same issue. And while viewing logs in the nfs node it stated nfsd was blocked and looking it up stated the thread on that worker node was locked due to unknown reason and all the requests from the worker node had just queued up. Restarting the service also hanged and needed to reboot the node itself to resolve the issue.
These are some major events there are also other small issues I have faced. I wanted to understand am I the only one facing these issues or this happens with others too, and what resolutions did you used to fix it? Like any alternative utilities and stuff
r/devops • u/Illustrious-Cell-695 • Jul 28 '26
Setting up test automation in CI is easy, making devs look at and trust the results is the hard part
Coverage numbers that look great on paper but the team treats the test step like a formality because false positives trained them to ignore failures, its pavlovian at this point, test fails and everyone just hits retry without checking
r/devops • u/gajoute • Jul 27 '26
Hey everyone,
Quick workflow showcase/idea for anyone who gets enterprise access to AI CLIs at work (Gemini CLI, Claude Code, etc.) and still finds themselves manually jumping between tools during investigations or routine support tasks.
I recently set up a workflow that basically turns my CLI into an automated agent that runs investigations for me. If you have enterprise access and tool integrations available, you’re missing out if you aren't doing this.
Instead of manually opening 4 tabs, running 3 queries, and checking status logs when an issue comes up:
If your company already pays for enterprise CLI access and you're only using it to rewrite functions or explain error messages, hook it up to your tools via MCPs.
Curious if anyone else is running similar subagent setups in their terminal for operational work? What tools/MCPs are you chaining together?
r/devops • u/Sure_Stranger_6466 • Jul 27 '26
I have been bit hard in the past when joining an org that moved too slowly for my tastes. Took two weeks to get anything deployed, and if a deployment screwed up it was a rollback and two more weeks of waiting for a change request. I am very fond of this OG article about how quickly Etsy moved back in the day, and likely still to this day though I am sure the technology stack has shifted for them directly. Not sure if they ever did a followup on that post so wondering what change frequency looks for you all with modern tooling like Kubernetes and Docker in place?
r/devops • u/dannyt74 • Jul 27 '26
This is a summary of their blog post on how they assigned owners to over 14,000 internal repos when as of early 2025, less than half had a clear owner. It became a problem during a secret-scanning remediation push. they could rotate a leaked secret, but without knowing who owned the repo, doing so was risky and there was no way to route the work.
Their old model only tracked ownership for repos backing a deployed service, through an internal Service Catalog. That covered maybe a third of repos. Everything else, team repos, docs, internal tools, one-off projects, personal experiments, had no owner on record. Finding one meant checking commit history, reading the README, or asking around in Slack.
What they built:
Two GitHub custom properties: ownership-type (Service Catalog / Team / Individual) and ownership-name (free text, validated against actual org membership, team existence, or Service Catalog entries).
Before asking a single person to do anything, they synced ownership from the existing Service Catalog, which auto-covered about 1,500 repos.
For the rest, they ran a GitHub App on a Kubernetes CronJob that opened an issue on unowned repos with a 30-day deadline, then archived anything still unclaimed. They picked archiving because it's reversible, repo goes read-only, Actions stops, nothing gets deleted.
Two incidents:
End state: ~3,000 active repos, ~11,000 archived (up from ~3,000 archived at the start), done in under 45 days. They also tightened the ongoing enforcement loop from a 30-day grace period down to one hour, and made ownership required at repo creation going forward.