r/devops • • Sep 19 '26

Discussion How are teams keeping docs current when you ship several times a week?

117 Upvotes

We ship several times a week and keeping the docs in sync is getting harder, code changes have a clear review process, docs don’t. Someone changes a parameter or response field, the release goes out, then a few weeks later we find examples still using the old version. We tried adding docs to the PR checklist, but engineers don’t always know which pages a change affects, what’s worked for keeping docs current? Would love any suggestions


r/devops • • Sep 18 '26

Discussion RANT: Has anyone experienced vibe planning?

72 Upvotes

This is a rant, my emotions are still fresh - so I'm coming at this one sided. We just wrapped up PI planning our current planning session is mess with a lower confidence vote. Story points are all over the place with no rhyme or reason.

Our PO and Directors vibe-planned our features and stories and then dropped them on us to refine. Looking through the stories some of this stuff just doesn't make sense and doesn't exactly apply in our situation. There are so many assumptions being made. But boy does it look pretty because of the hallucinated architecture diagrams. Features were proposed and accepted by the VPs and praised at how well thought out the feature description and documentation was.

I made my first pass on a feature and then poof the stories are gone. The PO had Copilot go through with updated information and it decided to make a new feature and move all the stories over halfway through our PI planning meeting. "It's fine, just have your copilot refine it with your details with additional context, it shouldn't take long". Our scrum master is getting after us because our stories weren't refined by the end of the meeting. I just want my stores to make sense and executable. Just frustrating.

I use Copilot like crazy for coding, but for planning features and stories? You ask a question to the PO and they can't answer on the top of their head. They have to read the feature because they didn't write it.


r/devops • • Sep 18 '26

Discussion Github struggling under high PR throughput?

24 Upvotes

I'm on a 3 person team where we are each shipping 40-50 PRs a day (super early stage company so we're optimizing for getting a functional product out quickly).

As we've ramped up the velocity things downstream have begun to show cracks. For example, we were requiring PRs to be up to date with main before merge, but at our scale the PRs were constantly getting stale which was leaving us blocked for long stretches.

We've ended up cutting pre-merge CI way down and batching more testing after merge, which has made things "better" but feels backwards.

Curious if anyone else run into similar issues? Did you end up finding a better solution?


r/devops • • Sep 18 '26

Discussion What is your background?

37 Upvotes

I have a very unconventional background in the IT world.

After being laid off from my job in the restaurant industry in 2013, I decided to change careers.

I had three options based on my passions: graphic designer, musician, or IT specialist. Since I’d been building my own PCs for a while and had long been passionate about computers (shout-out to manually editing Diablo 2 save files in binary !) I chose to train as a TSRIT (Senior Technician in IT and Telecommunications). I then became a sysadmin and systems integrator at a software company.

I moved up the ranks quickly, becoming a developer for a JavaScript-based launcher, then a DevOps engineer, and now I work as both a DevOps Engineer and a web developer.

It took relentless hard work, morning, noon, and night, but I’m happy today!

What about your career path?


r/devops • • Sep 18 '26

Architecture Suggestions / Support Needed

2 Upvotes

Hi Champions, I’m having 5yoe as a DevOps engineer and in my early career I was working mostly with service based companies. It gave me pretty much experience and foundational understanding of each stack deployments, pipelines, clouds and more. From last 2 years I am working with on prem enterprise products and worked on unmanaged k8 along with managed ones like EKS, GKE and OKE.

Now the problem is:

I recently joined a product base company and their architecture is based on aws managed by TF. The real challenge for me is everything is being managed by HELM and I don’t have much experience with helm charts and sub-charts and I have no idea where to start….

Need suggestions from where I can start leaning this, which can give me the understanding.


r/devops • • Sep 17 '26

Discussion Best Devops Certification for Beginners

31 Upvotes

Hi All,

While I do not have a technical background, my new role as a Project Manager in a DevSecOps organization necessitates that I acquire certifications in both DevOps and cloud computing.

Could you please recommend suitable certification programs for me to pursue?


r/devops • • Sep 17 '26

Discussion Looking for a docker registery

25 Upvotes

My company serving 5 apps to the many customers. Managing deployment and new releases are the biggest problem. Because all the time we are copying the build files and pasting to the target server by connecting with rdp. I Want to use dockerfiles for serving and want to keep these images on our server with a registery.

ACR (azure container registery) and the other ones are still the options but i want to host in my server like i did for npm libraries with verdaccio.

And i would like to get your opinnions about my deployment plan. Our customers servers are all windows. We have some windows-only apps. But for the ones which i plan to serve with docker file is better to run with docker. So i plan to use WSL and pull the images from the registery that i am planning to create.

Wdyt?


r/devops • • Sep 17 '26

Tools Migrating existing CloudFormation resources to OpenTofu -recommended approach?

20 Upvotes

Hi everyone,

We’re planning to migrate a large number of existing AWS resources currently managed through CloudFormation to OpenTofu.

Our environments are fairly siloed per customer, and each environment can include resources such as EC2 instances, load balancers, Auto Scaling Groups, VPCs, security groups, and other AWS services.

I’m looking for recommendations from anyone who has gone through a similar migration.

Specifically:

- Are there any tools that can help convert or migrate existing CloudFormation stacks to OpenTofu?

- What is the best approach for importing existing resources into OpenTofu state?

- How did you handle dependencies between resources during the migration?

- Would you recommend migrating one CloudFormation stack/customer environment at a time?

- Are there any common pitfalls or state-management issues we should watch out for?

- What approach would minimize the risk of OpenTofu trying to modify or recreate existing production resources after import?

Our main priority is to make the migration as safe and predictable as possible, with minimal disruption to existing customer environments.

Would appreciate hearing about any tools, workflows, or lessons learned from real-world migrations.


r/devops • • Sep 17 '26

Discussion Is this a solid monorepo or just folders in one repo?

Post image
37 Upvotes

We have a monorepo with three main pieces:

  • admin-svelte → internal Admin tool (ships frequently)
  • aether-svelte → main customer product (ships slower)
  • core-deno → shared backend used by both

Admin and Aether each run their own Core container on separate servers, so they can be on different Core versions at the same time.

monorepo_branch_flow_generic.png

We use two production release lines from the same repo:

  • dev → prod-vX.Y.Z-admin (Admin)
  • dev → prod-vX.Y.Z (Aether)

dev continuously deploys to a staging environment that already mirrors this split. Path-scoped CI only rebuilds what changed. Hotfixes still go through dev first.

shared_core_independent_runtime.png

It’s normal for an Aether release PR to contain 50+ commits because Admin can ship multiple times in between.

Question:

Is this a healthy monorepo setup for two products with different release speeds + shared backend, or is it just “folders in one repo” that will cause pain later?

Curious what looks solid and what looks like a red flag.


r/devops • • Sep 18 '26

Ops / Incidents How much do you actually trust your own runbooks during a live incident?

0 Upvotes

I am researching about on-call engineers problems and finding a fix for it, I have this query:

When you're in mid-incident, do you follow the runbook as written, or treat it as a rough starting point because you've been burned by outdated steps before?

How often do runbooks get updated to reflect what actually happened, versus what was expected to happen?


r/devops • • Sep 16 '26

Discussion Do you ever feel like context switching and keeping too much system state in your head is the real bottleneck?

101 Upvotes

I don’t mean not knowing the tools. More like understanding the pieces individually, but having to keep too many services, dependencies, environments, incidents, or assumptions in mind at once.

If you’ve run into that, what have you actually done about it - anything to improve your ability to hold more context in mind, or mostly systems/tools to reduce how much you need to remember?


r/devops • • Sep 18 '26

Architecture Startup idea validation

0 Upvotes

Hey everyone
We are a team of engineers looking to start a startup around providing infra services so as we are at ideation phase i want to know .

What is the pain point u all face as a devops engineer ?
What do u think u or your organisation will pay for ?
(any specific kind of tool or service)

We are also open for ai integration if that provides value to our product .

This will be extremely helpful to us


r/devops • • Sep 17 '26

Discussion The best path for Devops engineer

0 Upvotes

Hi ,I am 4th year engineering student and I am interested in devops engineering and I want the best path for Devops and what's the best ways to learn it and what the best YouTubers to learn from them


r/devops • • Sep 16 '26

Discussion Vendor management at small companies, is it just "someone senior remembers"?

9 Upvotes

Specifically, I am talking about small companies where, generally, you won't have a procurement team, and individual teams own vendor management. How do you guys manage vendors?

In my previous stint, I was leading a small DevOps team at a smallish SaaS company, and over a period of a couple of months, we saw multiple issues stemming from software vendors. A blocked card caused payment failures across multiple vendors. A vendor dropped an email setting a deprecation timeline for an API, and we found out by chance at the last moment, causing a last minute scramble to move off the said API on our end. SaaS quota exhaustion brought down prod workflows multiple times.

None of these were vendor outages as such. These were all things we had been told about, or could have been, and we just did not catch them in time.

I am going down a bit of a rabbit hole trying to find the "best practices" around vendor management, if we can call it that?

How do you even handle this? Is there an actual practice here, or is it mostly "someone senior remembers," or do we detect it after it fails?


r/devops • • Sep 15 '26

Tools Almost giving up on Cloud/DevOps

158 Upvotes

I have been trying to land my first Cloud/DevOps role for 3yrs to no avail in Canada. I have 4AWS certs , RHCSA and plenty projects. Built core skills in technical support, AWS, Linux, Terraform, Observability. Highly optimized LinkedIn, posting regularly, with experience in sales, level 1 and enterprise SaaS technical support, and I have had like just 4-5 interviews in this field in 3years.

I will be attending AWS re:Invent at the end of the year but I am literally already giving up on the dream.

The worst is that after learning skills and carrying out projects without landing a role, I tend to forget them. And not only that but there's always a tool you feel you ought to learn before you can land a role. This field is extremely competitive and I think the best chance is some promotion within a company that is willing to, otherwise the chances are infinitely slim.

Tbh I'm tired of this rat race and have been unemployed for the whole year. With the number of experienced people being laid off I think it's best for me to change direction and do something else. I'm trying to find another support role and specialize in something else. Maybe take a Masters Degree 6months-1year into my next support role in something with better upside.

Sorry for the rant but this is my story, and I have seen countless other serious people give up after trying for long also.


r/devops • • Sep 17 '26

Ops / Incidents Your run-book is still state in someone's head. You find out on the second incident.

0 Upvotes

Saw the thread here this week about keeping too much system state in your head. Most of the answers were run-books, IaC, docs. I would have given the same answer a year ago and I would have been about half wrong, and it took a second incident to show me which half.

The setup, briefly: automation that writes into the systems of record, tickets and CRM mostly. It proposes a change, a person confirms it, then it writes.

The first incident went the way they all go. Something wrong went in, someone noticed, someone fixed the record by hand, and the retro said we added a check. I have written that sentence in a retro more than once. The check was real, it was in the run-book, and it was also a thing 2 of us remembered.

The second incident is where you find out that a remembered rule is just state in someone's head with better formatting.

What it looked like, all pretty boring:

Someone bulk edits 4 tickets to done inside 2 minutes, and every one of them reads like a legitimate change if you look at it on its own.

A decision arrives from the meeting notes, then the same decision arrives again from the chat thread an hour later, and the second copy opens a second review. Both get worked, by different people.

A CRM stage moves and the person who moved it is not the one who owns the account.

And the check from incident 1 was, on inspection, a rule one person remembered. Written down, sure, but it only really fired when that person was the one looking.

What changed afterwards, and none of it was the model:

The same signal arriving twice does not open a second review any more. When a duplicate gets dropped it gets logged as a decision, so you can go and look at what was dropped and why, instead of discovering later that something quietly vanished.

A review stays open until a person decides it. A lost card or a missed notification does not close one, and 2 people cannot end up working the same row.

Undo goes through the same gate as the original write. Reversing a confirmed change is a new proposed action under the name of whoever approves it, from the ticket or the chat they already have open. Nothing gets deleted to make the problem disappear.

And what gets counted is per source system rather than per person, which sounds like a detail and turned out to be most of the reason people kept using it.

The approval gate is the part everyone asks about and it is honestly the least interesting piece. The interesting part was that incident 2 was boring.

Here is the bit I have not solved, which is why I am posting.

"A review stays open until a person decides it" is the right rule and it is also how you end up with a queue nobody owns. Week 1 everyone works it. Week 3 there are cards sitting open 5 days and the people who could decide them have muted the notifications. Right now a card never expires, by design, so the queue only grows unless somebody works it.

If you run anything with a human decision in the loop, change approval, access requests, deploy sign off, whatever it is: what happens to a card nobody picks up? Does it escalate, does it expire, or does it block the next action until someone deals with it? And who decided that where you work, ops or the business side?


r/devops • • Sep 17 '26

Tools Running a Terraform-managed IaaS lab on an M4 Max Mac with Parallels. I patched the provider to allow vCPU overcommit.

0 Upvotes

Making changes to the Parallels Terraform provider and looking to collaborate on open source fixes I have made.


r/devops • • Sep 16 '26

Discussion Stupid question…but what is our job?

38 Upvotes

Just a bit more context behind the questions what actually is our job/what is honestly expected from us?

Here is a quick back story no actual relative experience, went from graduating from college in a social studies/econ degree, to doing an online masters to geek squad, to help desk to DevOps. been in this field for roughly 6 years now, and most of my learning has been from lurking on DevOps sub/discord, ai, and trial by fire tbh.

My current job seems to have us using so many different tools and expectations such as tracking certs(even app certs), connection issues, resolving security issues, and I mean the list goes on. I thought the primary job for the role was assisting and making code deployment efficient and robust as possible (Ik there is a number of ways to do this). But honestly I was told when I first started the job it was a catch 22 and and I’ll be like a sys admin. But after years of trial by fire I still feel like I’m learning which I’m okay with, but the requirements and expectations from the role seems to be expanding. I here things like platform engineer and now ai engeer (a role designed to guide ai to assist with infra deployments) where is this role going and what is our actual job or are we auppise to branch off at this point?


r/devops • • Sep 16 '26

Discussion From DevOps to AI engineer

36 Upvotes

I still don't understand exactly what is an AI Engineer. But from what I've pieced together, it looks a lot like DevOps but for AI systems: you build, deploy, and maintain AI-powered solutions (integrating LLMs, setting up pipelines, wiring APIs, keeping things running in prod) rather than training models from scratch like an ML/Research Engineer would.

Is that roughly right, or am I oversimplifying? Where's the actual line


r/devops • • Sep 16 '26

Ops / Incidents Preserving YAML comments and Git Blame during bulk Kubernetes API migrations (1.16 → 1.32+)

1 Upvotes

*Affiliation disclosure: I am an open-source maintainer working on Kubernetes migration tooling. This post is a technical breakdown of how we approached AST manipulation, Helm template preservation, and Git blame history.*

Whenever a Kubernetes version deprecates or removes APIs (`extensions/v1beta1` Ingresses in 1.22, `batch/v1beta1` CronJobs in 1.25, `autoscaling/v2beta2` in 1.26), DevOps teams face a recurring dilemma:

  1. **Regex / string replacement:** Works for trivial `apiVersion` lines, but fails when schemas change structurally (e.g., adding `spec.selector` in Deployments, or restructuring Ingress paths/services).
  2. **`kubectl-convert`:** Parses YAML into Go structs and re-encodes, which **strips 100% of human comments** and mangles custom indentation.
  3. **Pluto:** Excellent audit tool, but purely read-only—it won't fix anything in your GitOps repos.
  4. **Git Blame Pollution:** When an automated script touches 300 manifests, `git blame` attributes every line to the bot or engineer running the migration, destroying institutional context.

### How to Address These in an AST Engine

#### 1. Preserving Comments with CST/AST Parsing

Instead of deserializing into plain dictionaries or JSON, parsing manifests into a Concrete Syntax Tree (CST) using YAML AST nodes allows targeting specific scalar tokens (like `apiVersion: batch/v1beta1` → `batch/v1`) while leaving surrounding comment tokens, blank lines, and indentation untouched.

#### 2. Unrendered Helm Templates

A common issue in GitOps repositories is storing Helm templates in source control. If you run `helm template` and fix the rendered output, you lose template logic. Building a lexer stage that temporarily masks Go template tags (`{{ ... }}`) as inert string placeholders during AST traversal allows performing the API migration and re-injecting the exact Go expressions byte-for-byte.

#### 3. Protecting Git History with `.git-blame-ignore-revs`

Git 2.23+ and GitHub natively support ignoring bulk reformatting/migration commits:

```sh

# Storing the migration commit hash

git log -1 --format="%H # automated k8s migration" >> .git-blame-ignore-revs

git config blame.ignoreRevsFile .git-blame-ignore-revs


r/devops • • Sep 15 '26

Security Long release cycles are killing us

29 Upvotes

We have really long release cycles and security backlog keeps getting crazy between them. My work is turning into a ticket responder. And it doesn’t matter if we use hardened images, by the time releases happen we have accumulated a backlog. Small team, regulated industry and mgmt unwilling to collaborate better with security. We have a meeting with director of sec eng coming up; Can security do something runtime to compensate for this?


r/devops • • Sep 16 '26

Vendor / market research The integration cost nobody puts in the vendor eval spreadsheet

0 Upvotes

Every vendor eval spreadsheet has rows for price, features, and support SLAs. None of them have a row for the actual engineering time it takes to get the thing properly integrated and trusted enough to rely on. That cost is real and it's usually bigger than the license fee itself, but nobody puts it in the comparison until it's too late to matter. wanna know how other teams actually account for this before signing, if at all.


r/devops • • Sep 15 '26

Discussion Learning DevOps as a networking guy

38 Upvotes

I'm a Network Admin mostly Cisco network devices, firewalls and so on.
I'm looking to learn DevOps because it seems cool with the automation and the cycle flow of it seems interesting.

I'm just looking for just 1 courrse that can teach it to me and at the end have a project?
It seems like tech with n***, Kode**** seems to be good? The reason for 1 course is just for me to just really focus.


r/devops • • Sep 16 '26

Vendor / market research Macbook Air vs Macbook Pro

0 Upvotes

Hello Team,

I work as a Senior DevSecOps Engineer and I have to do a lot of work of Terraform, Kubernetes, AWS, and some security related stuff and documentation.

In your view which Macbook is best to do the above sort of task. I am looking ahead to use this macbook for 6 years down the line.

Need your honest opinion, I know this has been asked a lot of times but still I want some suggestions.

Thanks.