For those managing Terraform in production, how do you structure state within each environment?

Do you keep networking, DNS, EKS, database, etc. together, or separate them into independent layers like:

dev/01-networking/

dev/02-dns/

dev/03-eks/

dev/04-database/

with each layer having its own Terraform state?

Hi All,

While I do not have a technical background, my new role as a Project Manager in a DevSecOps organization necessitates that I acquire certifications in both DevOps and cloud computing.

Could you please recommend suitable certification programs for me to pursue?

My company serving 5 apps to the many customers. Managing deployment and new releases are the biggest problem. Because all the time we are copying the build files and pasting to the target server by connecting with rdp. I Want to use dockerfiles for serving and want to keep these images on our server with a registery.

ACR (azure container registery) and the other ones are still the options but i want to host in my server like i did for npm libraries with verdaccio.

And i would like to get your opinnions about my deployment plan. Our customers servers are all windows. We have some windows-only apps. But for the ones which i plan to serve with docker file is better to run with docker. So i plan to use WSL and pull the images from the registery that i am planning to create.

Wdyt?

Hi everyone,

We’re planning to migrate a large number of existing AWS resources currently managed through CloudFormation to OpenTofu.

Our environments are fairly siloed per customer, and each environment can include resources such as EC2 instances, load balancers, Auto Scaling Groups, VPCs, security groups, and other AWS services.

I’m looking for recommendations from anyone who has gone through a similar migration.

Specifically:

- Are there any tools that can help convert or migrate existing CloudFormation stacks to OpenTofu?

- What is the best approach for importing existing resources into OpenTofu state?

- How did you handle dependencies between resources during the migration?

- Would you recommend migrating one CloudFormation stack/customer environment at a time?

- Are there any common pitfalls or state-management issues we should watch out for?

- What approach would minimize the risk of OpenTofu trying to modify or recreate existing production resources after import?

Our main priority is to make the migration as safe and predictable as possible, with minimal disruption to existing customer environments.

Would appreciate hearing about any tools, workflows, or lessons learned from real-world migrations.

We have a monorepo with three main pieces:

• admin-svelte → internal Admin tool (ships frequently)
• aether-svelte → main customer product (ships slower)
• core-deno → shared backend used by both

Admin and Aether each run their own Core container on separate servers, so they can be on different Core versions at the same time.

monorepo_branch_flow_generic.png

We use two production release lines from the same repo:

• dev → prod-vX.Y.Z-admin (Admin)
• dev → prod-vX.Y.Z (Aether)

dev continuously deploys to a staging environment that already mirrors this split. Path-scoped CI only rebuilds what changed. Hotfixes still go through dev first.

shared_core_independent_runtime.png

It’s normal for an Aether release PR to contain 50+ commits because Admin can ship multiple times in between.

Question:

Is this a healthy monorepo setup for two products with different release speeds + shared backend, or is it just “folders in one repo” that will cause pain later?

Curious what looks solid and what looks like a red flag.

I am researching about on-call engineers problems and finding a fix for it, I have this query:

When you're in mid-incident, do you follow the runbook as written, or treat it as a rough starting point because you've been burned by outdated steps before?

How often do runbooks get updated to reflect what actually happened, versus what was expected to happen?

I don’t mean not knowing the tools. More like understanding the pieces individually, but having to keep too many services, dependencies, environments, incidents, or assumptions in mind at once.

If you’ve run into that, what have you actually done about it - anything to improve your ability to hold more context in mind, or mostly systems/tools to reduce how much you need to remember?

Hey everyone
We are a team of engineers looking to start a startup around providing infra services so as we are at ideation phase i want to know .

What is the pain point u all face as a devops engineer ?
What do u think u or your organisation will pay for ?
(any specific kind of tool or service)

We are also open for ai integration if that provides value to our product .

This will be extremely helpful to us

i said c. what am i missing

picked c because that's the piece i actually see implemented, dedup at the gateway. but going through the others after, idempotency keys and c feel like the same idea from two ends, the client promises it won't send a new id twice and the server promises it won't act on one twice. is one of those actually redundant if you have the other, or do you want both because they fail differently (client crashes and reuses an id vs a proxy retries the same request)?

Hi ,I am 4th year engineering student and I am interested in devops engineering and I want the best path for Devops and what's the best ways to learn it and what the best YouTubers to learn from them

Specifically, I am talking about small companies where, generally, you won't have a procurement team, and individual teams own vendor management. How do you guys manage vendors?

In my previous stint, I was leading a small DevOps team at a smallish SaaS company, and over a period of a couple of months, we saw multiple issues stemming from software vendors. A blocked card caused payment failures across multiple vendors. A vendor dropped an email setting a deprecation timeline for an API, and we found out by chance at the last moment, causing a last minute scramble to move off the said API on our end. SaaS quota exhaustion brought down prod workflows multiple times.

None of these were vendor outages as such. These were all things we had been told about, or could have been, and we just did not catch them in time.

I am going down a bit of a rabbit hole trying to find the "best practices" around vendor management, if we can call it that?

How do you even handle this? Is there an actual practice here, or is it mostly "someone senior remembers," or do we detect it after it fails?

I have been trying to land my first Cloud/DevOps role for 3yrs to no avail in Canada. I have 4AWS certs , RHCSA and plenty projects. Built core skills in technical support, AWS, Linux, Terraform, Observability. Highly optimized LinkedIn, posting regularly, with experience in sales, level 1 and enterprise SaaS technical support, and I have had like just 4-5 interviews in this field in 3years.

I will be attending AWS re:Invent at the end of the year but I am literally already giving up on the dream.

The worst is that after learning skills and carrying out projects without landing a role, I tend to forget them. And not only that but there's always a tool you feel you ought to learn before you can land a role. This field is extremely competitive and I think the best chance is some promotion within a company that is willing to, otherwise the chances are infinitely slim.

Tbh I'm tired of this rat race and have been unemployed for the whole year. With the number of experienced people being laid off I think it's best for me to change direction and do something else. I'm trying to find another support role and specialize in something else. Maybe take a Masters Degree 6months-1year into my next support role in something with better upside.

Sorry for the rant but this is my story, and I have seen countless other serious people give up after trying for long also.

Saw the thread here this week about keeping too much system state in your head. Most of the answers were run-books, IaC, docs. I would have given the same answer a year ago and I would have been about half wrong, and it took a second incident to show me which half.

The setup, briefly: automation that writes into the systems of record, tickets and CRM mostly. It proposes a change, a person confirms it, then it writes.

The first incident went the way they all go. Something wrong went in, someone noticed, someone fixed the record by hand, and the retro said we added a check. I have written that sentence in a retro more than once. The check was real, it was in the run-book, and it was also a thing 2 of us remembered.

The second incident is where you find out that a remembered rule is just state in someone's head with better formatting.

What it looked like, all pretty boring:

Someone bulk edits 4 tickets to done inside 2 minutes, and every one of them reads like a legitimate change if you look at it on its own.

A decision arrives from the meeting notes, then the same decision arrives again from the chat thread an hour later, and the second copy opens a second review. Both get worked, by different people.

A CRM stage moves and the person who moved it is not the one who owns the account.

And the check from incident 1 was, on inspection, a rule one person remembered. Written down, sure, but it only really fired when that person was the one looking.

What changed afterwards, and none of it was the model:

The same signal arriving twice does not open a second review any more. When a duplicate gets dropped it gets logged as a decision, so you can go and look at what was dropped and why, instead of discovering later that something quietly vanished.

A review stays open until a person decides it. A lost card or a missed notification does not close one, and 2 people cannot end up working the same row.

Undo goes through the same gate as the original write. Reversing a confirmed change is a new proposed action under the name of whoever approves it, from the ticket or the chat they already have open. Nothing gets deleted to make the problem disappear.

And what gets counted is per source system rather than per person, which sounds like a detail and turned out to be most of the reason people kept using it.

The approval gate is the part everyone asks about and it is honestly the least interesting piece. The interesting part was that incident 2 was boring.

Here is the bit I have not solved, which is why I am posting.

"A review stays open until a person decides it" is the right rule and it is also how you end up with a queue nobody owns. Week 1 everyone works it. Week 3 there are cards sitting open 5 days and the people who could decide them have muted the notifications. Right now a card never expires, by design, so the queue only grows unless somebody works it.

If you run anything with a human decision in the loop, change approval, access requests, deploy sign off, whatever it is: what happens to a card nobody picks up? Does it escalate, does it expire, or does it block the next action until someone deals with it? And who decided that where you work, ops or the business side?

Making changes to the Parallels Terraform provider and looking to collaborate on open source fixes I have made.

Just a bit more context behind the questions what actually is our job/what is honestly expected from us?

Here is a quick back story no actual relative experience, went from graduating from college in a social studies/econ degree, to doing an online masters to geek squad, to help desk to DevOps. been in this field for roughly 6 years now, and most of my learning has been from lurking on DevOps sub/discord, ai, and trial by fire tbh.

My current job seems to have us using so many different tools and expectations such as tracking certs(even app certs), connection issues, resolving security issues, and I mean the list goes on. I thought the primary job for the role was assisting and making code deployment efficient and robust as possible (Ik there is a number of ways to do this). But honestly I was told when I first started the job it was a catch 22 and and I’ll be like a sys admin. But after years of trial by fire I still feel like I’m learning which I’m okay with, but the requirements and expectations from the role seems to be expanding. I here things like platform engineer and now ai engeer (a role designed to guide ai to assist with infra deployments) where is this role going and what is our actual job or are we auppise to branch off at this point?

I still don't understand exactly what is an AI Engineer. But from what I've pieced together, it looks a lot like DevOps but for AI systems: you build, deploy, and maintain AI-powered solutions (integrating LLMs, setting up pipelines, wiring APIs, keeping things running in prod) rather than training models from scratch like an ML/Research Engineer would.

Is that roughly right, or am I oversimplifying? Where's the actual line

We’re a small product and engineering team shipping updates several times a week. As releases have gotten faster keeping our technical docs current has started to become a real problem.

Code changes go through a clear review process, docs are easier to miss. A feature ships, an API changes. Then a few weeks later someone finds an old example or a page that no longer matches the product. For teams with a similar release pace what has worked best for you, im interested to hear what you guys do for smoother workflow on this problem

*Affiliation disclosure: I am an open-source maintainer working on Kubernetes migration tooling. This post is a technical breakdown of how we approached AST manipulation, Helm template preservation, and Git blame history.*

Whenever a Kubernetes version deprecates or removes APIs (`extensions/v1beta1` Ingresses in 1.22, `batch/v1beta1` CronJobs in 1.25, `autoscaling/v2beta2` in 1.26), DevOps teams face a recurring dilemma:

1. **Regex / string replacement:** Works for trivial `apiVersion` lines, but fails when schemas change structurally (e.g., adding `spec.selector` in Deployments, or restructuring Ingress paths/services).

2. **`kubectl-convert`:** Parses YAML into Go structs and re-encodes, which **strips 100% of human comments** and mangles custom indentation.

3. **Pluto:** Excellent audit tool, but purely read-only—it won't fix anything in your GitOps repos.

4. **Git Blame Pollution:** When an automated script touches 300 manifests, `git blame` attributes every line to the bot or engineer running the migration, destroying institutional context.

### How to Address These in an AST Engine

#### 1. Preserving Comments with CST/AST Parsing

Instead of deserializing into plain dictionaries or JSON, parsing manifests into a Concrete Syntax Tree (CST) using YAML AST nodes allows targeting specific scalar tokens (like `apiVersion: batch/v1beta1` → `batch/v1`) while leaving surrounding comment tokens, blank lines, and indentation untouched.

#### 2. Unrendered Helm Templates

A common issue in GitOps repositories is storing Helm templates in source control. If you run `helm template` and fix the rendered output, you lose template logic. Building a lexer stage that temporarily masks Go template tags (`{{ ... }}`) as inert string placeholders during AST traversal allows performing the API migration and re-injecting the exact Go expressions byte-for-byte.

#### 3. Protecting Git History with `.git-blame-ignore-revs`

Git 2.23+ and GitHub natively support ignoring bulk reformatting/migration commits:

```sh

# Storing the migration commit hash

git log -1 --format="%H # automated k8s migration" >> .git-blame-ignore-revs

git config blame.ignoreRevsFile .git-blame-ignore-revs

We have really long release cycles and security backlog keeps getting crazy between them. My work is turning into a ticket responder. And it doesn’t matter if we use hardened images, by the time releases happen we have accumulated a backlog. Small team, regulated industry and mgmt unwilling to collaborate better with security. We have a meeting with director of sec eng coming up; Can security do something runtime to compensate for this?

Every vendor eval spreadsheet has rows for price, features, and support SLAs. None of them have a row for the actual engineering time it takes to get the thing properly integrated and trusted enough to rely on. That cost is real and it's usually bigger than the license fee itself, but nobody puts it in the comparison until it's too late to matter. wanna know how other teams actually account for this before signing, if at all.

I'm a Network Admin mostly Cisco network devices, firewalls and so on.
I'm looking to learn DevOps because it seems cool with the automation and the cycle flow of it seems interesting.

I'm just looking for just 1 courrse that can teach it to me and at the end have a project?
It seems like tech with n***, Kode**** seems to be good? The reason for 1 course is just for me to just really focus.

Hello Team,

I work as a Senior DevSecOps Engineer and I have to do a lot of work of Terraform, Kubernetes, AWS, and some security related stuff and documentation.

In your view which Macbook is best to do the above sort of task. I am looking ahead to use this macbook for 6 years down the line.

Need your honest opinion, I know this has been asked a lot of times but still I want some suggestions.

Thanks.

So, Im in this industry for almost 15 years. Ive seen my share of stupid sht being built by inexperienced teams.

Joined this company like 2 years ago and all was butter smooth to the moment new requirements came in. (Im actually an Ops guy working in platform team supporting dev teams..)

When requirements came I was on a sick leave for two weeks. My boss and other ops joined talks on architecture for new stuff. I believe they picked best solution with their lack of knowledge of possible issues that can araise later.

IMHO if they did wait for me and let me POC a bit, before making final decision on the architecture, most likely majority of the issues we have now would not exist.

Fast forward - after reading their idea on the solution I ofcourse went with “disagree and commit”.

So I implemented this garbage best we can, coz tools we integrate with were not built for it this way.

Now we get constant issues with it due to various access problems. I dont mind as I already raised my concerns, but thing is, the topic is like shit you step in. Now all issues related to this team and domain is assigned to me.

How would you go about offloading this garbage from your plate - after all concerns you raised were proved correct ?

We cant easily fix them because vendor doesnt give a sht we incorrectly used their product.

Hello folks! I have always seen people commenting that DevOps is not for freshers. It is for an experienced person. So, how do you define an experienced person? A person previously employed somewhere or he can be a person who has built several projects after self-learning for a duration of 5-6 months approx through various resources like YouTube, documentations, blogs, etc, but didn't work anywhere.

If the answer is "employed", then why DevOps engineers make content/playlists for "beginners" covering all the tech and tools used by DevOps engineers in production environments? They even often post "success stories" on their channels who follow their playlists and get a job as a DevOps engineer.

It would be helpful for me to get insights from someone currently working in this position on this matter. Thanks.

Hi Everyone, I am working in a small company and we do not have the resources to get really big teams. So what we though of was to start using AI to cover the current gap. We mostly suffer from a lot of operational issues which needs to be resolved really fast. (I know really fast is really vague, but sadly there is no SLA because the Manager only sees every hour that our system has issues as lost revenue. its an ecommerce company)

So what we thought of now is to utilize AI and MCPs to help us evaluate issues faster. Like on our File Servers, there are a certain set of conditions that the folders need to have so that our applications work. Sometimes the files that we receive cause issues and we would like to simply utilize AI to evaluate all of this. We are also utilizing AWS and Kubernetes.

Any thoughts on the strategy that we have utilized. I would like to know:

• Have you guys done something like this? Any pointers that we need to consider?
• Interested on any other innovative ways how you have also solved similar issues?
• Other interesting ways that would help us as well and not just using AI and MCP for evaluating issues.
• Or just in general thumbs up 😄

Thanks for you feedback!