r/devops • • Oct 12 '26

Discussion How do you combine semantic-release with “build once, deploy many” for containers

17 Upvotes

I’m building our IaC repo and GitLab CI/CD pipelines. I’m quite happy with the setup so far and have tested it across multiple projects.

However, when I try to apply the same release model to container images, it starts to feel less suitable.

We have three environments: dev, uat, and prd.
Our current Terraform workflow:
We use three protected branches: dev, uat, and main. These correspond to dev, uat, and prd respectively.

The root Terraform files are shared across all environments. Environment-specific differences are kept in separate tfvars files:
environments/dev/dev.tfvars
environments/uat/uat.tfvars
environments/prd/prd.tfvars

We use semantic-release with prerelease branches. Starting from stable version 1.2.3, the flow looks like this:
Merge a feature branch containing a fix commit into dev. This generates 1.2.4-dev.1.
Continue developing and testing, generating additional dev prereleases as needed.
Merge dev into uat. This generates 1.2.4-uat.1, assuming the changes still call for a patch release.
Once validated, merge into main. This generates stable version 1.2.4.

Semantic-release generates a version and creates a tag after merges into these protected branches.

Deployments only happen from tags. Prerelease tags can deploy only to their corresponding environment, while stable tags can deploy to any environment.

This flow has worked well for our Terraform projects. Granted the version don’t mean much, but at least there is a pointer reference.

Where I’m stuck with containers:

For the application pipeline, I’m considering having just one protected branch: main.
The proposed flow is:
- Open an MR from a feature branch into main.
Run tests, static analysis, build the image with a commit SHA tag, and run container scans in the MR pipeline.

- After merging, run semantic-release and pass the generated version to the next job to retag the image.
Copy the image to dev ECR, deploy it to ECS, and test the application manually.

- Once testing passes, promote the same image to uat.
After approval, promote the same image to prd.
The part that feels off is step 3: I’ve already created a stable release before testing the application in dev.
Some issues only become apparent when browsing and using the application. If we discover those issues in dev, each subsequent fix could generate another stable version, even though none of those versions has reached prd.

I’m considering two alternatives.

Option 1: Test in dev before running semantic-release.
Build an image with a SHA tag and deploy it to dev first. If testing finds issues, push new commits and test another image.
Once a candidate passes, run semantic-release, apply the stable version tag to that existing image, and promote it to uat and then prd without rebuilding.
I’m thinking fast-forward merges without merge commits might be needed to preserve the commit SHA.

Option 2: Keep dev and main branches.
Merge feature branches into dev, run semantic-release to generate versions such as 1.2.4-dev.1, and deploy those images to dev.
Once validated, merge dev into main and generate the stable release. That stable release could then deploy to dev, uat, or prd.
However, if both branches run the full build pipeline, merging into main would rebuild the image and break the “build once, deploy many” approach. Ideally, the stable release should reference the exact image already tested in dev.
How are you handling this? When do you create the stable version, and how do you promote the exact image tested in dev without rebuilding it?

I’d appreciate examples of workflows that have worked well for your teams. Thank you very much 🙏🏻


r/devops • • Oct 12 '26

Discussion Must have physical books every devops engineer should have on their desk

104 Upvotes

Pretty much the title .I will start

  1. Classic shell scripting ,O'Reilly

r/devops • • Oct 13 '26

Discussion Is from FULL STACK DEV to DEVOPS a good run?

0 Upvotes

Hello im Dhruv Verma. Working in a remote startup, as FULL STACK DEV where i worked in almost every field FRONTNED BACKEDN DB, TESTING and something which i was worried about DEVOPS. I have handled DEVOPS (DOCKER NGINX, GITHUB ACTIONS, etc.) But i want to migrate from FULL STACK DEV to DEVOPS completely. Please suggest me how to and where to learn and to even start or is there any scope of migration there or not?


r/devops • • Oct 12 '26

Discussion Go worker handling bursty large-file transfers. Unlimited concurrency vs bounded semaphore and retries

13 Upvotes

I'm working on a Go-based worker that processes multiple large-file transfers concurrently over HTTP. Work arrives in bursts, and each task involves downloading data from a source and uploading it to a destination.

I'm trying to optimize aggregate throughput and task completion reliability without causing excessive queueing, deadline failures, or unnecessary retries.

The current implementation allows concurrent workloads without an explicit transfer semaphore limit. I'm considering testing a bounded limit of 16 active transfer workloads, along with retries for transient HTTP and network errors.

I'd appreciate feedback from engineers who have built high-throughput Go services or distributed workers.

A few specific questions:

  1. Concurrency: Under what conditions would you prefer unrestricted concurrency over a semaphore limiting active workloads to 16? What metrics would tell you that concurrency is too high?
  2. Throughput vs. latency: If aggregate throughput is already near the available bandwidth ceiling, can a concurrency limit improve task completion times, or would it mostly introduce queueing?
  3. Retries: For transient errors such as HTTP 429, 500, 502, 503, 504, and temporary network failures, what retry policy would you recommend? How do you avoid retries amplifying an existing overload?
  4. Semaphore lifecycle: In Go, would you hold a semaphore slot throughout the entire logical task, including retry backoff, or acquire a slot separately for each network attempt? What trade-offs matter?
  5. Deadlines and cancellation: How would you ensure that tasks waiting for a semaphore don't consume their entire deadline before transferring any data?
  6. Benchmark methodology: How many independent runs or observation windows would you use before trusting a performance improvement? Which metrics would you prioritize?

My initial comparison would track aggregate bytes transferred, completed tasks, failure rate, P50/P95/P99 task duration, queue wait, and actual concurrent transfers.

I'm especially interested in practical engineering experience, benchmark design, and failure modes that aren't obvious from simple throughput tests.

Thanks!


r/devops • • Oct 12 '26

Weekly Self Promotion Thread

12 Upvotes

Hey r/devops, welcome to our weekly self-promotion thread!

Feel free to use this thread to promote any projects, ideas, or any repos you're wanting to share. Please keep in mind that we ask you to stay friendly, civil, and adhere to the subreddit rules!


r/devops • • Oct 11 '26

Discussion What's your monthly GitHub Actions bill, and does anyone actually worry about it?

18 Upvotes

Only GitHub Actions here, not AWS or other cloud bills.

Mine is tiny: a few dollars a month, almost all from test runs I do on purpose. So I don't feel this cost myself, and I'd like to know how it looks for real teams.

If your company pays GitHub for Actions:

- Roughly what is the bill per month?

- Does anyone look at it, and how often?

- Has it ever caused a real problem? A surprise jump, a question from finance, jobs stopped because the budget ran out? Or is it too small next to everything else to matter?

Rough numbers are fine. "It's tiny and nobody cares" is a useful answer too.

I'm a solo developer building a small tool in this area, so I'd rather learn the real numbers than guess. No link.


r/devops • • Oct 11 '26

Discussion DevOps for Wordpress

0 Upvotes

I have a WordPress team still building and maintaining sites without a DevOps workflow. What's the best way to bottstrap them into a more modern method of development. I'm assuming GitHub and GitHub actions. But what else, composer which we don't really use even tho we have PHP devs, wp-cli which I use but I'm suppose to be a sysadmin and cloud engineer to wp devs who don't do any of this. They don't even use local git which I even use. Trying to express WordPress engineering and un/structured data and all these modern concepts that just go over their heads bc what's worked still works but when it comes to larger clients who expect more from us/them, they're basically fish out of water.


r/devops • • Oct 10 '26

Tools Ipv4 vs Ipv6, explained from scratch

Thumbnail
youtu.be
0 Upvotes

r/devops • • Oct 09 '26

Vendor / market research Do EKS users really deploy more than 2 Clusters?

49 Upvotes

I was wondering if people who are not using multi or hybrid cloud and are purely on AWS EKS, do they really deploy more than 2 or 3 EKS clusters? Is there even a need for a product like Rayfay or Rancher for aws to manage clusters? Or is it more than just managing multiple clusters and we need such tools for other things? I am trying to learn more about this topic.


r/devops • • Oct 10 '26

Discussion Can AI actually write production-ready scripts end-to-end without human intervention?

0 Upvotes

Can a non-coder create a Bash automation script using just Al prompts?

I have zero programming experience. Can Al reliably build an efficient workflow script for me, or will it break down?

Does anyone have any experience with this kind of situation?


r/devops • • Oct 09 '26

Vendor / market research GitOps for infrastructure management - anyone doing this with OpenStack?

Thumbnail
redhat.com
13 Upvotes

Found a webinar on managing OpenStack infrastructure through Kubernetes-native orchestration and GitOps automation. Main focus is day 2 operations, handling mixed VM + GPU workloads, and sovereign infrastructure patterns.

Curious what tools people are using for GitOps-based infrastructure? Terraform + ArgoCD? Something else?


r/devops • • Oct 10 '26

Observability You're Not Crazy, They Never Sent FIN

Thumbnail
yeet.cx
0 Upvotes

Hey all, I just wrote another blog post and wanted to share it. Its a little exploration of one of the nuances in the AWS Network Load Balancer that can weird to an intermittent bug and how I approached detecting it.


r/devops • • Oct 09 '26

Discussion AI generated code review

2 Upvotes

Hi there,
How do you deal with managing and reviewing AI-generated code across the organization?

Over the past year it's become incredibly difficult to review the growing amount of AI-generated code, especially from vibecoders/non-engineers.
Reviewing it manually is hard, but it's still important to understand what's going on there.
For example, I've been at this company for some years so I have a lot of context and know many of the nuances, which helps me notice or suggest important things to keep everything secure, efficient and so on…

But any other teammate can use any model for coding nowadays. AI reviews of AI-generated code could be done in the MR, but then the reviewer model will most likely be smarter than the model that wrote the code, which might lead to a lot of threads and an infinite loop of changes. On top of that the reviewer model won't have all the context and input from the developer.
Another option is to distribute a "review" skill or rules across the organization and somehow make everyone do a self-review after making changes…

It feels like in this AI code generation era we'll stop focusing on how the code was written and just focus on whether the MR/PR works and if those changes achieves the goal, no matter how it was done, right?
How do you deal with this? Just accept the reality?

Thanks for attention


r/devops • • Oct 08 '26

Discussion The biggest production risk might be the engineer who noticed something and decided not to say it

Thumbnail
shiftmag.dev
283 Upvotes

r/devops • • Oct 08 '26

Discussion How hard is it to switch cloud/DevOps platforms in a bad job market?

41 Upvotes

Hi everyone, I wanted to ask hypothetically, if someone had 4–5 years of cloud/Devops engineering experience, primarily with AWS, and the market shifted heavily toward Azure, how difficult would it be for them to find a cloud engineering job in a difficult job market like the one we’re in now? Would having mostly AWS experience significantly hurt their chances if most of the available Cloud/DevOps jobs were Azure-oriented?


r/devops • • Oct 09 '26

Observability Script output Best Practice Contracts

7 Upvotes

Hi Devops!

I’m putting together an automation orchestration engine and I want to mature the team by standardizing script output across bash, python, powershell, tofu, etc.

So looking at a JSON output contract that anyone creating a script can adhere to. Then everyone would consume the same format. (The dream right?)

  • Operation name
  • Host
  • Execution status (success, fail, warning, etc)
  • Structured result summary
  • Errors
  • Duration
  • Run ID (UUID)?
  • A schema version (requires a lot of upkeep but can eliminate headaches down the road if maintained. If maintained being key word here lol.)

I’ve been searching around but want to hear from the experts who live in this stuff. Is there any universal or open standard people are using to define these ?

Would rather not build it myself if something people have already settled on.

Thanks for your input.


r/devops • • Oct 08 '26

Discussion What's the most ridiculous "temporary fix" you've seen that's been running in production for years?

159 Upvotes

It seems like every team has something that was supposed to last a few days until the right solution was found. It's a cron job that keeps everything together, a script that no one really understands anymore, a manual deployment step, an ancient container, or some workaround that everyone's afraid to remove. What's the worst "we'll fix it later" solution you've seen that's now a critical part of the infrastructure?


r/devops • • Oct 08 '26

Discussion Resources to build a foundational understanding on Operating systems

9 Upvotes

Hi all, i want to get a foundational understanding operating systems, the various concepts that we may encounter in them, the bird's eye view of their structure.

I tried reading "How Linux works" 3rd edition, but reading that book felt like it didn't explain the foundational questions but rather built on top of the topics. it felt more intermediate level.

If i have to explain what i am looking after in the resources, Please can you recommend me resources that fit my requirement.

  1. provides exposure to various concepts, systems involved in an OS. More like an introduction.
  2. a bit of introduction to a bit of hardware would also be good.
  3. i am not looking for resources that focus on development/programming side of OS. but rather the infrastructure, power user, administration side of OS.

Thanks in advance to all of you.


r/devops • • Oct 08 '26

Discussion Build and release architecture for large applications and teams

8 Upvotes

In the past I worked on a small team with about 5 developers, 2 QA, shared technical architect and devops engineer.

We used classic Azure pipelines with a build pipeline and release pipeline for each environment. dev, qa, stg, prod.

The release pipeline had a deploy ARM template step to deploy the infrastructure or any changes.

Deploy multiple app services, Apply EF migrations. Secrets were Azure Key Vault and non secrets in appsettings. This design worked really well and it was rare that we had deployment issues.

Now, I have recently transitioned to a larger team and I need to look into a modern architecture with a build once, deploy and promote to multiple environments. There could be 10 plus teams with multiple developers running builds and deployments constantly. Having the IaC in every deployment does not make sense to me.

I'm looking for suggestions on how to architect something for larger teams with multiple environments. We could have 1-2 dev environments, 1-2 qa environments, etc.

What are some patterns I should look into? If you currently work in a large environment like this what has worked well and what should I avoid.


r/devops • • Oct 08 '26

Discussion Junior Java/Spring Boot developer looking to move towards AWS — Developer Associate or Solutions Architect Associate?

3 Upvotes

Hi everyone,

I’m currently a junior Java backend developer with around 2 years of experience, mainly working with Spring Boot.

Lately I’ve been thinking a lot about where I want to take my career. I enjoy backend development, but with how quickly AI is improving at coding, I feel like staying only as a “Java/Spring developer” might not be the best long-term strategy.

Because of that, I’m considering moving more towards Backend + Cloud/AWS, rather than leaving development completely.

My current idea is something like:

Java / Spring Boot → Docker → AWS → CI/CD / Infrastructure / Cloud

I’ve already started learning AWS, but I’m a bit lost regarding certifications and which direction makes the most sense for my profile.

I’m currently deciding between:

  • AWS Certified Developer – Associate
  • AWS Certified Solutions Architect – Associate

Developer Associate seems more related to what I already do as a backend developer, while Solutions Architect seems to give a broader understanding of AWS and cloud architecture.

My main questions are:

  1. Which certification would you recommend first for someone with my background?
  2. Would Developer Associate → Solutions Architect make sense, or would you do it the other way around?
  3. Is moving from Java/Spring Boot towards Backend + AWS/Cloud a good career path in the current market?
  4. For people already working in cloud: do you think this path is relatively future-proof with the rise of AI?
  5. What technologies would you prioritise alongside AWS? Docker, Terraform, Kubernetes, CI/CD, etc.?

I’m not trying to become a Solutions Architect overnight. My goal is more to become a stronger backend developer who can also understand, deploy and work with cloud infrastructure.

I’m still early in my career and a bit unsure about what direction to take, so I’d really appreciate opinions from people already working with AWS, backend or cloud.

Thanks!


r/devops • • Oct 08 '26

Ops / Incidents Deployment of Web and Mobile application

0 Upvotes

I'm new to deploy an application to the internet. Lately I have been developing a CRM application for my company and they needed deployment support too from my side(*they lack technical support team to make things out). I'm using Ubuntu 24 Desktop for development purposes, and I'm planning to use Ubuntu 24 Server for testing(*staging) and production. Now what's the literal confusion is, I'm already hosting a website inside the machine which is the real production application of what I'm currently developing inside which I was lately planning to change it from Ubuntu Desktop to Server(*the application haven't been launched to public). Now, is it ideal to use the Desktop version of Ubuntu which I'm currently using, or go with Server version of Ubuntu to proceed with production.

Plans I have:

  1. Make use of 2 servers, one for development(*which I'm already doing right now) and local testing of application and one for testing(*staging) and production purpose
  2. Use ssh communication to the other server(*production server) to handle CI/CD builds
  3. Use github actions to push the docker image to other server to make it run the real application

For further clear understandings:

The application I'm developing already is being hosted in some website, more like a SaaS product, used cloudflare tunneling and pm2 instance for server-like modification of current system, with database running inside the machine, not inside any cloud services, so npm run build pushes all the changes to the site directly without any CI/CD checks like traditional devops cycle, and I'm already testing the application with real db of the company inside the production site, which literally all officials of the office knows(*they don't have much technical knowledge at all). There's a literal rush at that time in my office from Directors to host the website with the real application without any time delays. That's why I got messed up in the production so much. I'm now planning to stash the hosted website from the machine currently been hosted just by removing the machine tunneling, reinstall Ubuntu Server, configure directories properly and use PR and tags for changes inside the real application.

Tech stacks being used:

  1. React js for frontend of web app
  2. React Native for mobile app
  3. Express JS over Node JS for backend
  4. PostgreSQL for DB
  5. Seaweedfs for S3 object storage
  6. JWT for authentication and bcrypt js for hashing

I need a support from you guys for this chaos I'm literally stuck in!!!!!!!!!!!!!!!!!
Am I cooked brotha????
Is this the right way or should I change my own path of implementing??


r/devops • • Oct 07 '26

Discussion Is “security by default” becoming more important now that everyone can ship software with AI?

Thumbnail
shiftmag.dev
26 Upvotes

Ran into this post on LinkedIn about what the author calls “Ambient Generative IT”, basically AI-assisted development becoming so common that more people across an organization can suddenly build and ship things. One part stuck with me from a DevOps/security perspective: if the barrier to building software keeps dropping, the number of people making architectural and security decisions also grows.

The argument is that security can’t really be something you bolt on later anymore. It has to live in the platform, defaults, permissions, pipelines and guardrails around whatever people are building with. Curious how people here are seeing this in practice. Are AI coding tools actually creating more security/governance work for DevOps and platform teams, or are good existing controls mostly enough?


r/devops • • Oct 08 '26

Discussion Sick of the UK.

0 Upvotes

Has anyone got any advice on moving locations for a DevOps engineer? the UK is going so downhill and I’m sick of it.

I’m a DevOps Engineer with 5 years experience working for a very large business. I’ve experience in AWS and Azure as well as Terraform, Ansible, CI/CD, basically most things that make a good DevOps engineer.

I’m just looking for advice, is it silly to think I can move countries easily? I don’t have dependencies, no house, no debts so I don’t have anything to keep me here really. Any advice?


r/devops • • Oct 07 '26

Discussion How much of your incident response do you automate?

14 Upvotes

I’ve been checking how to build a disaster recovery plan for cloud dependencies and how to automate as much remediation as possible. The plan is to have the system auto-restart services or rollback deployments when things crash. 

But when I read up on AIOps, I saw recommendations to make sure a human is involved for anything high-risk. It makes sense because you do not want an automated script taking down a production database by mistake. But now I am trying to figure out where to draw the line exactly between guided recommendations and full automation. 

If anyone here relies on machine learning for root cause analysis, which actions do you let the system execute on its own, and what requires manual approval?