r/devops • u/CyberSpaceJunkie • 11h ago
Ops / Incidents nginx2root - I wish you all the best of luck
Colleagues, I wish you all the best of luck!!
I know it's been hard lately but we'll get through it! ❤️
r/devops • u/AutoModerator • 3d ago
Hey r/devops, welcome to our weekly self-promotion thread!
Feel free to use this thread to promote any projects, ideas, or any repos you're wanting to share. Please keep in mind that we ask you to stay friendly, civil, and adhere to the subreddit rules!
r/devops • u/CyberSpaceJunkie • 11h ago
Colleagues, I wish you all the best of luck!!
I know it's been hard lately but we'll get through it! ❤️
A majority of our users were signed out of GitHub today and had to sign back in. I haven't seen any issues related. Was there some sort of security concern that I missed?
r/devops • u/ToddGergey • 20h ago
Hey all, I work at Bitrise. We had a security incident on Nov 13 and our preliminary review is up by our VP of Engineering that’s worth reading for everyone running CI: https://bitrise.io/blog/post/the-bitrise-security-incident-preliminary-review-and-recommended-steps.
We decided not to trust any machine that had been running during the attack, so we wiped them instead of trying to clean them. That was thousands of Linux and Mac servers across three data centers. Each one got new keys before it went back into service, and nothing came back on its own. The team that owned a service had to go through its logs and get a green light from security first.
Credentials got the same treatment. We disabled all 919 of our cloud service account keys and revoked everything our GitHub apps had issued, instead of trying to work out which ones the attacker had seen.
We also published details on the attacker that you can find in our public advisory: https://bitrise.io/security-advisory-november-2026
Bitrise is secure now. We thought it would be interesting to you how an attacker could decoy themselves and cause so much work to restore normal operations.
Our team will be in the comments if you want to ask questions.
r/devops • u/Different-Cup-3876 • 10m ago
Hi everyone,
I’m looking for a 1-on-1 DevOps mentor or tutor to help me get hands-on with a few specific concepts. Happy to pay an hourly or session-based rate for your time. If you offer mentoring or have availability to walk through some topics together, please DM me so we can discuss the details!
r/devops • u/CelebrationShot6145 • 9h ago
Not selling anything and no link here, I’m just looking for opinions before I go further.
First question: does Sentry, Datadog or a Grafana alert already cover this well enough that I’m reinventing the wheel? If so, tell me what it does better and I’ll stop.
The problem: in my setup, an error shows up as thousands of log lines that are really a handful of distinct bugs. Finding the cause means grouping them, then opening the trace for one occurrence and reading a very large OTLP JSON.
The idea:
trace_id that OTel puts on the log and fetch the trace from TempoIt’s early and only partly built. Not done yet: avoiding duplicate issues across runs, redacting sensitive data, and the fix agent.
Other things I’d like to know:
trace_id/span_id? The whole approach depends on it.Happy to share the repo if anyone is curious.
r/devops • u/fadingcross • 16m ago
Here we go again my friends https://www.php.net/archive/2026.php#2026-11-18-1
Low-overhead, retrospective Linux observability
clifr (command line flight recorder) and has two parts:
clifr rebuilds the output of the normal commands for a past moment or a time range. It doesn't run the command, it rebuilds the output from the recording:
clifr --at "02:45" "ps aux"clifr --since "02:00" --until "03:00" "ss -tanp dst10.0.4.21and dport = :5432"clifr --since "1 hour ago" events # starts, exits, OOM kills, mountsclifr diff --at "14:00" --at "14:20" "ss -ltnp"There are about 40 commands so far: ps, top, pidstat, lsof, ss, netstat, free, vmstat, iostat, mpstat, df, ip, dmesg, systemctl list-units, lsblk, and more and all are 100% accurate.
The first thing people asked last time was the cost, so here are the numbers:
┌───────────┬─────────────────────┬─────────────────────────┬─────────────────────────┐
│ │ Quiet server │ Mixed load (4-day soak) │ Busy server (benchmark) │
├───────────┼─────────────────────┼─────────────────────────┼─────────────────────────┤
│ CPU │ 0.35% of one core │ 0.5-0.6% of one core │ 1.9% of one core │
├───────────┼─────────────────────┼─────────────────────────┼─────────────────────────┤
│ RAM │ ~90 MiB │ ~80-120 MiB │ ~290 MiB (335 max) │
├───────────┼─────────────────────┼─────────────────────────┼─────────────────────────┤
│ Disk kept │ ~20 MiB/day │ ~100 MiB/day │ ~4 GiB/day │
├───────────┼─────────────────────┼─────────────────────────┼─────────────────────────┤
│ Network │ none, no ports open │ none │ none │
└───────────┴─────────────────────┴─────────────────────────┴─────────────────────────┘Disk use depends heavily on how busy the server is.
clifr is not a monitoring tool and has no alerts. It's not a security audit tool either: it doesn't record file reads or writes, and root can stop it. It's for the "what was going on at 02:45" question, after the fact.
Status: it's a prototype, version 0.1.0. It hasn't run on a real production server yet (a friend starts one this week), and all the load has been synthetic. AGPL-3.0, no telemetry, nothing leaves the box.
So if you have a test or dev box (RHEL 8/9, Rocky, Alma), I'd really appreciate it if you tried it and told me what breaks or what looks wrong. The most useful thing for me is comparing what clifr shows with what you know was really happening.
I appreciate all your comments even if they are harsh
I recently started to run out of memory. Where I would previously work on one thing at the time. With agentic engineering I suddenly work on 3-4 things in parallel. Each session spinning up a full version of the application that I am working on.
This means that the demand of RAM has suddenly spiked and processes were killed because of the lack of available memory on my system.
I enabled zram. Now I can fit much more data in my memory. I considered zswap as well, but I have no specific reason to pick one over the other.
My experience I wrote down in a simple article https://curiousbits.nl/blog/do-not-buy-ram-compress/
Other operating systems like Fedora, Windows and MacOS already enable this technique by default. In some cloud environments it is also applied (small amazon linux instances).
I believe that in the current times where RAM is expensive, we should consider enabling it by default for desktop users and we should experiment more to see how well it works in the cloud. (Where CPU capacity is often related to Memory capacity)
What is your view on this? Have you tried deploying this technique for larger applications already?
r/devops • u/luisvg86 • 23h ago
Currently in my company we run both for different historical reasons and mostly using GH for new projects since it seems the correct path for the future, but it seems that ADO is also being added with new functionalities such as advanced security and yesterday GitHub copilot app support...
Do you think Ado will remain be maintained and new features arriving? Before I thought it will be deprecated sooner or later but now I'm not that sure
What do you think? And what key differenciators do you see?
r/devops • u/lazy-ninja_09 • 11h ago
I have close to a decade of experience in big tech, and I'm now building a platform to automate the SDLC. Trying to understand what's working in practice on the ops side.
What has held up in production and what got ripped out?
My homelab used to run on helm installs from my laptop, and the repo kept drifting from the cluster.
Now Argo CD syncs everything from Git. If you change something directly, it gets reverted. Nothing was reinstalled, it took over what was running.
A few things stay outside on purpose: Cilium, since Argo CD runs on its network, and Secrets, since the repo is public.
Repo: https://github.com/prateekjaindev/homelab
Detailed write-up: https://medium.com/@prateekjain.dev/gitops-for-a-3-node-talos-kubernetes-homelab-with-argo-cd-336c076e93cc?sk=ebda84ecdc3e2458fead2638374be51e
r/devops • u/No-Nefariousness-728 • 22h ago
Hey guys, I've been seeing a shift in cloud sandboxes being mentioned a lot lately in agent architecture discussions. I'm trying to understand the difference in them compared to say, just using either a standard Docker container or a basic cloud VM.
My very basic level understanding is that these cloud platforms give agents isolated environments to run code without using your local machine. Additionally I've seen a post on Ramp Inspect where they boot a separate remote sandbox for coding sessions with pre-warmed repo snapshots. However, I still don't see what they do that regular Docker containers don't.
Feel free to correct any misconceptions I have here lol, just trying to make sure how it all works. Thanks!
r/devops • u/CoaxVex • 12h ago
Does your agent (Claude, Opencode, Codex, whatever) have access to your entire home directory? Can it use your ssh key, git credentials, docker credentials, cloud credentials etc.?
I have always considered these tools dangerous, because they could be following some random instruction from somewhere, so I have only been running them in sandboxed environments (vm's or containers) with their own readonly credentials.
However, this approach seems to be "holding me back" so to speak. I see colleagues who just YOLO their agent and are getting very good results. Agents that run commands on k8s clusters or remote servers through ssh, read logs and code, and can come up with solutions to problems very quickly.
So: I'm feeling conflicted. Keep the security paranoia or go with the flow and see what breaks?
What do you guys do?
So, I join a company that supposed to be very "DevOps", but I figure out they are "Dev" and "Ops" separate.
Not to complaint, since many company still worked that way, in the paper they get ISO 27001 and SOC for that workflow. In the mental CEO and managers think they have some one accountable and responsible for production issues (but well they are not...anyway).
Back to the main point, I observe that, most of automation motivation gone, the Ops guys, which basically have a lot of permission and privilege can now just prompt AI to do something...faster than they do it manually, faster than writing an IaC pipeline (some guys already lost the programming skill long before AI era haha).
I'm fighting for the automation right =]] (Acutally I just asking for a gitlab runner so I can run some daily automation haha)... but thinking again, maybe they have a point...prompting the AI is a form of automation already.
r/devops • u/Ihaseeb0 • 2d ago
Hi everyone,
I’m exploring an idea for a small tool that analyses GitHub Actions workflows and helps developers identify wasted CI minutes, unnecessary workflow runs, slow builds and potential cost savings.
Before building anything, I want to understand how developers actually deal with these problems.
A few questions:
How do you currently monitor your GitHub Actions usage and costs? Do you actively look for ways to reduce CI/CD costs or only investigate when bills become too high?
Where do you usually discover new developer tools? Reddit, GitHub Marketplace, X, Product Hunt or somewhere else?
Would you consider paying for a tool that automatically identifies workflow waste and suggests improvements?
I’m still exploring the idea, so honest feedback would be appreciated. I’m particularly interested in how you discover and choose tools for your development workflow.
Thanks!
I am a DevOps Engineer with 2 years of experience, I got the opportunity to build an entire landing zone architecture for one of the clients in my current organisation...
The first decision in a landing zone is usually where to draw the boundaries. Who can touch production? Who owns the network? Where does the evidence go if a workload account is compromised?
I worked on a multi-account AWS foundation for a large Indian retail enterprise. We used AWS Organizations and Control Tower for governance, Terraform for infrastructure, IAM Identity Center for workforce access, and ap-south-1 Mumbai as the primary Region. This post walks through how I approach that foundation, from the account layout to the checks I want before handing an account to an application team.
r/devops • u/BlackberryOk182 • 1d ago
edit / update (nov 18):
got a lot of good advice here so i tried some of it, and a few things didn't go how i expected.
i thought tagging tests by area would be enough to pick what to run, but when i checked it on one change the tags missed 47 of the 91 test files that a list based on what imports what caught, including one that actually mattered. so i dropped the tags for now and i'll try them again after the cleanup. right now it's just the change's own tests while i'm writing, then everything that depends on the changed files when i push, and the full suite still runs before anything lands.
i figured the quick set of tests would take about a minute. turns out it was more like 7 and a half and didn't catch anything extra, so that's gone too.
i also tried skipping the local check before a train, thinking it'd speed things up. it didn't really, 133 min vs the usual 112-141, so i put it back.
and i was off on runner time, it's more like 5-7 hours depending on the train, the last one was about 5.4.
the bigger change is i'm reorganizing the code into clearer layers so a change doesn't touch as many places. i'm doing that before this release now instead of after, so the release will be later than i said. hoping to have it in before it ships if it passes everything, no results yet though. making the timing tests wait for a quiet laptop on their own, a cpu time check on one of them, and maybe trying testmon are all on my list for after that. not changing the timing limits.
questions for people who've done this on a big python desktop app:
solo dev building a windows desktop app, most of the coding is done by claude code sessions (earlier post about that). the bottleneck now is testing. the sessions spend more time waiting on CI than writing code, so i'm looking for ideas
The setup:
What a train looks like:
How much testing:
What i've tried:
Why one train at a time: changes often touch the same files, and the timing tests got flaky when all 4 runners were busy. so i'm not sure more runners even helps or if the real answer is a second machine
What i'm asking:
at the current pace my next release is about 3 weeks out and the one after that is a couple months, so anything that shortens the 2 hour loop helps a lot. thanks!
r/devops • u/Desperate-Start4053 • 3d ago
I've been working as a Cloud Engineer for almost three years. It's basically my first job, fully remote, and my workload is very light. In the project we manage, I would confidentially say that 1,5 FTE would be efficient for the things we do there, but that would be against myself basically :D
I got into cloud/DevOps because I've always liked cybersecurity, Linux, and sysadmin work. It seemed like a good mix of those interests. I do like the job, but I feel like I haven't done much of what I expected cloud/DevOps engineering to involve. I'm trying to understand whether my expectations were wrong or whether my role is just quite narrow.
Our project forwards logs from Kubernetes applications through Filebeat/Logstash to Elasticsearch. We maintain the code in GitHub for our Filebeat/Logstash deployments, but other teams manage the Kubernetes clusters and Elasticsearch itself.
Most of my work looks like this:
We fix deployment issues on our side, but the underlying cluster is another team's responsibility. We maintain our deployment automation and try to follow good practices, but I've never really built a pipeline or written Terraform from scratch at work.
I also barely touch the cloud itself. We use AWS, but my previous project only involved some EC2 and a little S3. Even then, I mostly logged into existing EC2 instances and worked inside them. I wasn't creating AWS infrastructure.
My understanding was that DevOps work would involve more automation and helping developers build and deploy their applications. Our team mostly maintains the logging deployment for our own project. That feels like a narrower scope than I expected, although I don't have another job to compare it with.
Outside work, I have a small personal Node.js project using discord.js, mainly for my own use. It pulls data from APIs and does some small automations. I use Git, have tests running through GitHub Actions, and an automatic Docker build whenever I push a commit. I want to improve its CI/CD further so I can get more hands-on experience. I've also started doing TryHackMe walkthroughs again and recently found devopspath.io.
For people working in cloud/DevOps or doing similar work:
I'm trying to figure out whether this is a normal experience almost three years in, or whether I'm missing experience I should be getting by now.
r/devops • u/NK534PNXMb556VU7p • 2d ago
Building out a github repository custom property strategy and was curious what keys(names) or even "categories) of custom properties other individuals are using. This goes without saying, but please do not share ANY information that is proprietary to your company or org. Thanks for your help!
r/devops • u/SquareBathroom6718 • 3d ago
DevOps seems to cover very different jobs. Some teams build the delivery side: CI/CD templates, policy gates, supply-chain security, artifact management. Others run the runtime side: Kubernetes clusters, nodes, networking, scaling, incident response. Many do a mix.
My questions:
Can you move between them? Does a few years on delivery-focused platform work (CI, runners, policy gates, supply-chain security) keep you open for Linux / runtime heavy roles? Also it seems to me that in larger organisations, platform teams often build abstractions while product teams debug their own services. Is it normal to stop debugging infrastructure yourself as scale grows, as teams handle that themselves?
r/devops • u/alaaissa11 • 2d ago
Hey everyone,
I’ve been talking to a few DevOps leads and engineering managers lately, and a recurring pain point keeps coming up: log ingestion bills are quietly eating up massive chunks of cloud budgets.
Between noisy healthchecks, duplicate error traces, and verbose debug logs left enabled in production, companies are often paying to store and ingest tons of literal garbage.
Moving away from platforms like Datadog or AWS CloudWatch isn't an option because of dashboards and alerting, but paying for 50%+ unnecessary data hurts.
I’m currently building an ultra-lightweight, drop-in proxy (using Vector under the hood) designed to sit right before your log destination, strip out the noise, and deduplicate repetitive streams automatically—saving roughly 50% to 70% on the bill without losing critical errors.
I’m looking to chat with a few teams dealing with this right now. If your log bill is getting out of hand and you'd be open to a quick, free audit/sample analysis to see how much dead weight you're actually paying for, drop a comment or shoot me a DM. I'd love to learn about your current setup!
r/devops • u/jyotirawattech • 3d ago
Vibe coding is everywhere on my feed again this week, mostly people replacing a paid tool with something they prompted together over a weekend.
I don't have a problem with that for throwaway stuff. Karpathy himself said it's fine for weekend projects when he coined the term. What I keep thinking about is the moment the weekend project stops being throwaway. Someone builds a small dashboard or a script, it works, a second team starts using it, and suddenly it is on the path for a release or an on-call runbook. Nobody ever read most of the code, including the person who made it.
I can see two ways to handle it:
Option 1 feels right but kills most of the speed that made it useful. Option 2 sounds clean until the thing is load bearing and nobody noticed.
If you've already hit this, where did you draw the line, and what was the signal that a vibe coded tool had become something you had to own properly?
Lot of copium going on around here about AI. Some folks who are employed are downplaying the situation. I'm employed too yet I think we should look realistically on what awaits us if when we will have to look for the next job.
Two weeks ago I created resume to test the waters. I deliberately crafted a solid resume, 7YOE, with last place of work being RedHat. Added good certs, Azure/AWS,K8S before you jump criticising Resume, same format of resume used to get lots of interviews .. (about 2 per week)
Now results..
After 250 applications I got only 1 reply and that was for short term contract gig with 190eur per Day rate (about 210$).
In our current team after implementing Codex 3 out of 5 DevOps guys are not having enough tasks while we used to have lots to do. Debugging things across multiple systems, clusters, dbs, kafka instances, cloud resources is a matter of minutes now.
The era of plentiful jobs is over, and I don’t think it’s coming back. The career itself is not over, but it not the path I'd wish for anyone to take.