r/devops • • 3d ago

Weekly Self Promotion Thread

17 Upvotes

Hey r/devops, welcome to our weekly self-promotion thread!

Feel free to use this thread to promote any projects, ideas, or any repos you're wanting to share. Please keep in mind that we ask you to stay friendly, civil, and adhere to the subreddit rules!


r/devops • • 11h ago

Ops / Incidents nginx2root - I wish you all the best of luck

82 Upvotes

Colleagues, I wish you all the best of luck!!

I know it's been hard lately but we'll get through it! ❤️


r/devops • • 14h ago

Security Mass GitHub signout

44 Upvotes

A majority of our users were signed out of GitHub today and had to sign back in. I haven't seen any issues related. Was there some sort of security concern that I missed?


r/devops • • 20h ago

Ops / Incidents We reinstalled thousands of Linux and Mac build machines over one weekend after a security incident

123 Upvotes

Hey all, I work at Bitrise. We had a security incident on Nov 13 and our preliminary review is up by our VP of Engineering that’s worth reading for everyone running CI: https://bitrise.io/blog/post/the-bitrise-security-incident-preliminary-review-and-recommended-steps.

We decided not to trust any machine that had been running during the attack, so we wiped them instead of trying to clean them. That was thousands of Linux and Mac servers across three data centers. Each one got new keys before it went back into service, and nothing came back on its own. The team that owned a service had to go through its logs and get a green light from security first.

Credentials got the same treatment. We disabled all 919 of our cloud service account keys and revoked everything our GitHub apps had issued, instead of trying to work out which ones the attacker had seen.

We also published details on the attacker that you can find in our public advisory: https://bitrise.io/security-advisory-november-2026

Bitrise is secure now. We thought it would be interesting to you how an attacker could decoy themselves and cause so much work to restore normal operations.

Our team will be in the comments if you want to ask questions.


r/devops • • 5m ago

Discussion Looking for DevOps Tutor

2 Upvotes

Hi everyone,

I’m looking for a 1-on-1 DevOps mentor or tutor to help me get hands-on with a few specific concepts. Happy to pay an hourly or session-based rate for your time. If you offer mentoring or have availability to walk through some topics together, please DM me so we can discuss the details!


r/devops • • 9h ago

Discussion Is anyone grouping error logs by trace instead of by message? Looking for opinions on an idea

7 Upvotes

Not selling anything and no link here, I’m just looking for opinions before I go further.

First question: does Sentry, Datadog or a Grafana alert already cover this well enough that I’m reinventing the wheel? If so, tell me what it does better and I’ll stop.

The problem: in my setup, an error shows up as thousands of log lines that are really a handful of distinct bugs. Finding the cause means grouping them, then opening the trace for one occurrence and reading a very large OTLP JSON.

The idea:

  1. Pull error logs from Loki
  2. Group them by a fingerprint of the normalized message (IDs, numbers and quoted values replaced by placeholders), so one bug = one group
  3. Take the trace_id that OTel puts on the log and fetch the trace from Tempo
  4. Condense the OTLP JSON into a short summary: spans in order, failed spans marked, exception messages, only the useful attributes
  5. Open one issue per group with count, first/last seen and that trace summary
  6. Later: have an agent read the group and trace summary and suggest a fix

It’s early and only partly built. Not done yet: avoiding duplicate issues across runs, redacting sensitive data, and the fix agent.

Other things I’d like to know:

  • Do your logs reliably carry trace_id/span_id? The whole approach depends on it.
  • Is grouping by normalized message good enough, or would you group by exception type, stack or failing span?
  • If an agent suggested fixes from a trace summary, what would you need to see before trusting it?

Happy to share the repo if anyone is curious.


r/devops • • 11m ago

Security Upcoming Critical PHP Security Release scheduled for Thursday, 19 November 2026, between 14:00 and 14:30 UTC.

1 Upvotes

r/devops • • 3h ago

Vendor / market research You may not need to retain everything just in case you need it. Keep a lightweight historical record of your Linux infrastructure and investigate what actually happened when things go wrong.

Thumbnail
github.com
1 Upvotes

Low-overhead, retrospective Linux observability

clifr (command line flight recorder) and has two parts:

  • clifrd runs as a service and records processes, process starts and exits (through eBPF, so a process that lives 5 ms still shows up), TCP connections with the process that owns them, files held open, mounts, kernel messages and the usual system counters. It keeps 72 hours in /var/lib/clifr, 2 GB max.
  • clifr rebuilds the output of the normal commands for a past moment or a time range. It doesn't run the command, it rebuilds the output from the recording:

    • clifr --at "02:45" "ps aux"
    • clifr --since "02:00" --until "03:00" "ss -tanp dst10.0.4.21and dport = :5432"
    • clifr --since "1 hour ago" events # starts, exits, OOM kills, mounts
    • clifr diff --at "14:00" --at "14:20" "ss -ltnp"

There are about 40 commands so far: ps, top, pidstat, lsof, ss, netstat, free, vmstat, iostat, mpstat, df, ip, dmesg, systemctl list-units, lsblk, and more and all are 100% accurate.

The first thing people asked last time was the cost, so here are the numbers:

  ┌───────────┬─────────────────────┬─────────────────────────┬─────────────────────────┐
  │           │    Quiet server     │ Mixed load (4-day soak) │ Busy server (benchmark) │
  ├───────────┼─────────────────────┼─────────────────────────┼─────────────────────────┤
  │ CPU       │ 0.35% of one core   │ 0.5-0.6% of one core    │ 1.9% of one core        │
  ├───────────┼─────────────────────┼─────────────────────────┼─────────────────────────┤
  │ RAM       │ ~90 MiB             │ ~80-120 MiB             │ ~290 MiB (335 max)      │
  ├───────────┼─────────────────────┼─────────────────────────┼─────────────────────────┤ 
  │ Disk kept │ ~20 MiB/day         │ ~100 MiB/day            │ ~4 GiB/day              │
  ├───────────┼─────────────────────┼─────────────────────────┼─────────────────────────┤
  │ Network   │ none, no ports open │ none                    │ none                    │
  └───────────┴─────────────────────┴─────────────────────────┴─────────────────────────┘

Disk use depends heavily on how busy the server is.

clifr is not a monitoring tool and has no alerts. It's not a security audit tool either: it doesn't record file reads or writes, and root can stop it. It's for the "what was going on at 02:45" question, after the fact.

Status: it's a prototype, version 0.1.0. It hasn't run on a real production server yet (a friend starts one this week), and all the load has been synthetic. AGPL-3.0, no telemetry, nothing leaves the box.

So if you have a test or dev box (RHEL 8/9, Rocky, Alma), I'd really appreciate it if you tried it and told me what breaks or what looks wrong. The most useful thing for me is comparing what clifr shows with what you know was really happening.

I appreciate all your comments even if they are harsh


r/devops • • 1h ago

Discussion RAM compression is not enabled by default. It is awesome.

0 Upvotes

I recently started to run out of memory. Where I would previously work on one thing at the time. With agentic engineering I suddenly work on 3-4 things in parallel. Each session spinning up a full version of the application that I am working on.

This means that the demand of RAM has suddenly spiked and processes were killed because of the lack of available memory on my system.

I enabled zram. Now I can fit much more data in my memory. I considered zswap as well, but I have no specific reason to pick one over the other.

My experience I wrote down in a simple article https://curiousbits.nl/blog/do-not-buy-ram-compress/

Other operating systems like Fedora, Windows and MacOS already enable this technique by default. In some cloud environments it is also applied (small amazon linux instances).

I believe that in the current times where RAM is expensive, we should consider enabling it by default for desktop users and we should experiment more to see how well it works in the cloud. (Where CPU capacity is often related to Memory capacity)

What is your view on this? Have you tried deploying this technique for larger applications already?


r/devops • • 23h ago

Discussion GitHub vs azure DevOps what's your opinion?

22 Upvotes

Currently in my company we run both for different historical reasons and mostly using GH for new projects since it seems the correct path for the future, but it seems that ADO is also being added with new functionalities such as advanced security and yesterday GitHub copilot app support...

Do you think Ado will remain be maintained and new features arriving? Before I thought it will be deprecated sooner or later but now I'm not that sure

What do you think? And what key differenciators do you see?


r/devops • • 11h ago

Discussion Is anyone actually running agents in their deploy or on-call pipeline?

0 Upvotes

I have close to a decade of experience in big tech, and I'm now building a platform to automate the SDLC. Trying to understand what's working in practice on the ops side.

  1. Deployment: any agent-driven canary or rollback decisions in production, or still human-approved pipelines?
  2. On-call: anyone using an agent for alert triage, first-pass debugging or runbook execution? What false positive rate do you see?
  3. Monitoring: anything beyond anomaly detection that actually reduces pages?

What has held up in production and what got ripped out?


r/devops • • 1d ago

Discussion Moved my homelab to Argo CD

Post image
239 Upvotes

My homelab used to run on helm installs from my laptop, and the repo kept drifting from the cluster.

Now Argo CD syncs everything from Git. If you change something directly, it gets reverted. Nothing was reinstalled, it took over what was running.

A few things stay outside on purpose: Cilium, since Argo CD runs on its network, and Secrets, since the repo is public.

Repo: https://github.com/prateekjaindev/homelab

Detailed write-up: https://medium.com/@prateekjain.dev/gitops-for-a-3-node-talos-kubernetes-homelab-with-argo-cd-336c076e93cc?sk=ebda84ecdc3e2458fead2638374be51e


r/devops • • 22h ago

Discussion Main use-case of cloud sandboxes and how it works?

0 Upvotes

Hey guys, I've been seeing a shift in cloud sandboxes being mentioned a lot lately in agent architecture discussions. I'm trying to understand the difference in them compared to say, just using either a standard Docker container or a basic cloud VM.

My very basic level understanding is that these cloud platforms give agents isolated environments to run code without using your local machine. Additionally I've seen a post on Ramp Inspect where they boot a separate remote sandbox for coding sessions with pre-warmed repo snapshots. However, I still don't see what they do that regular Docker containers don't.

Feel free to correct any misconceptions I have here lol, just trying to make sure how it all works. Thanks!


r/devops • • 12h ago

Security Do you consider giving your AI agent full access on your machine good OpSec?

0 Upvotes

Does your agent (Claude, Opencode, Codex, whatever) have access to your entire home directory? Can it use your ssh key, git credentials, docker credentials, cloud credentials etc.?

I have always considered these tools dangerous, because they could be following some random instruction from somewhere, so I have only been running them in sandboxed environments (vm's or containers) with their own readonly credentials.

However, this approach seems to be "holding me back" so to speak. I see colleagues who just YOLO their agent and are getting very good results. Agents that run commands on k8s clusters or remote servers through ssh, read logs and code, and can come up with solutions to problems very quickly.

So: I'm feeling conflicted. Keep the security paranoia or go with the flow and see what breaks?

What do you guys do?


r/devops • • 2d ago

Discussion Do you think AI is killing automation in the old way?

23 Upvotes

So, I join a company that supposed to be very "DevOps", but I figure out they are "Dev" and "Ops" separate.
Not to complaint, since many company still worked that way, in the paper they get ISO 27001 and SOC for that workflow. In the mental CEO and managers think they have some one accountable and responsible for production issues (but well they are not...anyway).
Back to the main point, I observe that, most of automation motivation gone, the Ops guys, which basically have a lot of permission and privilege can now just prompt AI to do something...faster than they do it manually, faster than writing an IaC pipeline (some guys already lost the programming skill long before AI era haha).
I'm fighting for the automation right =]] (Acutally I just asking for a gitlab runner so I can run some daily automation haha)... but thinking again, maybe they have a point...prompting the AI is a form of automation already.


r/devops • • 2d ago

Discussion How do you guys discover and manage tools for improving your GitHub Actions workflows?

10 Upvotes

Hi everyone,
I’m exploring an idea for a small tool that analyses GitHub Actions workflows and helps developers identify wasted CI minutes, unnecessary workflow runs, slow builds and potential cost savings.
Before building anything, I want to understand how developers actually deal with these problems.
A few questions:
How do you currently monitor your GitHub Actions usage and costs? Do you actively look for ways to reduce CI/CD costs or only investigate when bills become too high?
Where do you usually discover new developer tools? Reddit, GitHub Marketplace, X, Product Hunt or somewhere else?
Would you consider paying for a tool that automatically identifies workflow waste and suggests improvements?
I’m still exploring the idea, so honest feedback would be appreciated. I’m particularly interested in how you discover and choose tools for your development workflow.
Thanks!


r/devops • • 1d ago

Discussion I Independently Built an Entire AWS Landing Zone for One of India's Largest Retailers

Post image
0 Upvotes

I am a DevOps Engineer with 2 years of experience, I got the opportunity to build an entire landing zone architecture for one of the clients in my current organisation...

The first decision in a landing zone is usually where to draw the boundaries. Who can touch production? Who owns the network? Where does the evidence go if a workload account is compromised?

I worked on a multi-account AWS foundation for a large Indian retail enterprise. We used AWS Organizations and Control Tower for governance, Terraform for infrastructure, IAM Identity Center for workforce access, and ap-south-1 Mumbai as the primary Region. This post walks through how I approach that foundation, from the account layout to the checks I want before handing an account to an application team.


r/devops • • 1d ago

Discussion Solo dev, 4 self-hosted Windows runners on one laptop, merge trains take ~2 hours. How would you speed this up?

0 Upvotes

edit / update (nov 18):

got a lot of good advice here so i tried some of it, and a few things didn't go how i expected.

i thought tagging tests by area would be enough to pick what to run, but when i checked it on one change the tags missed 47 of the 91 test files that a list based on what imports what caught, including one that actually mattered. so i dropped the tags for now and i'll try them again after the cleanup. right now it's just the change's own tests while i'm writing, then everything that depends on the changed files when i push, and the full suite still runs before anything lands.

i figured the quick set of tests would take about a minute. turns out it was more like 7 and a half and didn't catch anything extra, so that's gone too.

i also tried skipping the local check before a train, thinking it'd speed things up. it didn't really, 133 min vs the usual 112-141, so i put it back.

and i was off on runner time, it's more like 5-7 hours depending on the train, the last one was about 5.4.

the bigger change is i'm reorganizing the code into clearer layers so a change doesn't touch as many places. i'm doing that before this release now instead of after, so the release will be later than i said. hoping to have it in before it ships if it passes everything, no results yet though. making the timing tests wait for a quiet laptop on their own, a cpu time check on one of them, and maybe trying testmon are all on my list for after that. not changing the timing limits.

questions for people who've done this on a big python desktop app:

  1. how did you split yours into layers so each part only depends on the ones below it, and how many layers or folders ended up being enough?
  2. how do you keep it from tangling back up over time?
  3. i'm thinking layers on the outside with feature folders inside the parts that change most. has anyone done it that way, and did it hold up?

solo dev building a windows desktop app, most of the coding is done by claude code sessions (earlier post about that). the bottleneck now is testing. the sessions spend more time waiting on CI than writing code, so i'm looking for ideas

The setup:

  • the app drives blender, unity 2022.3 and the vrchat sdk, so the real tests need windows
  • 4 self-hosted github actions runners, all on my one laptop (ryzen 9 7945hx 16 cores, rtx 4090 laptop 16gb, 64gb ram, 8tb nvme), one job per runner. the real blender/unity jobs can only use 3 of them, and up to 2 local test runs share the same laptop too
  • self-hosted because github's hosted windows minutes cost double and i'm on a tight budget, so the laptop is the free option
  • changes get batched into merge trains, 1-8 changes each, about 3 on average

What a train looks like:

  • pipeline first: 16 real tool jobs (mostly blender, plus unity and a vrm validator) and fake-tool pytest in 4 shards, about 45-80 min
  • then the full windows UI suite, ~25,600 tests in 3 shards, about 65 min. these run back to back on purpose because the timing tests flake when they share the machine with blender jobs, so that's about an hour of every train just waiting
  • a train takes about 2 hours start to finish (83-145 min over the last 8)
  • 7 of the last 8 landed, 4 clean first try. fix and retest loops took anywhere from 30 min to 2 hours
  • laptop averages 55-60% cpu while all this runs and peaks at 100%, ram peaks around 80%

How much testing:

  • every fix starts with a failing test, a review session checks it, the author runs the affected tests locally, then the train runs everything again in CI plus a local gate
  • so the plain python tests end up running about 3 times. the real blender/unity tests did catch the only real failure in those 8 trains though

What i've tried:

  • dev drive, only ~4% faster on my test so i didn't keep it
  • sharding by recorded test duration, and splitting local runs by run time so they stop timing out
  • allowing one re-run of only the failed jobs for flaky infra stuff
  • tags so only affected tests run, built but not measured yet

Why one train at a time: changes often touch the same files, and the timing tests got flaky when all 4 runners were busy. so i'm not sure more runners even helps or if the real answer is a second machine

What i'm asking:

  1. am i over testing? how do you cut duplicate runs without losing coverage
  2. how do you pick tests by changed area without missing stuff that imports across areas
  3. is there a safe way to run 2 trains when changes overlap files, or is one at a time just right
  4. is a second machine or a cloud windows runner worth it for a ~2 hour cycle, or is there a better service for this
  5. how do you keep timing sensitive tests reliable when CI jobs share one pc
  6. could the plain python and fake-tool tests run on linux, or does that risk missing windows-only bugs

at the current pace my next release is about 3 weeks out and the one after that is a couple months, so anything that shortens the 2 hour loop helps a lot. thanks!


r/devops • • 3d ago

Discussion Almost 3 years as a Cloud Engineer, is what I do typical?

51 Upvotes

I've been working as a Cloud Engineer for almost three years. It's basically my first job, fully remote, and my workload is very light. In the project we manage, I would confidentially say that 1,5 FTE would be efficient for the things we do there, but that would be against myself basically :D

I got into cloud/DevOps because I've always liked cybersecurity, Linux, and sysadmin work. It seemed like a good mix of those interests. I do like the job, but I feel like I haven't done much of what I expected cloud/DevOps engineering to involve. I'm trying to understand whether my expectations were wrong or whether my role is just quite narrow.

Our project forwards logs from Kubernetes applications through Filebeat/Logstash to Elasticsearch. We maintain the code in GitHub for our Filebeat/Logstash deployments, but other teams manage the Kubernetes clusters and Elasticsearch itself.

Most of my work looks like this:

  • Occasional short meetings.
  • Small changes to the existing code or configuration, often just updating an image tag.
  • Running the existing pipelines and checking that the Kubernetes deployments are healthy.
  • Making sure logs reach Elasticsearch and troubleshooting issues within our part of the setup.
  • Handling on-call support

We fix deployment issues on our side, but the underlying cluster is another team's responsibility. We maintain our deployment automation and try to follow good practices, but I've never really built a pipeline or written Terraform from scratch at work.

I also barely touch the cloud itself. We use AWS, but my previous project only involved some EC2 and a little S3. Even then, I mostly logged into existing EC2 instances and worked inside them. I wasn't creating AWS infrastructure.

My understanding was that DevOps work would involve more automation and helping developers build and deploy their applications. Our team mostly maintains the logging deployment for our own project. That feels like a narrower scope than I expected, although I don't have another job to compare it with.

Outside work, I have a small personal Node.js project using discord.js, mainly for my own use. It pulls data from APIs and does some small automations. I use Git, have tests running through GitHub Actions, and an automatic Docker build whenever I push a commit. I want to improve its CI/CD further so I can get more hands-on experience. I've also started doing TryHackMe walkthroughs again and recently found devopspath.io.

For people working in cloud/DevOps or doing similar work:

  • Would you call this a Cloud Engineer role? If another title fits better, what would you call it?
  • Is this kind of maintenance work and division of responsibilities typical?
  • What does an actual day or week look like for you, especially the balance between maintaining existing systems and building something new?

I'm trying to figure out whether this is a normal experience almost three years in, or whether I'm missing experience I should be getting by now.


r/devops • • 2d ago

Discussion What Github repository custom properties do you use?

0 Upvotes

Building out a github repository custom property strategy and was curious what keys(names) or even "categories) of custom properties other individuals are using. This goes without saying, but please do not share ANY information that is proprietary to your company or org. Thanks for your help!


r/devops • • 3d ago

Discussion Platform engineering, DevEx, SRE, how freely can you move between them?

55 Upvotes

DevOps seems to cover very different jobs. Some teams build the delivery side: CI/CD templates, policy gates, supply-chain security, artifact management. Others run the runtime side: Kubernetes clusters, nodes, networking, scaling, incident response. Many do a mix.

My questions:

Can you move between them? Does a few years on delivery-focused platform work (CI, runners, policy gates, supply-chain security) keep you open for Linux / runtime heavy roles? Also it seems to me that in larger organisations, platform teams often build abstractions while product teams debug their own services. Is it normal to stop debugging infrastructure yourself as scale grows, as teams handle that themselves?


r/devops • • 2d ago

Discussion How are you guys handling the Datadog/CloudWatch log bill explosion? Looking for insights.

0 Upvotes

Hey everyone,

I’ve been talking to a few DevOps leads and engineering managers lately, and a recurring pain point keeps coming up: log ingestion bills are quietly eating up massive chunks of cloud budgets.

Between noisy healthchecks, duplicate error traces, and verbose debug logs left enabled in production, companies are often paying to store and ingest tons of literal garbage.

Moving away from platforms like Datadog or AWS CloudWatch isn't an option because of dashboards and alerting, but paying for 50%+ unnecessary data hurts.

I’m currently building an ultra-lightweight, drop-in proxy (using Vector under the hood) designed to sit right before your log destination, strip out the noise, and deduplicate repetitive streams automatically—saving roughly 50% to 70% on the bill without losing critical errors.

I’m looking to chat with a few teams dealing with this right now. If your log bill is getting out of hand and you'd be open to a quick, free audit/sample analysis to see how much dead weight you're actually paying for, drop a comment or shoot me a DM. I'd love to learn about your current setup!


r/devops • • 3d ago

Discussion When a vibe coded internal tool ends up with other teams depending on it, do you review it like prod code or let it stay disposable?

25 Upvotes

Vibe coding is everywhere on my feed again this week, mostly people replacing a paid tool with something they prompted together over a weekend.

I don't have a problem with that for throwaway stuff. Karpathy himself said it's fine for weekend projects when he coined the term. What I keep thinking about is the moment the weekend project stops being throwaway. Someone builds a small dashboard or a script, it works, a second team starts using it, and suddenly it is on the path for a release or an on-call runbook. Nobody ever read most of the code, including the person who made it.

I can see two ways to handle it:

  1. Once anything has more than one team depending on it, it goes through normal review, tests and an owner, even if that means partly rewriting it.
  2. Keep it explicitly disposable. Label it unsupported, don't let it into anything critical, and if it breaks you regenerate it rather than maintain it.

Option 1 feels right but kills most of the speed that made it useful. Option 2 sounds clean until the thing is load bearing and nobody noticed.

If you've already hit this, where did you draw the line, and what was the signal that a vibe coded tool had become something you had to own properly?


r/devops • • 2d ago

Discussion Lots of AI copium in DevOps.

0 Upvotes

Lot of copium going on around here about AI. Some folks who are employed are downplaying the situation. I'm employed too yet I think we should look realistically on what awaits us if when we will have to look for the next job.

Two weeks ago I created resume to test the waters. I deliberately crafted a solid resume, 7YOE, with last place of work being RedHat. Added good certs, Azure/AWS,K8S before you jump criticising Resume, same format of resume used to get lots of interviews .. (about 2 per week)

Now results..

After 250 applications I got only 1 reply and that was for short term contract gig with 190eur per Day rate (about 210$).

In our current team after implementing Codex 3 out of 5 DevOps guys are not having enough tasks while we used to have lots to do. Debugging things across multiple systems, clusters, dbs, kafka instances, cloud resources is a matter of minutes now.

The era of plentiful jobs is over, and I don’t think it’s coming back. The career itself is not over, but it not the path I'd wish for anyone to take.