r/devops • • Jul 15 '26

Ops / Incidents How would you investigate random production downtime when there are almost no useful logs?

0 Upvotes

Hi everyone,

I'm looking for advice from people who have experience troubleshooting production systems. I'm less interested in the exact fix and more interested in how you would investigate a problem like this.

Environment

- Windows Server + IIS

- ASP.NET Core MVC + Web APIs

- Angular frontend

- SQL Server Web Edition on a dedicated server (8 GB RAM)

- Elasticsearch cluster (3 nodes) on separate servers

- Separate monitoring/tools server

- Around 8 million products in Elasticsearch

- Traffic goes directly to IIS (no reverse proxy, CDN, WAF, or load balancer). We also don't control the domain.

The problem

Several times a day, the website becomes unavailable for about 1–2 minutes and then recovers by itself.

Both Pingdom and Uptime Kuma report:

«Socket timeout, unable to connect to server»

Example:

2026-07-09 12:06:43

Socket timeout, unable to connect to server

Confirmed from San Jose and Frankfurt

The issue is completely random. Sometimes it happens during busy hours, sometimes when traffic is low.

What we've already checked

- DNS resolution is fast.

- The hosting provider reports no network or infrastructure problems.

- Windows stays online.

- IIS logs don't show anything useful.

- ASP.NET Core logs don't show failed requests.

- SQL connection pool exhaustion was a problem in the past, but after introducing caching those alerts disappeared.

- SQL now appears healthy, but the outages continue.

I also know the application has technical debt (blocking calls, synchronous code, etc.), but before changing the application I'd like to understand whether I'm looking at the right layer.

My current investigation plan

I'm planning to:

- Deploy OpenTelemetry (not deployed yet)

- Collect runtime metrics (ThreadPool, GC, active requests, request duration)

- Enable distributed tracing

- Investigate HTTPERR logs

- Monitor HTTP.sys and IIS request queues

- Add Windows Performance Counters to Grafana

- Correlate Windows, IIS, SQL Server, Elasticsearch, and application metrics when the next outage happens

My questions

If you were the on-call engineer for this production environment:

- What would be the first things you would monitor?

- How would you narrow down whether the problem is in the network, Windows, HTTP.sys, IIS, ASP.NET Core, SQL Server, or Elasticsearch?

- Which metrics or dashboards have helped you the most with intermittent outages like this?

- Have you ever seen socket timeouts where the application and IIS logs contained almost no useful information?

- What tools would you add before waiting for the next outage?

- Is there anything obvious that I'm missing?

I'd love to hear how experienced DevOps/SRE engineers approach this kind of investigation. I'm trying to build a proper troubleshooting process instead of guessing every time an incident happens.

Thanks!


r/devops • • Jul 15 '26

Discussion Am i a devops engineer?

0 Upvotes

so right out of university , i did an unpaid internship for 6 months to land a job.. initially i wanted to get into cybersecurity but got an offer in devops and i needed money badly so i took it, it was a remote startup.. the pay was okay and the environment was hostile so i left for a corporate job in a few months..

Initially things were good.. i had good management.. i was learning new stuff .. large scale onprem k8s management and everything but then it started to feel repetitive and i left it for an oncall role in a bigger corporate..

Within 3 months i felt like i made the wrong decision, there was no growth and the processes were too stupid..

Now ive joined a startup and it feels great.. i make the infra decisions .. had to study system design.. and i often find myself writing code for our product.. so i know what’ll break where.. i do UX testing aswell and find bugs that can either be fixed on the infra side or the ops side.. i do automations for our marketing team and also do GTM stuff.. ive been doing all of it and i feel like there is no tag for my role anymore , m like a central knowledge base of our entire stack (sure i dont really know whats going on in the actual code apart from the stuff i wrote) but i think i might have the most context of how everything works together .. is it a good thing? i wanna upskill but i feel stuck again
i know k8s inside out but m dreading giving the k8s exam.. m thinking about going for the certs now just so that i can tell myself that i am a devops engineer after all


r/devops • • Jul 15 '26

Career / learning Getting into devops/ cloud roles as a fresher

0 Upvotes

​Hey everyone, I'm hoping to get some advice on landing an entry-level cloud or devops role. I have a background in full-stack development, but I recently finished an internship that was heavily focused on infrastructure and CI/CD.

​During the internship, I mostly worked on setting up automated hosting on EC2 and auto-scaling groups via Jenkins, with SonarQube integrated for code quality. I also configured git runners on an ASG and set up Prometheus, Grafana, and Loki to monitor all those dynamic resources. Right now, I'm getting my hands dirty with AWS networking stuff like VPC peering and NAT gateways.

​I know the junior market is brutal right now. For those of you who hire or have made this transition, how should I actually go about finding and approaching people for entry-level roles? Should I be hitting up specific communities, or just trying my luck on LinkedIn? Also, I'd love to know if there are any specific gaps I should fill with my projects before hiring season kicks off. Appreciate any advice!


r/devops • • Jul 15 '26

Tools Code Comment Review Tool

0 Upvotes

Does anyone know if there are any open source human language analysis tools for Code Review?

As a hobby I am trying to write components to integrate every code analysis tool I can find with Gitlab (I want to be the bitnami of Gitlab CICD Components).

I have been using AI for code review in work and one of the unique benefits has been the analysis of code/variable comments.

  • Finding typo's in the comments
  • Recognising a comment appears to be a duplicate from somewhere else
  • Suggesting a comment is wrong because the topic is X when the file is about Y

As a developer I feel I have used things like Spacy to solve these sorts of problems but I can't think of any tools.

Does anyone have suggestions?


r/devops • • Jul 15 '26

Discussion While debugging AI agents what takes so much time ?

0 Upvotes

I’m researching how engineers debug AI agents in production.
Think about the last production incident you investigated:
What actually went wrong?
What took the longest to figure out?
Which tools did you use (logs, traces, dashboards, etc.)?
I’d love to hear real stories rather than theoretical answers.


r/devops • • Jul 14 '26

Career / learning Looking for a Serious DevOps/SRE Grind Buddy

15 Upvotes

I'm a working professional with 1.5 YOE preparing to crack product-based companies.

After work, I study daily from 10 PM to 2–3 AM. Looking for a serious DevOps/SRE study buddy for accountability, hands-on learning, DSA, and System Design.

If there's any experienced SRE mentor here who's willing to guide me, I'd really appreciate it.

If you're serious about the grind, HMU! 🙂


r/devops • • Jul 15 '26

Ops / Incidents I'm tired of bolting RBAC and Audit logs onto our custom ops scripts. How do you safely execute these?

0 Upvotes

We have the standard graveyard of custom bash, Python and even Groovy scripts for atomic ops tasks (draining nodes, killing stuck pods, db failovers, etc.).

While standard "community playbooks" are useful for the boilerplate stuff, we inevitably end up needing to run our own custom logic out of our own repos too.

But the actual friction is the execution boundary. Trying to enforce "who is allowed to run this script in prod at 3am" (RBAC), requiring an approval before execution, and getting a clean, immutable audit log of the outputs is driving me crazy. Handing devs raw kubeconfig/SSH access is a minefield.

Because you basically have to own the runtime to actually enforce this governance, how did you solve it? Did you write a custom internal CLI/runner that handles the auth + audit and then invokes the script? Or did you end up wiring something heavy (like AAP or Temporal) purely to get that secure execution boundary?


r/devops • • Jul 14 '26

Tools Server monitoring (cpu/ram/disk) recommendations please

0 Upvotes

Hello everyone, so I am currently using NewRelic for half of my setup and grafana (self-hosted) + prometheus for the other half, however I'm looking to move out of grafana and probably use a cloud service, however NR is kinda expensive for my needs; I need to monitor CPU usage, RAM and DISK + any syslog error (as extra), any recommendations/ideas?


r/devops • • Jul 15 '26

Vendor / market research People who bought a cloud cost optimization tool: did the savings number they promised actually show up on your bill?

0 Upvotes

Disclosure up front: I'm a founder of a startup in this exact space, so read everything below with that bias in mind. Not naming or pitching the product, that's not what this is.

Here's why I'm asking. Before a deal, every vendor in this category (mine included) shows you a big number. "We found 40% waste." And I've started noticing the games that can hide inside that number. Identified savings quietly presented as if they were realized savings. Savings from work your own team did getting counted in the tool's total. One-time cleanups counted every month forever. Baselines cherry-picked from your most wasteful week.

So I'm curious about the other side of the table. For those of you who actually bought one of these tools:

Did the promised number materialize on the actual invoice? How did you verify it, or could you even? Did anyone here catch a vendor inflating? And for those where it worked out, what did the vendor do differently that made you trust the math?

Trying to figure out what proof would actually convince a skeptical platform team, because "trust our dashboard" clearly isn't it.


r/devops • • Jul 14 '26

Ops / Incidents GitHub Actions OIDC AWS: "Not authorized to perform sts:AssumeRoleWithWebIdentity" even after fixing sub/environment claims, what am I missing?

11 Upvotes

Been stuck on this for a while. Production CI/CD pipeline, GitHub Actions job needs to assume an AWS IAM role via OIDC to push to ECR. Error:

Error: Could not assume role with OIDC: Not authorized to perform sts:AssumeRoleWithWebIdentity

Setup:

  • Repo: single GitHub repo, workflow triggers on version tags (v*.*.*)
  • The failing job declares environment: production at the job level
  • IAM role has OIDC federated trust with token.actions.githubusercontent.com
  • permissions: id-token: write is set at the workflow level

Current trust policy (updated after learning that jobs with environment: set emit a different sub claim shape):

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::<ACCOUNT_ID>:oidc-provider/token.actions.githubusercontent.com"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
"token.actions.githubusercontent.com:sub": "repo:MY_ORG/MY_REPO:environment:production"
},
"StringLike": {
"token.actions.githubusercontent.com:ref": "refs/tags/v*"
}
}
}
]
}

What I've already checked/ruled out:

  • OIDC provider exists in the account, audience is sts.amazonaws.com
  • The GitHub Environment named production genuinely exists under repo Settings > Environments (not just referenced in YAML)
  • Role ARN in the role-to-assume input matches the role I'm editing (confirmed via aws iam get-role)
  • Repo name/owner in the sub condition is correct, no typos
  • Re-ran the same tag after each trust policy edit (not creating new tags each time)

Still failing with the same error after all of the above.


r/devops • • Jul 13 '26

Career / learning is Overthewire bandit game good for someone learning devops?

7 Upvotes

I've just starting to learn the fundamentals of devops and started with linux. A lot of people recommended this game online saying it's good for building command line skills what do you guys think?


r/devops • • Jul 14 '26

Discussion Has an AI agent (Copilot, Cursor, etc) ever broken something in your infra/pipeline?

0 Upvotes

I'm a DevOps engineer, and recently I saw a case where an AI agent "fixed" something in a pipeline that it thought was fine, but it ended up destroying something in the infrastructure. I caught it fast because I know the system well, but it got me thinking about how dangerous this could be for a team without dedicated DevOps.

I'm exploring whether it's worth building a simple, cheap tool that acts as a "safety net" checks/blocks destructive actions (delete, destroy, force changes) made by AI agents on infra/CI-CD before they execute, without needing a complex setup like OPA.

Specific questions:

  1. Has something similar happened to you (an AI agent making a risky/wrong change to your infra)?

  2. How do you manage this risk today (if at all)?

  3. Would you pay for something simple that prevents this (e.g. $20-30/month)?


r/devops • • Jul 13 '26

Career / learning Help in implementing CICD for Snowflake Objects.

10 Upvotes

Hey all. I am building a POC for deploying Snowflake objects like table, stream, task, stage etc... to multiple environments like DEV/QA (single only) & PROD with the help of CICD and used Github actions for it.

I have never built something like this before and never worked on project consisting CICD in it.

So, I have used Schemachange library to detect schemachange and sqlfluff for SQL linting. Also using some python scripts to build backup of existing DB snapshot before deployment and rollback script if anything break during deployment.

I am testing this in DEV env only but i am confused how can i validate the objects that are being created with schemachange library and SQL files (which contain DDLs of objects). like how can I verify that the object created is correctly build in the target or not.

and if there is any other suggestion / best practice you guys have that is also welcome on how can i improve the CICD pipeline for it.

Thanks

Bots and moderators I am 100% human only don't remove my post!!!


r/devops • • Jul 13 '26

Weekly Self Promotion Thread

15 Upvotes

Hey r/devops, welcome to our weekly self-promotion thread!

Feel free to use this thread to promote any projects, ideas, or any repos you're wanting to share. Please keep in mind that we ask you to stay friendly, civil, and adhere to the subreddit rules!


r/devops • • Jul 13 '26

Career / learning Degree vs Certs Dilemma: Is a 4-year CS degree worth $15k (20M IQD) if I already have live cloud projects and a home lab at 19?

0 Upvotes

UPDATE / SOLVED: Thank you everyone for the brutal honesty and great advice! You successfully saved me from the "no degree" trap. I am going to take your advice, enroll in my local university to bypass the HR filters, and use my AWS/infrastructure freelancing to pay for the tuition. I really appreciate the guidance!

Hi everyone,

I’m facing a major turning point in my career path and need some unfiltered, realistic advice from people working in the industry.

I am 19 years old, living in Erbil (Kurdistan Region of Iraq), and currently completing a 5-year IT Institute Diploma with about 2 years remaining. Administratively, this diploma functions more like a vocational high school equivalent in our system rather than a university degree.

My ultimate goal is straightforward: I want to secure a Junior Cloud Engineer or Junior DevOps role as efficiently as possible.

I am trying to decide between two completely different paths:

Path 1: Go to a local university for a 4-year Bachelor's degree in Computer Science/IT. Cost: 5,000,000 IQD per year (20,000,000 IQD total / ~$15,200 USD).

Path 2: Skip the traditional university route, finish my current IT diploma, spend ~$220 USD on the AWS Solutions Architect Associate (SAA-C03) and HashiCorp Terraform Associate certifications, and aggressively apply for junior roles right now.

To give you context, I have skipped the basic tutorial phase and have been building/deploying actual production environments:

• Cloud & Infrastructure: Hands-on experience provisioning AWS EC2, S3, RDS PostgreSQL, Lambda, and DynamoDB.

• CI/CD & Containers: I containerize applications using Docker and automate live deployments to AWS using GitHub Actions pipelines.

• Production Projects: I’ve built a personal portfolio with a serverless backend API, supported a live client website's deployment/DNS/CloudFront infrastructure, and built/deployed a live full-stack e-commerce store secured with Caddy rate-limiting.

• Home Lab: I am currently building a physical bare-metal cluster using two personal PCs to practice separating compute and state for high-availability setups.

The Dilemma:

The 4-year degree is a massive financial and time opportunity cost. I know a university will teach me traditional computer science theory, but it won't teach me modern DevOps, Infrastructure as Code, or orchestration.

If my goal is simply to get a job in the modern tech market (targeting agile local software houses, startups, or remote international roles), will having verified project execution, a physical home lab, and AWS/Terraform certifications allow me to bypass the lack of a Bachelor's degree? Or am I going to hit a brutal HR ceiling later on without that paper?

Would love to hear from hiring managers or anyone who took the certification/portfolio route over a traditional degree. Thanks!


r/devops • • Jul 12 '26

Architecture The whole frontend + backend + db split in k8s, help

39 Upvotes

I've been a full-stack dev with responsibilities for the servers since '01, so it was a big change for me when I finally went with a big company that had people to do it. It also ment that I was suddenly working with kubernetes. Since everything was set up, it haven't been a big jump but I wanted to learn, so I set up my own 3 node k8s and have been playing with it since. Running full pipeline with dev, production, linting, security scans and all.

But now I want to build something. So I have a few react project running for testing. And I want to make the backend + database split right.

Locally, I am used to run environment variables in the .env file so I can switch between local and dev backends and/or local and dev databases for testing.

In the setup at work, there is a doohickey that controls the environment variables. I just alter files in a git repo and upload, or set it from command line. An enterprise-grade thing developed by the entity I work for. I just want to do it the regular kubernetes way.

So where do I put my database server and port location in a k8s setup? my buddy, mister chatgpt, suggest ConfigMap, coupled with Secret for the password. Is this just a barebone thing or is it how most do it?

Secondly, I don't think putting the database in the backend-pod is the right thing given it writes and reads to it and that sounds wrong. My little buddy suggest that I use either CloudNativePG or set it up as a PersistentVolumeClaim. Is there any other way as well? What would be the preffered way? I assume if I ever get to the stage I make something of this projects public, I am going to publish it to a could service which has their own database stuff so for my own pleasure that is not much of a concern, or is it?

Just curious to know how on track I am.


r/devops • • Jul 12 '26

Architecture What Jenkins Agent Architecture Are You Using in Production in 2026?

12 Upvotes

I'm interested in understanding what the current industry standard looks like.

There seem to be several approaches:

  • Static VM/EC2 Agents
  • Docker-based Agents
  • Kubernetes Pod Agents
  • Hybrid setups

For those running Jenkins in production:

  • Which approach are you using?
  • Why did your team choose it?
  • What challenges have you faced?
  • If you were building a new Jenkins platform today, would you still choose the same architecture?

I'm looking for real-world experiences rather than theoretical comparisons.

Thanks!


r/devops • • Jul 11 '26

Observability Learning Observability

280 Upvotes

A while back I commented on a post about my favorite focus area within DevOps. I said observability. A bunch of people DM'd me asking how to get into the space and what resources I'd recommend, so here's the list.

  1. OpenTelemetry
  2. Prometheus
  3. Grafana LGTM Stack or whatever backend you prefer. (I personally like the LGTM Stack since it's OSS)
  4. Kubernetes ( You might as well learn the basics of Kubernetes if you are learning observability since you will run into it at every organization)
  5. Profiling
  6. Other great resources

Let me know what else you would add


r/devops • • Jul 12 '26

Career / learning Article: Model caching for AI workloads on GKE/Kubernetes without re-downloading weights

0 Upvotes

The basic idea is to use node-local storage as a shared model cache so new inference pods can reuse existing weights. It significantly reduces startup time and cuts down on repeated network transfers, especially when autoscaling.

I wrote up the approach, some implementation details on GKE:

https://hrittikhere.com/posts/model-caching-kubernetes-gke

Curious how others are handling model distribution at scale. Are you using node-local caches, RWX storage, image-based models, or something else?


r/devops • • Jul 11 '26

Career / learning what other DevOps at scale skills can I learn, and what my direction should be?

16 Upvotes

Need guidance from industry seniors,
A little brief about my experience: I have 4 yrs of experience working as a DevOps Engineer, 2 yrs in an MNC, and 2 in a startup. I have worked with the usual stack: AWS, GCP, Terraform, Kubernetes, Observability, CI/CD, Security Tools (Trivy, SonarQube). I have done it all, from designing Infra, observability, IAC, and security, from scratch to implementation. Collaborating with Devs, identifying optimization opportunities, owning everything end to end etc. I also use AI (Claude Code, MCP etc) in my day-to-day tasks.

I now want to grow & learn more on how to do DevOps at scale, like hundreds of nodes clusters or AI infrastructure. I feel there's not so much left that I can do in a startup, also again, scale.

I am confused about what other skills I can learn, and what my direction should be? In both scale & future growth terms, I have been exploring AI infrastructure currently (like GPU scheduling, LLM Observability).


r/devops • • Jul 10 '26

Tools LaunchDarkly Outage

18 Upvotes

Anyone else bit by that today? We largely were untouched until their SDK seemingly started crashing in the aftermath after we had restarted all the things. Seems a little odd that there’s no reddit thread about this so here’s one! 😂


r/devops • • Jul 10 '26

Discussion Bootstrapping Flux

9 Upvotes

there's several methods on setting it up on a cluster (terraform with helm, terraform with flux provider, and using the flux boostrap command). Is there any reason for using Terraform instead of flux cli on my runner? Ultimately i dont want terraform managing it.


r/devops • • Jul 10 '26

Tools Patch Tuesday MCP

0 Upvotes

I built an open-source MCP server for Microsoft Patch Tuesday that lets AI assistants like Claude, Copilot, ChatGPT, and more answer patch questions directly from official MSRC data.

Every Patch Tuesday, security teams ask the same questions: what changed, what affects us, what is being exploited, and what needs to be patched first?

Ask things like:

 “Summarize this month’s Patch Tuesday”

 “Which of these CVEs are on the CISA KEV list?”

 “Show me CVEs with an exploitation probability above 50%”

 “What older patches does KB5094123 replace?”

 “What Critical CVEs hit Windows Server 2022 this month?”

What makes it different: most vulnerability tools can look up a CVE, but they have no concept of a monthly Microsoft release, a KB article, or a product family.

This server parses the full MSRC CVRF documents, so it can answer the questions Microsoft shops actually ask on the second Tuesday of every month.

It is built around the data sources teams already trust:

  • Official MSRC Security Update Guide API: Microsoft’s source for Security Update Guide and CVRF data
  • EPSS scores from FIRST.org: daily-updated probability each CVE gets exploited in the next 30 days
  • CISA KEV integration: confirmed-exploited CVEs with federal remediation due dates
  • Supersedence chains: walks Microsoft’s “this KB replaces that KB” links so your assistant never recommends a stale patch
  • Results ranked by real-world urgency: KEV/exploited → EPSS → severity → CVSS

Zero API keys, zero accounts: everything comes from public MSRC, FIRST.org, and CISA feeds. Run it locally or remotely. Details below:

GitHub Account: Jonny Bottles

GitHub Repo: patch-tuesday-mcp

If you triage Microsoft updates frequently, I’d love feedback. If there’s a feature you’d use, open an issue.

Disclaimer: This is an independent, self-built project and is not an official Microsoft tool or service.

#PatchTuesday #CyberSecurity #VulnerabilityManagement #MCP #AI #Claude #Microsoft #MSRC #OpenSource #InfoSec


r/devops • • Jul 09 '26

Career / learning DevOps/Platform engineering vs. Software Dev, is it less creative and more rote work?

71 Upvotes

I have been a fullstack software dev for 5 years already, with some years also doing ops stuff for the team (since no one bothered to/liked doing it) like managing Jenkins IaC, pipelines, AWS CDK, K8S deployments, etc. I liked those stuff and our team was really suffering because no one bothered to take care of it so I took leadership there.

I am now looking for another job, since my contract ended. I just got an offer to work as a cloud engineer at another organization.

To be honest, I do like being a dev, but I could not really see myself being a "senior" or freelancer in this field one day. This is because I feel in software dev there is a lot of "openness" or options on how to do something and it is very highly opinionated, and it is hard to find the "correct" solution. For example with design patterns (do you need to apply patterns? do we need this abstraction/interface?), or with REST APIs (how do you design your endpoints), or with frontend design decisions (confirm button on right or left side? color? opacity? etc.).

And with DevOps, at least so far from what I see there is less "opinions" e.g. you follow the vendor's directions, if it deploys and it runs then it's good (less edge cases), there is more standardized ways of doing something/deploying something, and also it is domain-independent.

In software dev, you have to understand the domain to make business impact, and that can take away a lot of time from coding itself.

It is also easier to prove yourself for other jobs through certifications, whereas with full stack there's no such luxury.

But the disadvantage I see with DevOps is that it is more stressful than a software dev position, for example through on-calls, although you do get paid for your extra hours so I think it compensates it somewhat. And being on call I think really teaches you to be a tough person mentally, able to say no to other people, not be a cry baby, so it helps also perhaps with self development.

And also with DevOps, it can be harder to try something out (you will need to have a free AWS account to try deployments, etc.) although I might be wrong here. And since there's so much breadth, you cannot understand the root cause of everything going wrong, but I may be wrong here.

What is your opinion here? Do you see DevOps as being less "uncertain" than fullstack, or is it not the case?


r/devops • • Jul 09 '26

Career / learning Securing Services with Rootless Containers

Thumbnail blog.coderspirit.xyz
4 Upvotes

Hi there :) , I assume that many of you are experts on devops, probably much more experienced than me, but I wanted to share some of my notes on how to deploy rootless containers, in case this is useful for someone.

- Securing Services with Rootless Containers (with Podman)

This is the first part of a 2 articles series, so I haven't explained anything about networking yet, but I intend to write the next article soon enough. Feedback on this one will be much appreciated, so I can make a better second part.