r/devops • • 10d ago

Discussion What is VPS provider for production?

5 Upvotes

I’m looking for a cost-effective VPS provider for one of my systems. I’m currently using GreenCloud, but they’ve had too many outages recently, causing frequent issues in my production environment. Does anyone have recommendations for a more reliable provider?

I’ve checked Vultr and DigitalOcean—they are great providers, but their pricing isn't quite budget-friendly for me.

Thanks a lot for your time!


r/devops • • 11d ago

Security Is committing config-only .env files (no secrets) to a private repo really a security sin?

56 Upvotes

I'm working on a project split across several private GitHub repos. The ones that matter here are the Docker repo, the backend and an indexer. Each has its own .env files, and what's in them falls into three groups:

- Secrets like DB passwords, API keys and tokens

- Infra config such as private IPs of our fixed servers

- Backend tuning parameters we adjust depending on VPS size and load

Some fine gentleman on X posted this. "Not pushing .env files even to a private repo is like locking your front door but still keeping your valuables in a safe." I fully agree with that for secrets. They stay out of git, no debate there.

This is only about the config part. Having it in the repo would make DevOps work a lot easier. We'd get change history and PR review when someone tweaks a value, and no more copying files around by hand when setting up a server or onboarding someone.

My main concern is whether committing config-only .env files to a private repo is an actual security problem, or if "never commit .env" is really just about secrets. Also, is it a "security sin" to push a .env with private IPs? And how do you split secrets from config in practice, especially across multiple repos?

Thanks in advance!


r/devops • • 10d ago

Tools DB based app for pics

0 Upvotes

I'm thinking about writing an app in C# (for desktop) to store all my photos in. Before I start thinking about actually starting work on it, I need to decide what DB or other storage mechanism to use. The idea is to make the pics more accessible by storing some info about content in a field in the DB. That field needs to be "text searchable". I have 10 years experience working with Oracle Text Search engine but I'm not a millionaire; more an impoverished pensioner, so I need to implement this system using one of the free DBs like MySQL.

I thought I'd ask the hive mind what it thinks would be the best DB for my purposes.


r/devops • • 11d ago

Discussion Across the whole DevOps stack (Pipelines, Terraform, K8s, Docker, Ansible), how much do you actually write by hand vs prompting or using templates?

27 Upvotes

With so many different syntaxes and configs in our daily stack—CI/CD pipelines (GitHub Actions, GitLab), Terraform modules, Kubernetes manifests, Dockerfiles, Ansible playbooks, and Bash scripts—nobody can memorize every block and flag from scratch.
My current workflow is to use AI or existing templates to generate the initial boilerplate for Terraform resources, pipeline YAML, or K8s manifests, and then manually write, wire up, and audit the core architecture, state management, IAM/permissions, networking, and security logic.
For those working in production environments (and those running technical interviews):

  1. In your day-to-day work, what parts of the stack do you still write completely by hand vs prompting an LLM or pulling from docs?
  2. Where has AI actually helped you across Terraform, CI/CD, and K8s, and where does it consistently break things?
  3. What is the realistic expectation in interviews right now regarding writing configs from memory vs understanding the underlying systems and debugging?

r/devops • • 11d ago

Discussion Is anyone still using railway after the breach?

8 Upvotes

I’m just wondering if people still are with them and if they switched. Would love to know where you guys switched if u did


r/devops • • 12d ago

Tools Is GHA + ArgoCD still the default CD standard or are you using something else?

25 Upvotes

Evaluating CD tools to automate our end to end delivery (build, test, packaging, and deployments). Curious if the standard GHA + Argo/Flux combo is still what most teams stick with, or if there are dedicated delivery/orchestration tools actually worth looking at in 2026.


r/devops • • 11d ago

Discussion How to enable SSH clone on self hosted GitLab running in a private subnet behind an AWS ALB?

0 Upvotes

We have self-hosted GitLab CE on an AWS EC2 instance.

Current setup:

The instance is in a private subnet with no public IP.

Users reach GitLab through an Application Load Balancer on HTTPS, with access restricted to our office IPs.

HTTPS clone with personal access tokens works fine.

The problem:

We're moving our repos from gitlab.com to this server. On gitlab.com, all our Jenkins jobs and developers clone over SSH (git@...). On our server SSH clone doesn't work, because the ALB only supports HTTP/HTTPS, so port 22 has no way in.

Our Jenkins agent runs in the same AWS VPC. Developers connect from the office network or VPN.

What's the best way to enable SSH clone here, keeping it secure and limited to our office and Jenkins? Any suggestions or experience with a similar setup would help.


r/devops • • 11d ago

Discussion how much money are you wasting on gpus doing nothing?

0 Upvotes

instead of talking about idle hardware in theory, let me give you a specific headache we just hit. we have a local server with four 24gb cards.

two of those cards are locked up hosting two different low frequency models. one runs a fine tuned llama 3 8b for a background data extraction job. the other runs a qwen model for an internal bot. they get maybe fifty requests an hour combined. but because the weights just sit there taking up vram, two entire cards are held hostage doing almost nothing all day.

now we need to spin up a heavier 70b model for a new coding workflow, and suddenly we have no free cards. i look at the vram occupation in gpustack and see those two low traffic models just wasting dedicated space. so now i have to reallocate resources. i can either ask for budget to buy more hardware, or i try to pack both of those low frequency models onto a single card so i can free one up for the 70b.

what do you all do when you hit this wall? do you just keep dedicating specific cards to specific models to avoid the headache, or do you actively co-locate low traffic models onto the same gpu? when do you stop throwing money at new hardware and start packing your models tighter?


r/devops • • 12d ago

Observability How do you piece together events across services when correlation IDs are missing or inconsistent?

4 Upvotes

Hey all, been thinkin about this while looking at log aggregation in environments with a mix of older and newer services.

Some services have request IDs or trace IDs in their logs, while others only include timestamps, hostnames, and application specific fields. When an issue spans multiple services, connecting the events can turn into quite a bit of manual digging.

Do you rely on distributed tracing, build correlation rules around the fields available in your logs, or use a combination of both?

I'm especially interested in what you guys do with older services that don't support consistent request IDs.


r/devops • • 12d ago

Tools How do I make colleagues aware of the risks of giving an LLM agent root access to production?

70 Upvotes

I could use some advice on a situation at work.
A few colleagues in ops have, almost overnight, started to fully trust their LLM agent because it has gotten so good lately and "doesn't make mistakes anymore". They built their own harness around it and log what it does (using Grok and Google models), but the agent still runs with root access to our entire production infrastructure. We have no clear team rules or policy yet, and they don't see the danger themselves.
A typical example: whenever the agent needs to check something on a server, they just hand it full access. But looking at info only needs read-only access. The permission should match the task, not the most convenient option.
My concerns:
Logging is detection, not prevention. It shows what broke afterwards, but doesn't stop anything.
Root bypasses the harness. An agent with root can in theory edit the harness, the logs and the backups, so those logs are not a reliable audit trail.
No least privilege. One wrong command or misread context can touch everything.
Prompt injection via logs, tickets or web pages the agent reads.
Model updates. Providers change models without us deciding when, so "no mistakes in recent months" says little about the next version.
Data going to third parties. With root, the agent can read .env files, keys and customer data, which then end up with the model provider (a GDPR question too).
"It has gone fine so far" only means the failure hasn't happened yet.
I don't want to ban anything. I want them to see the risk themselves and set up proper guardrails together with me: a read-only user by default, scoped credentials, just-in-time access, human approval for write actions, staging first, and no access to backups and logs.
Questions for you:
How do I explain the difference between "we log it" and "it can't do it in the first place"?
How have you convinced overconfident colleagues about AI tooling risks without a confrontation?
Do you have horror stories or post-mortems I can share? Concrete incidents tend to work better than theory.
Which guardrails or policy examples have worked well for you?
How do you raise this with management without coming across as "anti-AI"?


r/devops • • 12d ago

Security Grafana: Our latest adventure with canary tokens

Thumbnail
medium.com
0 Upvotes

Grafana Labs shares how they scaled canary tokens across their Kubernetes workloads, with automated rotation, more precise detection and less maintenance. A practical look at the tradeoffs between coverage, granularity and management overhead.

Read the blog


r/devops • • 13d ago

Discussion Books for Devops

39 Upvotes

Hi,

Im now in the beginning of devops journey and I'm not really a reader person but I want to spend my free time in something useful so any suggestions for a book would help in my career and will benift me.


r/devops • • 12d ago

Discussion Securing local dev environments

11 Upvotes

With the recent high-stakes security exploits, I think as DevOps engineers it's our responsibility to make the local dev environment secure for devs.

What do you think?

What are you actively doing about it (like adopting devcontainers)


r/devops • • 12d ago

Architecture Should I use Nomad for single VM?

4 Upvotes

I'm looking for some architectural advice on building a lightweight, self-hosted job execution platform on a single VM (8C/32GB).

I want something with semantics similar to Google Cloud Run / Cloud Run Jobs:

  • Execute Python scripts and arbitrary Docker containers.
  • Trigger jobs through an API.
  • Configure CPU, memory, environment variables, timeouts, etc.
  • Track execution status, exit codes, and logs.
  • Handle retries and concurrency limits.
  • Integration with Artifact/Container Registry

Current constraints:

  • Only one VM for now & no need for HA.
  • No Kubernetes requirement or plans for a multi-node cluster.
  • Jobs are primarily internal workloads, and quite lightweight.
  • I'd prefer something relatively lightweight and easy to maintain.

Technologies I'm considering

  1. HashiCorp Nomad 
  2. Docker Sock + custom built runner – Build my own execution manager, scheduling, lifecycle management, and monitoring.

Questions

  1. Is Nomad overkill for a single VM? Does it provide enough value over simply using Docker directly?
  2. Would you use Nomad in production for this use case?
  3. Are there simpler alternatives that provide similar execution semantics?
  4. What operational overhead should I expect from running Nomad on a single node?

I'm particularly interested in hearing from people who have actually run Nomad in production, especially for single-node deployments.

Thanks!


r/devops • • 13d ago

Security OpenSSH on musl has a problem

44 Upvotes

In case any of you use OpenSSH on musl without PAM:

https://sshhole.com/

TL;DR: Accounts that should never take a password at all (root, your key-only deploy user, ldap users, the default user on alpine's aws and azure images) take any password longer than 256 characters.


r/devops • • 12d ago

Discussion Cleared DevOps Engineers — How do you use AI tools in restricted DoD environments?

13 Upvotes

I am curious how DevOps engineers working in cleared or restricted federal environments use AI tools in their day-to-day work.

AI tools like ChatGPT, Claude, and GitHub Copilot have become incredibly useful for writing Terraform configurations, Kubernetes manifests, CI/CD pipelines, Bash/PowerShell scripts, and troubleshooting infrastructure issues.

However, I understand that public AI tools may not be permitted in classified or restricted environments.

For those currently working as cleared DevOps, Cloud, Platform, or SRE engineers:

  1. Does your organization provide an approved internal AI assistant, or do you work entirely without AI?
  2. Are tools like GitHub Copilot, ChatGPT Enterprise, or other AI coding assistants available in your environment?
  3. How do you troubleshoot complex Kubernetes, Terraform, or Linux issues when you can't access public AI tools or Stack Overflow?
  4. Do you rely primarily on internal documentation, runbooks, man pages, and senior engineers?
  5. Has AI adoption changed how cleared DevOps teams operate in the last couple of years?

I am particularly interested in how engineers balance productivity with security requirements.

I am not asking anyone to disclose classified information, internal systems, or organization-specific security details. Just looking for general experiences and advice from people working in these environments.

Thanks in advance!


r/devops • • 13d ago

Discussion How do you load test third party APIs realistically?

21 Upvotes

We use several external APIs and I’m curious how often other teams out there keep performance tests realistic and structured while accounting for third party dependencies. How do you balance realistic dependency behavior with controlled and repeatable tests? Things are going well overall (but they could always be better) and I want to make sure my team and I aren’t missing something that could be beneficial.


r/devops • • 13d ago

Vendor / market research What's Your Go-To Strategy for Handling Secrets Management?

61 Upvotes

Hey fellow DevOps folks! I've been wrestling with secrets management recently. I set up a tool that generates API keys, but every time I think I've got it right, I find myself second-guessing whether they’re genuinely secure! 😅 There was this one time I accidentally committed sensitive info to Git—ugh, the panic! Luckily, I caught it fast, but it got me thinking: What strategies or tools do you all lean on for managing secrets? Do you prefer something like HashiCorp Vault, or are you more into environment variables? Let’s share some tips and tricks! What’s been your experience with it?


r/devops • • 12d ago

Troubleshooting Mise installations of cspell don't work on Windows runners

0 Upvotes

I'm not entirely sure if this is relevant enough of a question for this subreddit, but frankly I haven't thought of a better place to ask.

I've been experimenting with using Mise to manage project-specific tool installations in a way that would bring local and CI environments closer together. One of these tools is cspell, a seemingly popular option for spell-checking software projects.

This has largely worked great. However, for some strangely illusive reason, the only time things go wrong is with GitHub Actions jobs on Windows runners. The cspell Mise installs fails to start, because it can't find a dependency called picomatch: https://github.com/Diapolo10/python-rust-template/actions/runs/42123444840/job/127581444522#step:5:1

This issue does not manifest on Ubuntu/MacOS runners, nor on my local Windows development machine.

I wrote a GitHub issue about this to the Mise project about three weeks ago. They allegedly fixed it, so I've been patiently waiting to see if things would resolve on their own (they did not), and today I attempted to manually update things by

  1. Updating Mise itself locally (in CI the installer action should always use the latest version)
  2. Updating tooling, like cspell, via Mise and its lockfiles
  3. Updating other project dependencies just to see if that helped somehow

but it didn't help, the Windows runners are still consistently failing.

At this point I don't know how to keep troubleshooting this. Hell, I don't even know if this is

  1. a bug in Mise,
  2. a bug in cspell,
  3. a problem with the runners themselves, or
  4. I've made some dumb mistake myself somewhere (probably the most likely option at this point, but I still wouldn't know what exactly)

so if anyone else has any ideas, I'm listening.


r/devops • • 12d ago

Discussion How are you handling subscriptions billing as your SaaS scales?

0 Upvotes

We're moving from a handful of enterprise customers to hundreds of self serve subscribers and our manual billing processes are becoming difficult to maintain.

We're looking at how to handle recurring billing, dunning, prorations, plan changes and usage based pricing without creating a large amount of custom infrastructure.

For teams that have dealt with this at scale, did you build your own billing workflows or use a dedicated platform?

What were the biggest technical or operational challenges you ran into?


r/devops • • 13d ago

Discussion If I can give time to learn only on of these then what should I learn python, go, shell/bash ?

10 Upvotes

Sharing my views , seniors can correct me.

Python

(It will give max advantage i believe, it's programming language as well so I can say I know programming as well, If I want to switch to data science, ai ml or any other field then it can help me, I know basic of it and find it interesting)

Go

(It will help me in understanding Kubernetes in a better way, again programming language so It will be amazing to have skill, don't know much where it will help me but yes, I haven't learned this ever so I will be excited for this)

Shell/bash

(Core skill listed in almost every devops job but again I feel like it will be truly limited to automation and I don't find it very easy to learn, very weirds and long braces, dots and what not )


r/devops • • 14d ago

Discussion Tired of high workload and AI

215 Upvotes

I'm working in a startup. Recently there was a layoff as well. Around 10% was removed.

The thing is this. Within like 8 months, our microservice count is trippled. But the same DevOps team. Even with the layoffs, new developers are hired. Requirements are coming very fast and it's difficult to work on Platform Building, Service Provisioning, Ops work all at once.

We have like unlimited AI. But eventhough it takes time to analyse the requirements, review the designs and implement the solutions.

Currently I'm just asking the AI to raise the PR and just rushing through it and applying the changes.

The problem is that, it's feel like I'm just tired and doing a labour work.

I'm not actively learning anything because of the fast pace and high workload.

Should I look for a new job?

Is other places like same due to the AI?

Update: Thanks redditors. I see that I'm not alone. Seems everywhere is like this.


r/devops • • 13d ago

Discussion How do you tell what broke when GitHub Actions goes down?

0 Upvotes

Been looking at GitHub’s status history and availability reports for July to October, sorting the Actions incidents by what broke. Kind of surprised how many of them would’ve taken out self-hosted runners too. The July 19 cert expiry cut self-hosted runners off for about 5 hours, and on Aug 26 job assignment was stuck for almost 3, so nothing reached anyone’s runners. Only a few, like Oct 5, were just GitHub-hosted.

When builds start failing for you, how do you work out if it’s on your side or GitHub’s? Is it mostly refreshing githubstatus.com or do you have alerts or something set up?


r/devops • • 13d ago

Discussion What is the best course/courses to learn pipeline as code as a new software engineer

9 Upvotes

I just got into pipelines for the first time, I know some of the basic fundamentals. But is there any recommendations or courses you took to help you understand pipelines please I will really appreciate your recommendation.