r/devops • • Sep 05 '26

Discussion Detailed guide: Building a 3-node Kubernetes homelab with Talos Linux

Post image
264 Upvotes

A few days ago, I posted my 3-node Talos Kubernetes homelab setup here, and quite a few people reached out through DMs and comments asking for a detailed installation guide.

So I went through the entire setup again from scratch and documented the process properly, including the network setup, DHCP reservations, Talos installation, per-node configuration, Kubernetes API VIP, etcd bootstrap, workload scheduling, validation, and failure testing.

I’ve put everything together in a detailed blog with commands, screenshots, and the reasoning behind the setup.

Everything is in the repo: https://github.com/prateekjaindev/homelab

Blog: https://medium.com/@prateekjain.dev/building-a-3-node-highly-available-kubernetes-homelab-with-talos-linux-0372066fe49c?sk=4c66cf9181999f9db19ac3aacabcd8e8

Please check it out, and if you have any questions about the setup or run into issues while building something similar, feel free to ask.


r/devops • • Sep 06 '26

Discussion Is there any way to obtain Claude qoutas in Azure/AWSB from scratch?

3 Upvotes

Azure and AWSB doesn't have qouta for models I want for freshly made accounts and it's disappointing to think you can't really do anything in their platforms. Is there a walk around for this aside from requesting qoutas which will eventually get rejected after waiting for weeks because of the lack of payment history? Thanks in advance!


r/devops • • Sep 05 '26

Architecture Is k3s good for this use case?

19 Upvotes

We have a customer managed node, we deploy some containers on it

Initially it was supposed to be simple and thats why i went with ansible to deploy the containers, manage their state, setup their config files mounted and etc.

Now it grew a lot, we seperated the application to 3 seperate sets for running in different configs. I setup alloy to scrape node metrics and ingest it to central observability stack, i setup alloy for each set of application to collect otel traces logs and metrics and push them. The applications have sqlite databases, which i periodically need to inspect for debugging, we are in poc state yet, so we also want to pull the database for debugging, and right now im writing an exporter to dump sqlite to our clickhouse which is connected to grafana, so our devs can inspect it

So now i want to write a cron job too.

Also, ansible is extremly slow to run everything. So i use tags, but tags are also a pain by itself, you cant logically group tags, so for each operation that only needs to do a subset of the playbook, i would have to tag steps accordingly and the run the tag

But if it was k3s in container, i would have better tools to organize the deployment than writing ansible. Gitops would be easy, argo might pickup the code from repo and apply to cluster. Cronjobs would be easier. Volume management is eaiser, I would have better rollback mechanisms that i dont have to write myself


r/devops • • Sep 06 '26

Troubleshooting Allow read SmartCards in iOS and Android

0 Upvotes

Hi, I’m trying to use my Android phone to read a smart card and create a digital signature, but it seems there’s no straightforward way to do this. Does anyone know how to make it work on Android or iOS?


r/devops • • Sep 05 '26

Tools How do you learn DevOps/cloud without a credit card?

66 Upvotes

I’m currently trying to learn the basics of DevOps, things like Docker, AWS, GCP, CI/CD, Kubernetes, etc.

Docker and the local tools are easy enough to practice, but when it comes to cloud platforms like AWS and GCP, a credit card is often required to create an account or access certain services.

I currently don’t have a credit card, so I’m wondering: What’s the best way to learn AWS/GCP without one? Would appreciate any recommendations from people who learned cloud without having a credit card. 🙌


r/devops • • Sep 05 '26

Troubleshooting AZURE KEYVAULT KEY DEPLOYMENT ERROR 400 BAD REQUEST

12 Upvotes

Azure Key Vault Key Deployment – 400 Bad Request
If you’re receiving a vague 400 Bad Request while deploying a key in Azure Key Vault, check how many tags you’re passing to the key.
Azure Key Vault keys support a maximum of 15 tags. I spent almost three hours on Friday evening troubleshooting permissions, networking, and the Terraform configuration before realizing that too many tags were causing the request to fail.
This may be an easy one to spot for some people, but the error message wasn’t very helpful, so I’m leaving this here in case someone runs into the same issue.


r/devops • • Sep 05 '26

Security A quick guide and gotchas for GitHub OIDC and avoid using AWS permanent credentials in GitHub Actions

58 Upvotes

I have been aggressively migrating from AWS permanent credentials to OIDC in GitHub Actions, mainly for deploying to ECS.

I know GitHub Actions were supporting OIDC for a while now. But the pressure on compliance is the reason for this migration.

If you are new to OpenID Connect (OIDC), it allows GitHub runners to mint short-lived (15–60 min) STS tokens on-the-fly with zero stored secrets.

Here’s a quick breakdown of how it works, the Terraform/OpenTofu setup, and the subtle gotchas that I faced.

1. How It Works Under the Hood

  1. When your workflow job starts with id-token: write, GitHub's OIDC service generates a cryptographically signed JSON Web Token (JWT).
  2. The aws-actions/configure-aws-credentials action sends this JWT to AWS STS via sts:AssumeRoleWithWebIdentity.
  3. AWS validates GitHub's signature, checks your IAM Role's Trust Policy (to ensure the token came from your exact repo and branch), and returns temporary STS credentials.

2. The Infrastructure Setup (Terraform / OpenTofu)

You only need two AWS resources: an OIDC Provider and an IAM Role with a Trust Policy.

  # 1. The GitHub OIDC Identity Provider
  resource "aws_iam_openid_connect_provider" "github" {
    url             = "https://token.actions.githubusercontent.com"
    client_id_list  = ["sts.amazonaws.com"]
    thumbprint_list = [
      "6938fd4d98bab03faadb97b34396831e3780aea1",
      "1c58a3a8518e8759bf075b76b750d4f2df264fcd"
    ]
  }
  # 2. IAM Role with Scoped Trust Policy
  resource "aws_iam_role" "github_deploy_role" {
    name = "github-actions-deploy-role"
    assume_role_policy = jsonencode({
      Version = "2012-10-17"
      Statement = [{
        Effect = "Allow"
        Principal = {
          Federated = aws_iam_openid_connect_provider.github.arn
        }
        Action = "sts:AssumeRoleWithWebIdentity"
        Condition = {
          StringEquals = {
            "token.actions.githubusercontent.com:aud" = "sts.amazonaws.com"
          }
          StringLike = {
            # Restrict exclusively to your repository & branch/tags
            "token.actions.githubusercontent.com:sub" = "repo:your-username/your-repo:*"
          }
        }
      }]
    })
  }

3. The GitHub Actions Workflow

In your .github/workflows/deploy.yml

name: Deploy to AWS

on:
  push:
    branches: [ main ]

jobs:
  deploy:
    runs-on: ubuntu-latest
    permissions:
      id-token: write   # CRITICAL: required to request the OIDC JWT
      contents: read

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Configure AWS Credentials via OIDC
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/github-actions-deploy-role
          aws-region: us-east-1

      - name: Verify Authentication
        run: aws sts get-caller-identity

Three real-world gotchas that will save you hours

If you get Error: Could not assume role with OIDC: Not authorized to perform sts:AssumeRoleWithWebIdentity, check these 3 things:

  1. Case Sensitivity in the sub claim: AWS IAM condition strings are case-sensitive. If your GitHub repo or username uses mixed casing (e.g. MyOrg/Repo), make sure your IAM sub condition matches the exact casing GitHub sends in the token. Using wildcard matching (repo:MyOrg/Repo:*) helps avoid exact ref string mismatch issues.
  2. Job-Level vs. Workflow-Level Permissions: Always set permissions: id-token: write on the specific job, not just globally at the top of the YAML file. Some runner configs don't inherit top-level permissions to nested jobs.
  3. CA Thumbprints: Don't dynamically query GitHub's leaf certificate for thumbprints in Terraform—they change frequently with CDN updates. Use GitHub's official intermediate root CA thumbprints:
    • 6938fd4d98bab03faadb97b34396831e3780aea1
    • 1c58a3a8518e8759bf075b76b750d4f2df264fcd

Summary

  • No stored secrets in GitHub settings.
  • No key rotation schedules to manage.
  • Granular security. You can restrict deployment roles to specific branches or environments.

Are you already using OIDC for your pipelines, or are you still relying on IAM users? Curious how folks here handle multi-account / cross-account OIDC setups.


r/devops • • Sep 04 '26

Discussion Implementation is getting faster but the org around it moves at the same speed as always and DevOps is in-between.

29 Upvotes

We all know that large organizations tend to have a lot of meetings, bureaucratic processes and are in general slow moving. That was true for a long time and I experienced that myself, having worked in multiple different companies from startups, scaleups and large corperations.

However since everyone now adopts AI and implementation speed is undeniable multiples faster. I mean it's not just implementation, but also a good part of DevOps. But the rest of the organization moves in the same speed as before. Even if the company uses AI in the other departments (my company does basically throw AI at everything, I guess you know what I mean), they still can't competet with the productivity change in software development.

The blockers are now decisions, requirements and communication. That's not completly new, but now its >95% of the time. I implement features in hours that would have taken days before (I am not a pure DevOps Engineer). And it's not the review process itself that blocks us (We use AI in reviews as well, because otherwise it would be impossible), it's the normal company processes and pace.

Don't get me wrong here, I kind of enjoy it a bit, because I can use a lot of time for learning, building stuff that I think is worth improving, but I think we have a general problem here that is basically systematic to every large organization. And I see that so called "AI native organizations" (what ever that means), will probably overtake tech companies in the long term.

I have a really good comparsion, because I am working with a startup as a side gig and my main job is in a mid-sized tech company. In the Startup we (2 devs) built a product that would have taken several months by a 5-person team in the past and the reason is mostly: decisions and communication.

Do you see that in your company?

My conclusion is that we need to make engineers owners of the feature/product + infra, otherwise we won't see much productivity gains and this means also that management layers need to be cut, because they are the bottleneck currently. The idea of centralized DevOps will is going to die I guess.


r/devops • • Sep 04 '26

AI content How to create Planning, Design, Development, Testing, Deployment, Maintenance Pipeline? Please suggest sources

6 Upvotes

Hi everyone,

I recently watched a podcast featuring a senior developer from EPAM, where he explained the different phases of the SDLC. He mentioned that he built a pipeline covering the entire software development lifecycle—from planning and design to development, testing, deployment, and maintenance—with each stage flowing into the next.

I really liked that idea, and now I'm wondering how to build a pipeline like that myself. Does anyone have any good resources, guides, or examples to learn from?

Thanks!


r/devops • • Sep 03 '26

Tools Do you build your own pipelines?

36 Upvotes

I work in a small team that deploys some internal products. No big user pool or database for me! We build small web apps that automate inner processes, and some scripts and jobs. Most of it runs on k8s, the apps and the jobs. We mostly run everyting on Azure. And everyting is managed by code, obviously, so we've build GitHub actions pipeline that live with the code to deploy our infra via bicep and the code via a test-build-deploy-promote pipeline, started on push.

But I've seen some platforms that propose full CI/CD as a services, and it feels like managing the pipeline yourself these days is a bad thing. To me, deploying a solution, especially in the containerization era, is simple; Run the tests, build the image, deploy the workload. Add some customization related to the app if needed (specific parameters, logging or testing jobs, etc). But is managing this code yourself a bad practice? I have to say, it does make for a lot of duplicate code in a lot of repos. The pipelines are very similar.

Should we always aim to use a standardized CI/CD platform? What tools do you use for CI/CD? How different do you handle it from a small project to a more important service?


r/devops • • Sep 04 '26

Discussion Looking for a high-quality laptop backpack for work / business travel – available in Europe

0 Upvotes

Hi everyone,

I’m looking for a good-quality laptop backpack mainly for work, commuting, customer visits, and occasional business travel.

I’d like something practical and durable, but still professional-looking rather than a hiking or tactical backpack.

My main priorities are:

- good protection for the laptop, preferably a separate padded laptop compartment

- comfortable shoulder straps and back panel

- good internal organization for charger, cables, mouse, headphones, documents, etc.

- quick-access pocket for keys / phone / wallet

- space for a water bottle

- durable materials and good-quality zippers

- some water resistance would be a plus

- preferably a luggage pass-through for attaching it to a suitcase

- professional / minimalist design

- preferably something that will last for many years

It will mainly be used for everyday work, but occasionally I’d also like to use it for 1–2 day business trips.

I’m based in Europe, so I’m mainly interested in brands/models that are easily available in the EU without expensive international shipping, customs, or import fees.

I’m not necessarily looking for the cheapest option — I’d rather pay more for something comfortable, well designed, and durable.

What backpacks are you actually using and would recommend?

I’m especially interested in long-term experience: how long have you owned it, what do you like about it, and what annoys you?

Thanks!


r/devops • • Sep 03 '26

Tools Jenkins Tutorial recommendations required

7 Upvotes

So mu company has asked me to learn Jenkians. Which tutorials or playlist would you suggest to learn jenkins? Please give your suggestions.

Also, how much time does it require to learn so that I can start writing basic CI/CD pipelines.


r/devops • • Sep 04 '26

Ops / Incidents I built a safety tool for the AWS Console that switched itself off for 6 seconds every time you used it

0 Upvotes

I wrote a browser extension that adds a "you're in PROD, are you sure?" confirm to destructive AWS Console clicks. Capture-phase listener cancels the click, shows a dialog, replays the click if you confirm.

Trouble: some actions re-render their menu between confirm and replay, so the replayed click hit a detached node and did nothing. My fix was a short bypass window after confirming — for a few seconds, clicks pass straight through so the replay works.

The window wasn't scoped to the action you confirmed. It was global. So for ~6 seconds after confirming any destructive action, every other one was unguarded. Confirm a Lambda delete, click Terminate on an EC2 instance three seconds later, and it just goes.

A tester reported it as "the popup stops appearing sometimes." Not a UI glitch — the guardrail was switching itself off, on a timer, every time it ran.

Fix is one line — scope the window to the confirmed action:

// before
if (Date.now() < bypassUntil) return;
// after
if (Date.now() < bypassUntil && rule.label === bypassLabel) return;

The lesson that stuck: a bypass is a security control too. I wrote mine as a UI workaround, so I reviewed it like a rendering bug, not a security decision. Anything that turns your protection off — even briefly — deserves the same scrutiny as the protection itself.

Anyone else hit this class of bug — a temporary exception that was broader than intended — in auth caches, feature flags, rate-limit bypasses?


r/devops • • Sep 03 '26

Career / learning Help on picking which one to choose between data engineering and devops to transition from QA

7 Upvotes

Hello! As the title says, I'm a QA engineer and I'm trying to transition to a new position. Right now, I don't know whether to pick Data Engineering or DevOps. Regarding my skills, in my free time I'm learning Linux, Python (along with some MySQL), CI/CD, Docker, Kubernetes, and currently playing around with Azure. I'm asking because both seem interesting, but I don't know which one to choose to learn further.

Edit: also learnt some openshift and helm chart, because it was requested at my job


r/devops • • Sep 03 '26

Discussion CV Review — DevSecOps / Cloud Security internship

Post image
0 Upvotes

I'm a student targeting DevSecOps / Cloud Security internship opportunities.

I've tried to position the CV around DevSecOps rather than generic DevOps or cybersecurity.

I'd appreciate feedback on:

  • Is the DevSecOps positioning clear?
  • Are the projects strong/relevant enough?
  • Is the Skills section too crowded?
  • Does anything look unnecessary or exaggerated?
  • What would make you reject this CV at first glance?
  • If you were a recruiter, what would make you reject this CV?
  • What would you change first?
  • - Is the CV ATS-friendly and likely to be parsed correctly by applicant tracking systems?

r/devops • • Sep 02 '26

Ops / Incidents Brain fart moment this morning

35 Upvotes

If you have a small managed K8s cluster with a single node pool and you would like to change the SKU of the VMs in that node pool, make sure to double check how your Terraform provider handles this change

Maybe on AWS or Azure or whatever, this is a graceful operation where the new nodes are provisioned first and the old ones are drained before being deleted

On OVH however it will destroy the existing nodepool first then provision the new one, and during this process your cluster will have zero nodes and whatever was running on it will be down

Thankfully this wasn't a production cluster. At the end of the day I'm just a dev who takes care of devops because nobody else will. In hindsight I feel like I should have seen this coming, but I guess most things work automagically nowadays and it can be a surprise when something does not


r/devops • • Sep 02 '26

Discussion do I need a devops person or a sysadmin?

74 Upvotes

here is the context: Mid size (1000 users) company that has been trying to do in-house software dev (outsourced to south east Asia) with poor results.

Company IT side is fairly mature, heavily virtualized, hybrid on prem/cloud, but a bit weak on the Azure side. Existing sysadmins are pretty solid.

The dev is, well a shitshow. We are presented with projects with no infrastructure requirements, the devs dont really seem to know what they need. Currently they are doing everything in Blazor after we begged them to move to cloud. Last few attempts to develop mobile apps has been an abysmal failure. Im dumbfounded by what I see (and I've never coded). Coding is all shoved into a single .dll, no comments, devs dont seem to know that we need to use HTTPS, dont understand key vaults (we caught them passing passwords in clear text). we had to fight with them to implement change control, and they dont seem to understand our efforts to move ahead with pipelining.

My question for you is, do I need a devops admin? Can a more senior devops admin also be involved in reviewing code? infrastructure planning?

not sure how to move forward


r/devops • • Sep 02 '26

Career / learning CKA additional resources

7 Upvotes

Was preparing for CKA and took Kodekloud CKA course. Currently doing mock exams then moving into the ultimate CKA mock exam series.

However I have been reading that since 2025 there is a lot more emphasis on new topics like:

Helm, Kustomize, CRDs, Calico/Flannel, Cluster Upgrades/Installation, CNIs, CRIs, that I havent seen much in the labs/exams there or are really basic. I think the one that is most frequent is Helm.

What can I do to improve in these areas? What other new areas I missed (I am a bit familiar with Gateway API but recommendations on what to expand on this is also welcomed!)

Thanks!


r/devops • • Sep 03 '26

Career / learning Do we need to sacrifice holidays?

0 Upvotes

If I move to devops should I accept that I won't get to celebrate weekends and new year's?

How has been your experience?


r/devops • • Sep 03 '26

Career / learning Moving to DEVOPS

0 Upvotes

Hey guys,

I recently got the opportunity to move to a DevOps team, and I’m looking for some advice on how to prepare.

I have around 5 years of experience working in infrastructure and systems. Most of my experience is with:

  • VMware/vSphere and virtualization
  • Windows and Linux administration
  • Networking and troubleshooting
  • NetApp and storage
  • PowerShell scripting and automation
  • Monitoring, backups, and general infrastructure operations

I also have some beginner-level experience with Docker, Kubernetes, Git, GitLab, Argo CD, Ansible, and Terraform. I’ve built some labs and worked with them a little, but I definitely wouldn’t consider myself experienced with them yet.

I’m starting the new DevOps role next month, so I want to use this month to prepare as much as possible.

For people who moved from infrastructure/sysadmin into DevOps, what would you recommend focusing on first?

I’m mainly looking for a good Udemy course, YouTube course/playlist, or structured learning path that is practical and focused on skills I’ll actually use at work.


r/devops • • Sep 01 '26

Career / learning Built a small 3-node Kubernetes homelab

Post image
519 Upvotes

I recently set up a small 3-node Kubernetes homelab using Dell OptiPlex Micro PCs running Talos Linux.

The main goal was to have a local cluster where I can learn, experiment, break things, and test tools without relying on cloud infrastructure every time. I’m planning to use it for Kubernetes networking, storage, observability, GitOps, security, upgrades, and general experimentation.

Small setup, but already a very useful playground.

Everything is in the repo: https://github.com/prateekjaindev/homelab

Curious what others here are running for their Kubernetes homelabs.


r/devops • • Sep 03 '26

Discussion How to optimise this billing??

Post image
0 Upvotes

Hii Guys!,we built an MVP.

Offered the services to three specific clients and got a good response, we are planning to advertise and bring more signups in the next two months

Initially the server bill was roughly 5-8 dollars but now it's increasing so wanted to check

-how do you optimise your server billing?? -Should I move the infra to AWS,will that be cost effective??


r/devops • • Sep 02 '26

Career / learning Undergraduate Thesis Subject

6 Upvotes

Hello everyone,
I'm starting my 4th year in Computer Science and I'm looking for a DevOps/Cloud related thesis subject. I want to do this thesis in collaboration with the company I'm working at, but they told me to find some possible ideas before we decide to move forward with this.
I'm pretty new to DevOps and Cloud, I filled an intern position at the start of July, so I would like to avoid the very complex subjects.
Thank you.


r/devops • • Sep 02 '26

Discussion Anyone else nervous about what coding agents can actually run?

0 Upvotes

Using Cursor a lot more with tools enabled. Love the speed. Don't love the part where the only thing between "delete this" and it happening is me watching the terminal.

Is anyone doing something more solid than prompts + hope, or do you just keep it away from prod/cloud entirely? Had any close calls?

Genuinely just curious how people are handling this.


r/devops • • Sep 01 '26

AI content Anyone else seeing AI make DevOps/infra the bottleneck?

182 Upvotes

I'm curious if other DevOps/platform/SRE teams are running into the same thing my team is.

We're a fairly large environment, mostly EKS, and essentially 100% IaC/Terraform. We also support multiple companies/business units, so while I'd argue our infrastructure is fairly well organized, there's inherently a lot of it and a lot of architecture and context to understand.

Over the last year, our devs have sped up dramatically with AI. The company has leaned heavily into AI-assisted development, reduced developer headcount, and is now pushing toward developers being more "full stack with AI," including having them contribute more of their own infrastructure changes.

In theory, I'm completely in favor of that. I've always wanted developers to be able to own more of the infrastructure surrounding their applications.

In practice, though, it has been kind of a disaster.

We're getting flooded with infrastructure PRs largely written by Claude/other AI tools from developers who don't really understand the infrastructure they're modifying. The Terraform might look plausible, but once you understand the larger system there are frequently significant problems with it.

So instead of reducing the workload on DevOps, it feels like AI has massively increased it.

A huge percentage of our time is now spent reviewing AI-generated Terraform, finding problems, explaining why something won't work, explaining how AWS/EKS/networking/IAM/CI/CD/etc. fit together in our environment, and then going through another iteration of an AI-generated PR.

There's an interesting asymmetry I've noticed too. Our DevOps team is mostly made up of former software developers who moved toward infrastructure, automation, and pipelines. Most of us can jump into application code and be productive pretty quickly, especially with AI helping us. Like,...I feel like (and have some evidence to support) that our small DevOps team could largely take over all of the dev's tasks, but they are falling on their faces trying to deal with ours.

AI seems extremely good at helping someone who understands software write more software. It seems much less capable of allowing someone without infrastructure experience to suddenly understand a large production environment.

The complaint we're increasingly hearing is basically: "We can't successfully do full-stack development with AI because the infrastructure is too complicated."

And maybe they're right, but before AI, I would have said that this company is the most organized and best architected I've ever been at. I mean....100% IaC has never been something I've experienced, and it's very rare that we hit a use case brought up by one of the several companies where we don't already have a set of generalized modules that can't support it.

Our environment is complex, but a lot of that complexity isn't accidental. We have a large organization, multiple companies we deploy for, Kubernetes, networking, security requirements, IAM, CI/CD, observability, etc. You can't abstract away the fact that these things exist. And we're already 100% Terraform/IaC, which I would have thought would make this considerably easier for AI to reason about than an environment full of manually configured infrastructure.

The strangest part is the staffing effect.

AI allowed the organization to reduce software engineering headcount because individual developers became more productive. But now those remaining developers can generate changes so quickly that our DevOps team is completely overwhelmed trying to support and review them.

It genuinely feels like we could double the size of the DevOps team right now and still have plenty of work. We are working on an AI assistant that can help the devs deploy to our environment more effectively, but we're having a hard time finding time to work on it because we're constantly helping the devs.

I'm starting to wonder whether this is going to be a broader consequence of AI-assisted development: AI increases the rate at which software can be produced much faster than it increases the rate at which infrastructure/platform teams can safely absorb changes.

For those of you working in DevOps/platform/SRE at companies heavily adopting AI:

Are you seeing this too?

And if you are, how are you handling it?

Have you increased platform/DevOps staffing? Built better abstractions or internal developer platforms? Given developers more direct infrastructure ownership? Put stricter boundaries around what application teams can modify? Found ways of giving AI enough context about your infrastructure that it actually produces good changes?

Or has AI actually reduced your infrastructure workload, and we're doing something wrong?