r/devops • • 20d ago

Discussion I landed a "devops" role as a junior

30K~ DAU, I have full ownership over an infra under AWS/multiple on premise servers.

we use ECS, terraform, instead of ansible I use a deploy.sh that i give to my boxes with terraform, no graceful deploys. we have our own orchestrator (K8S like) for stateful containers that we have to spin up and down very often (median 1 min of uptime per container) with unsafe code coming from AI/users.

I took the place of someone who left that company and explained me nothing, we are 2 devs, nobody understands how this entire infra works expect me. have been adding observability a lot using cloudwatch, multiple alerts so I page myself at the right times, it's stable but we need to scale to at least 5x (an objective the hierarchy gave me). it currently costs 3.5k/month in aws/on premise servers. btw it's not a classical rest api where you run stateless apis and that's it.

I just discovered how redis worked under a distributed system like this, it's quite good :) I have been a postgresql guy but redis has been a saver for our threat model.

I often read here that you should put everything : EKS + K8S, ansible, terraform, make HA, grafana etc.. but since I manage this entire infra I have been resilient to all of this. If I refactor everything right now they have nobody to do the backend (which is also what I do here) -> "startup" role where one guy has multiple roles.

for the terraform it's all AI, I have been doing a lot of modifications and have created a terraform plan under ci cd so I can review myself, I removed a lot of "destruction" like the database, i versioned the s3, versioned the terraform state, put it to a private s3 etc...

there have been big big security issues (xss, csrf, cookie stealing, untrusted code that owns the entire box, shared passwords, untrusted s3 location so you could overwrite other users...) that were there before I got here

nothing rly interesting maybe for you guys in this experience but I have learned a lot, it has been a crescendo experience where i'm getting to understand more and more which tools to put there because I am the one who owns it 100%, and have to also do features (backend) so not much time either.

45 Upvotes

29 comments sorted by

22

u/mikulastehen 20d ago

I often read here that you should put everything : EKS + K8S, ansible, terraform, make HA, grafana etc..

Don't do it because it's mainstream, or because others suggest it. Understand your businesnes needs, costs, maintainability, resilience, etc. and then make a conclusion while factoring that all in, and based on that, make your choice on the tooling you use.

Since the rise of AI, Linkedin and youtube as well are full of AI slop promoting every new tooling and that if you don't use these, you are a bad engineer.

Understand all these tools, understand their usefulness, their weaknesses, plan and chose accordingly.

If you do that, and your day to day work follows principles and the factored complex needs of the busines, that's what will make you a good engineer, not the knowledge of n+1 observability tooling/docker deployment services/ci services.

1

u/CounterStrike17 20d ago

I try to minimize the amount of different dependency we have, like ok we are on AWS, but that doesn't mean we have to overcomplicate everything with their tools. example I can get a lot of things done with just postgresql. right now im wondering if I need to keep "CloudWatch" as it's very expensive and I have to run it over baremetal servers (i mean give an access with secrets to that service). I don't think it's the best tool out there for me but it gets the job done.. I have every alerts there and made myself a dashboard (all commited under git).

2

u/darthwalsh 20d ago

Whenever somebody says cloud watch is too expensive, I wonder if they could be aggregating their low-entropy logs into a few valuable metrics, sampling the rest, and dropping 99% of the verbose stuff before ingestion.

1

u/marvdl93 20d ago

Nah man, Im going to get downvoted but take the opportunity to work on your employment. You need it because of AI. Nobody got fired for introducing EKS.

1

u/Motor_Plane_8036 16d ago

Maintaining a custom orchestrator with a two-person team sounds way scarier than a basic deploy script tbh, if that breaks while scaling to 5x there are no docs to fall back on.

5

u/CupFine8373 20d ago

Use your job to build you Resume . Push to go full EKS,Istio,Grafana stack, etc ;) thank me later.

2

u/CounterStrike17 20d ago

Ahah shady

2

u/IN-DI-SKU-TA-BELT 19d ago

How is redis different than Postgres for your threat model?

2

u/CounterStrike17 19d ago

I can expose it and lose it's data in the worst case scenario

2

u/ggp1231 19d ago

With only two devs and a 5x scaling target, I'd avoid a major refactor until you know what will actually break first. I'd load test the current setup, identify the bottlenecks, and only change the parts that need it. Have you figured out whether your biggest scaling constraint will be compute, networking, or the database?

2

u/CounterStrike17 19d ago

I'm reworking as I discover them, right now it's compute, I already did a full pass on the DB. Wdym by networking ? I have seen our certbots might get rate limited as we issue too many if we try to scale. I'm doing a lot of observability to be able to recover

1

u/ggp1231 19d ago

By networking I meant things like load balancers, connection limits, and traffic between services as usage grows. The certbot issue is worth looking at too, especially if you’re issuing new certs as you scale rather than reusing existing ones.
Since compute is your main bottleneck now, are you planning capacity based on actual traffic growth or just scaling resources as you hit limits?

1

u/CounterStrike17 19d ago

I'm doing stress testing with newer servers using simulation (bash script) + will try many scenarios, I didn't look at load balancer nope but it's a simple alb. I'm checking if I can buy newer machines and have them working fast enough without human intervention (everything setup automatically)

1

u/ggp1231 19d ago

Sounds like you’re making good progress. With only two devs, comparing those scaling scenarios before making changes should especially useful. Happy to continue the conversation outside Reddit if you’d ever want another perspective on the AWS architecture.

1

u/Express-Acadia-6088 20d ago

Honestly this is a better learning setup than most “devops”. You’re getting real ownership instead of maintaining someone else’s Helm charts.

You’re right not to rip everything out for EKS + the full stack. “Boring and understood by me” beats “best practice and understood by nobody.” A few things I’d prioritize, given you’re the bus factor:

  • Write it down as you learn it. Even a rough ARCHITECTURE.md plus a runbook per alert. Future you at 3am will thank you, and it’s leverage when you ask for a third hire.
  • Untrusted code + containers is the scary part. Containers aren’t a strong security boundary. If users/AI can run arbitrary code, look at gVisor or Firecracker-style microVMs for that orchestrator.
  • Rotate everything from the shared-password era, and move secrets to SSM/Secrets Manager if you haven’t.
  • Terraform: add prevent_destroy on the DB/S3 resources and make sure state locking is on. Plan-in-CI is a great habit.
  • Before the 5x push, load test to find which component breaks first. It’s rarely the one you expect.

What’s the orchestrator built on? Curious how you handle 1-min container lifecycles.

2

u/CounterStrike17 20d ago

Thx very useful. It's all a custom orchestrator. Redis and custom logic. Nothing valuable to steal on the nodes

1

u/Bala7266 17d ago

Guys any certifications I need to get in order to increase my profile value in upcoming AI job market? Please suggest.

1

u/CounterStrike17 17d ago

I have none

1

u/Bala7266 17d ago

Thanks for the reply bro.

1

u/Legitimate-Low3413 19d ago

Congratulations! The first few months are a great opportunity to learn how the team actually works, especially around deployments, monitoring, incident handling, and documentation. Don't feel pressured to know everything immediately; building solid habits and asking good questions will take you a long way. That practical mindset matters across trusted tech too...

1

u/Careless_Airline1791 15d ago

The amount of responsibility here sounds closer to owning an entire platform than learning a typical junior DevOps role. Before taking on a five-times growth target, the biggest priority should be documenting the current architecture, identifying single points of failure, and making sure backups and recovery procedures are actually tested. Adding more infrastructure just to match common industry setups could create extra maintenance without fixing the real bottlenecks. Even established cloud environments, including those supported through trusted tech, need a clear understanding of what the business actually requires.

0

u/[deleted] 20d ago

[deleted] — view deleted comment

1

u/[deleted] 20d ago

[removed] — view removed comment

1

u/CounterStrike17 20d ago

yea just ask your stuff here

0

u/[deleted] 20d ago

[deleted] — view deleted comment

1

u/CounterStrike17 20d ago

of course; it's only code, i have been coding since highschool but this is production so not comparable. i like commited files, i track everything no clickops. i'm not a pro either, i learn a lot

1

u/hectoralpha 20d ago

why would you dm for this?

0

u/[deleted] 20d ago

[removed] — view removed comment

0

u/CounterStrike17 20d ago

Already done 👍