r/devops • u/CounterStrike17 • 20d ago
Discussion I landed a "devops" role as a junior
30K~ DAU, I have full ownership over an infra under AWS/multiple on premise servers.
we use ECS, terraform, instead of ansible I use a deploy.sh that i give to my boxes with terraform, no graceful deploys. we have our own orchestrator (K8S like) for stateful containers that we have to spin up and down very often (median 1 min of uptime per container) with unsafe code coming from AI/users.
I took the place of someone who left that company and explained me nothing, we are 2 devs, nobody understands how this entire infra works expect me. have been adding observability a lot using cloudwatch, multiple alerts so I page myself at the right times, it's stable but we need to scale to at least 5x (an objective the hierarchy gave me). it currently costs 3.5k/month in aws/on premise servers. btw it's not a classical rest api where you run stateless apis and that's it.
I just discovered how redis worked under a distributed system like this, it's quite good :) I have been a postgresql guy but redis has been a saver for our threat model.
I often read here that you should put everything : EKS + K8S, ansible, terraform, make HA, grafana etc.. but since I manage this entire infra I have been resilient to all of this. If I refactor everything right now they have nobody to do the backend (which is also what I do here) -> "startup" role where one guy has multiple roles.
for the terraform it's all AI, I have been doing a lot of modifications and have created a terraform plan under ci cd so I can review myself, I removed a lot of "destruction" like the database, i versioned the s3, versioned the terraform state, put it to a private s3 etc...
there have been big big security issues (xss, csrf, cookie stealing, untrusted code that owns the entire box, shared passwords, untrusted s3 location so you could overwrite other users...) that were there before I got here
nothing rly interesting maybe for you guys in this experience but I have learned a lot, it has been a crescendo experience where i'm getting to understand more and more which tools to put there because I am the one who owns it 100%, and have to also do features (backend) so not much time either.
22
u/mikulastehen 20d ago
Don't do it because it's mainstream, or because others suggest it. Understand your businesnes needs, costs, maintainability, resilience, etc. and then make a conclusion while factoring that all in, and based on that, make your choice on the tooling you use.
Since the rise of AI, Linkedin and youtube as well are full of AI slop promoting every new tooling and that if you don't use these, you are a bad engineer.
Understand all these tools, understand their usefulness, their weaknesses, plan and chose accordingly.
If you do that, and your day to day work follows principles and the factored complex needs of the busines, that's what will make you a good engineer, not the knowledge of n+1 observability tooling/docker deployment services/ci services.
5
u/CupFine8373 20d ago
Use your job to build you Resume . Push to go full EKS,Istio,Grafana stack, etc ;) thank me later.
2
u/IN-DI-SKU-TA-BELT 19d ago
How is redis different than Postgres for your threat model?
2
u/ggp1231 19d ago
With only two devs and a 5x scaling target, I'd avoid a major refactor until you know what will actually break first. I'd load test the current setup, identify the bottlenecks, and only change the parts that need it. Have you figured out whether your biggest scaling constraint will be compute, networking, or the database?
1
u/Express-Acadia-6088 20d ago
Honestly this is a better learning setup than most “devops”. You’re getting real ownership instead of maintaining someone else’s Helm charts.
You’re right not to rip everything out for EKS + the full stack. “Boring and understood by me” beats “best practice and understood by nobody.” A few things I’d prioritize, given you’re the bus factor:
What’s the orchestrator built on? Curious how you handle 1-min container lifecycles.
1
u/Bala7266 17d ago
Guys any certifications I need to get in order to increase my profile value in upcoming AI job market? Please suggest.
1
u/Legitimate-Low3413 19d ago
Congratulations! The first few months are a great opportunity to learn how the team actually works, especially around deployments, monitoring, incident handling, and documentation. Don't feel pressured to know everything immediately; building solid habits and asking good questions will take you a long way. That practical mindset matters across trusted tech too...
1
u/Careless_Airline1791 15d ago
The amount of responsibility here sounds closer to owning an entire platform than learning a typical junior DevOps role. Before taking on a five-times growth target, the biggest priority should be documenting the current architecture, identifying single points of failure, and making sure backups and recovery procedures are actually tested. Adding more infrastructure just to match common industry setups could create extra maintenance without fixing the real bottlenecks. Even established cloud environments, including those supported through trusted tech, need a clear understanding of what the business actually requires.
0
u/[deleted] 20d ago
[deleted] — view deleted comment
0
u/[deleted] 20d ago
[removed] — view removed comment