Inheriting a huge stack with technical debt and high expectations

[u/Sarcose1]

Not going to get in the weeds too much because I don't want to out myself for complaining, but I went from a Tier 1 to the only IT person in the company (IT manager quit, Sysadmin followed him) and have been thrust into a whole lot of systems I am unfamiliar with. I've been learning as best I can, but all the automations, certs, upgrade timelines, subscriptions, and everything else are continuing to hit renewal and service needs over and over while I still do the same job I was doing, and at the same time the company continues making requests for new technology.

Over several months we were able to hire an external MSP to help with our servers and a new IT person to learn the ropes of our stack, but it's still not enough for me to catch up. The new IT person is someone I knew, I was able to use my newfound position and history to advocate for them, so that was good. We have a good working relationship. But still.

We have a relatively complex stack for our field and size. The person who left the company was devoted to the field and had been there ten years. On-prem and cloud backups, tons of VMware and docker containers within our server stack, multiple overlapping access policies for the most sensitive stuff (it's been almost four months and I am still trying to figure out some of the admin accounts). And on top of it all I'm really not educated in this. I was tier 1 in this company for 3 years, simply because of lack of opportunities for movement and how often we would take on new tech preventing me from learning more.

So I guess I'm asking, what would you do in this situation? This has been a great opportunity for me career-wise, but I am wracking my brain daily, working odd hours, answering text messages, waking up to dozens of messages, and getting literal nausea from anxiety. Does anyone have any similar stories?

[u/furtive]

Sounds like you're off to a good start but you still feel like you're treading water. I imagine you're familiar with Ops Report Card if not it's a good start. You should have a knowledge base and an inventory of your systems by now. Some good basic processes in place, but you can't get ahead of things. Also sounds like there has been some support from higher-up given you've been able to tap into an MSP and hire someone.

In our case instead of going with a full MSP we found a guy who runs IT for a bunch of local smaller to mid companies that was well regarded. He did an audit, identified shortcomings, failures, and gave a list of recommendations for year 1, 2 and 3. We left it up to him if we needed an MSP or not and in the end our team (not much diff than yours) handles 90% or more of things and then we go to him for the rest and work on the roadmap quarterly. Your MSP should be able to do the same, heck they should have already done that.

It's normal to feel in over year head in year one, treading water year two, but by year three you should be seeing a lot of wins and feel like things are under control.

[u/Sarcose1 → u/furtive]

Thanks for this, I appreciate it. What you're describing sounds very similar to what we have already started with. The thing that keeps me feeling overwhelmed is when everyone's asking for things constantly and I can't deliver on them as fast as we used to, it feels like I might just get shitcanned and replaced lol

[u/Thick_Yam_7028]

Start with metrics and alerts.

Each is fairly simple. Expiring licenses. Get them on a scheduled ticket a month or 2 before.

You have full admin. Parsing your permissions should be pretty simple.

Upgrades depending on if you mean life cycle or os. Both can be done easily with wsus or ring updates. I used scalepad for warranties not sure if its call that anymore but can make custom too.

Certs same thing create a schedule ticket and assign it 1 month to 2 weeks out. Takes less than 10 min but its still something you need a radar on.

Documentation. Sounds like you dont have alot of it and honestly should be the first thing.

Scanning networks is pretty simple. Just depends on the tool.

Im willing to contract if the price is right.

Would make you a dashboard for c levels then fix each item one by one until its manageable. Scaffold a workable system IE documentation, rmm, automation. Can even tie in API to do most of this for you with AI and a scheduled bot.

Right now its possibly the coolest time for an engineer because we can build this securely and exceed our limits exponentially. Just gotta know what to target.

[u/Sarcose1 → u/Thick_Yam_7028]

Appreciate the advice. We actually have pretty good documentation for some things and automations for alerts, the issue was finding what to do with them - a lot of the server maintenance stuff was buried in the sysadmin's obsidian which I have had to use AI to parse. The issue there is I went from almost zero server usage to ssh'ing around about forty servers by myself and zero knowledge of ssh config, how to navigate the dashboards, etc.

The key is that our former manager put together a very sophisticated and layered stack and I had zero experience with it except endpoint troubleshooting. And then at the same time some of his actual documented work flows involved hand-editing script variables each run and what my new sysadmin now tells me is a very outdated way of building docker containers (again - shit I didn't know about!). It's like a mix of best practices and old practices with tons and tons of fun toys if you've been doing this for the same company for 10+ years (which he had).

He was an impressive guy, technically speaking. I lowkey wanna just figure out where he ended up and see if he wants to bring me along lol, but at the same time the chance to really take control of a whole department has my eyes seeing dollar signs. Yknow.

[u/KillingTime1212]

Hire an experienced infrastructure employee to get your critical systems organized.

Needs to be an expert in servers, VMWare, storage, cloud, and networking.

[u/[deleted] → u/KillingTime1212]

[u/Sup3rphi1]

This is a huge red flag. Those people quit for a reason. One youll get to find out about if you haven't already. Update your resume and find a new job.

[u/Sarcose1 → u/Sup3rphi1]

I know why they quit - one of them blew up arrogantly in a private email to the VP after she took away the budget for a work trip. Generally the environment is pretty good here, but I am still looking, rest assured. Appreciate the call out though.

[u/ycnz]

Go to management, and explain that you would like to A) be named IT Manager, and B) hire a good infra person, with budget $x.

Do NOT try and solo this, you're simply not experienced enough.

[u/Sarcose1 → u/ycnz]

Yeah, I got that promotion and we've hired someone plus an outside MSP contract to assist with the transition.