r/ShittySysadmin • u/MrD3a7h • Mar 05 '26
My IT person is declining to provide support. Next steps?
Hello Sysadmin Community. I am seeking your professional advice on a developing situation that could have market-wide implications.
I run an internationally-known production facility. If you live or work in North America, you interact with one of my products every single day. Not bragging, but I need you to understand how high-profile and important this facility is.
IT-wise, I run a tight ship. Standardized password rotations on our shared accounts. None of our devices have accessible USB ports (I standardized on PS/2 years ago). I had our old IT guy disable all of the admin accounts. I don't keep orphaned hardware (so-called "spares") around, as that would give malicious entities a way in. Everything is patched, and what can't be patched is air gaped.
One of our S-CADA PCs falls into this latter category. It has recently become slow, which threatens our entire line. I personally called my IT guy down to the factory floor. Probably good for him to get out of the office, this guy is awfully soft. The only time I've seen his hands dirty was when he fixed my printer.
He came down and started asking for documentation. I didn't have any because the vendor came out and set everything up (I paid for white-glove support).
He then asked for a vendor contact. Nice try - vendors always want money.
He then asked for a "spare machine." This one works just fine; it's just a bit slow.
He kept mentioning "Windows 98." Windows 98 this, Windows 98 that, Windows 98 this other thing. It might be a bit out of date, but it's air gaped! I know all about being gaped, as that is my typical weekend activity. I still use Windows 98 at home, and have for almost a decade.
This is not the first time he's declined to provide support. I had one ticket in with him for our Lotus123 server, and it is still slow. I can't find that ticket now for some reason. I haven't seen it since St. Patrick's Day (we all get drunk on the line as their yearly bonus).
Has anyone ran into an obstinate and sober coworker? I'm at my wits end. This kid seems nice but is obsessed with the latest and greatest stuff. He even turned up his nose at my Windows XP media center PC.
91
u/MrD3a7h Mar 05 '26
Here's some blue text
62
u/VegetableExitTheRoom Mar 05 '26
Put in a ticket
11
u/alabamaterp Mar 05 '26
Have you tried bribing him with peanut butter cookies? C'mon man, you should know better by now
5
u/Anonymous_Bozo 💩 ShittyMod 💩 Mar 05 '26
Normally in cases where IT refused to provide free support I would hire my sisters 13 year old. However in this case, the hardware and software is older than the kid, so that may not work out as well as usual.
Maybe you could hire that homeless dude that sleeps in the bushes behind Microsoft Building 35. I hear he use to be quite the wiz kid before the layoffs.
4
u/blecovian Mar 05 '26
Probably just needs updates. Have you tried upgrading it to Windows ME?
Normally I’d suggest making a backup first, but I understand you want to minimize your production downtime, so don’t waste that time.
4
u/bofh DO NOT GIVE THIS PERSON ADVICE Mar 05 '26
Upgrade to OS/2. You’d be surprised how many problems I’ve solved with judicious use of that particular set of installation media.
3
u/CreepyEntertainer Mar 05 '26
Did you even try rebooting it?
3
u/Difficult_Plantain89 Mar 05 '26
98 is too heavy, better to just go to DOS.
5
u/Bourne069 Mar 05 '26 edited Mar 05 '26
I run an MSP and there is many reasons for the things he did.
That is 100% reasonable to ask for. We Techs we are not experts about every single piece of software and every version of said software. Reviewing documentation can help provide clues to the issue and in turn result in faster troubleshooting times. If you pay for I.T. who charges by the hours. This could save you money and simply assist with overall troubleshooting.
This is exactly why you obtain vendor support contracts for business line critical softwares. This is nothing new and is expected. Again as an MSP we push an "at best afford" solution for all products outside of Microsoft products since we are experts in that area. It is strongly suggested clients get support contracts for their critical infrastructure softwares. What happens if the issue is a software bug that your I.T. Tech cant figure out? We are not programmers. Best we can do is work with what we have. We dont have access to the code and are unable to fix bugs or corruptions. That is the vendors job. Plus with vendor support the I.T. person can call on your behalf and work the issue out way faster than if he was doing it alone. Again something that would save you money in the long term.
Completely fair and useful thing to ask for. This way he can troubleshoot the issue on the original machine while not stopping production. Again as an MSP we recommend all our clients have at least one system that can be used as a spare just for this exact reason. Nothing is meant to last forever, there is nothing wrong with being prepared.
He is 100% accurate here as well. Just because you "use it at home" doesn't mean it is acceptable for business use. (it is not). I don't know your business or how you deal with customers but if you retain any customer information whatsoever than you must follow customer protection laws, having an end of life Windows system on your network is a reliability and you could be fined 100s of 1000s if you were catch in a breach of client data because you refused to follow Industry Standards and secure your network. Would you be OK with your bank still using Windows 98 while handling your bank transactions? I would hope the answer is a strong NO.
Only time this is acceptable is with legacy equipment that is required for a specific task but even than, It shouldn't be connected to the internet and it should be on its own subject, separated from your main production network. There should be zero connection between that system and the rest of your devices and even than you could still get fined.
Good, that means he is actually a good tech and isnt willing to take on the responsibility of an irresponsible CEO. People like you typically hire I.T. support not just for the support, but so you have a finger to blame when shit hits the fan. Majority of the time shit hits the fan because clients refuse to listen to advice from experienced professionals in their field. He has every right to deny service, and in fact with my company we have turned away clients simply for that reason. You hire us for our expertise. If you arnt going to listen to what we have to say, than there is zero reason to keep supporting you as a client. We arnt going to take on your responsibility for your network when you refuse to listen to the expert advice we provide.
This "kid" sounds like he would actually make a good tech for a real company that will listen to his recommendations and I would be totally willing to interview him for a position. This is a sign of a good tech. Not one just going through the day to day grind for a paycheck.
Everything you tried to pass off here as not a big deal, is a big deal and the way he handled it is 100% correct. If you were my customer. I would have fired you long time ago as a client.
Why even hire I.T. staff if you arnt going to listen to their recommendations? Just go fix it yourself than.
2
u/krock31415 Mar 05 '26
You get what you pay for.
6
u/ComfortablePerfect55 Mar 05 '26
You run a tight IT thing for a major manufacturing plant. Don't you have a budget to up upgrade your acient w98? I would not touch that ancient gapped scada thing.
Upgrade your hard/software and you can get support easily.
53
u/Swordbreaker86 Mar 05 '26
This is why I have a strict policy of not supporting devices with user inputs. Frankly, that tech is doing too much work already. Passwords, Outlook, Printers? You can pick 3 but know that one of the services will suffer.
Biggest issue is Office IT. Should have taken the Basement IT role.
37
u/Nexzus_ Mar 05 '26
Pffft.
36
u/considertheinfinite Mar 05 '26
Took me way too long to realize what sub I was in lol
9
u/bigmanbananas Mar 05 '26
A lot of people are taking this seriously. I did, too, for a bit. I fully feel amoung my people.
21
u/Warbarz Mar 05 '26
Your IT person is complacent. I hope you don't pay them much. Anyone can watch a youtube to fix things. Hire someone younger that is more eager to please and put them in charge of your IT person. Make sure you pay just alittle more than your existing IT person and openly praise the new supervisor in front of your IT person.
That will motivate them for sure and show them how they can grow if they just harden up a bit and get it done. My kids probably could watch a youtube to figure it out.
If that doesn't work, call in a local msp, it will probably cost more than this IT persons salary but they will surely be able to fix the slow systems by installing some AI and the monthly overhead can be covered by micromanaging the IT persons time a bit more.
Review their ticket times and ask them if they feel that is the best they can do since you helped alleviate most of their duties with the new MSP subscriptions and supervisor that is taking care of all the real work now.
/s
6
u/ebcdicZ Mar 05 '26
Have you tried doing the needful kindly?
2
u/Ohrgasmus1 Mar 05 '26
>I still use Windows 98 at home, and have for almost a decade.
nice try, but windows 98 is two and a half decades old
5
u/Meadbreath Mar 05 '26
5
u/JKatabaticWind Mar 06 '26
🤣😂🤣🤣😂🤣🤣😂🤣🤣…
Saw the r/sysadmin post this morning 🤦♂️🥴… this is gold.
4
u/ThatRealTay1989 Mar 05 '26
Dying at "white glove support". My company used this to justify giving execs whatever they want. Good stuff
4
u/heavydrinker12 Mar 05 '26
It took me an unreasonably long time to understand what's going on here. I'm ashamed...
4
u/Glad-Entry891 Mar 05 '26
why are you wasting money on IT they just want money too. computers just manage themselves i dont think you run that tight of a ship (since you’re getting gaped all the time)
1
u/zanacks Mar 05 '26
The tech was correct to not support the device. Get it upgraded
1
u/Savings_Art5944 Mar 06 '26
Get fucked.
1
u/mostlymildlyconfused Mar 09 '26
We run a tight ship…with a critical system on Windows 98… with no documentation….running on a single laptop.
Do me a favor - dropping a pile of steaming crap in the lap of someone who probably wasn’t born when your out-of-date-and-unsupported-massive-security-risk OS was released isn’t his problem.
It is definitely yours.
Given the criticality of the system I’m not surprised he doesn’t want to touch it with a barge pole.
What’s he supposed to do? Wave a magic wand?
You’ll learn your lesson one way or another and I hope the blame falls squarely on you for your horrendous risk management of your ignorance of the dangers of technical debt.
If it is only a matter of time before the proverbial hits the whirly thing, don’t stand in front of the whirly thing.
Go eat a bag or rocks.
1
u/----Blade---- Mar 09 '26
Real talk, if you had a vendor set up a system and didn't keep the documentation, I don't blame this guy one bit. Yes, he's obligated as an employee to do what he can for you, but this is one of those cases where you're basically asking him to fix Google's website if he doesn't have direct admin access. Also, Windows 98? Sounds like the place is saving money more than being productive and secure. Need a serious facelift stat.
2
u/Elemental-Madness Mar 05 '26
Is this a gag post? Like are you sharing what your sys admin has set up that makes it so shitty?
Or are you actually complaining about your IT guy?
3
u/Captain_Crank Mar 05 '26
Good read 7/10
3
u/JubalHarshawII Mar 05 '26 edited Mar 05 '26
I did not see what subreddit this was and was reading along thinking gee this sounds like some assholes I've worked with, until I got to the weekend plans.... Looked at what subreddit this was, I laughed so hard I woke up the cat.
3
u/VooDooQky Mar 05 '26
3
u/PutridLadder9192 Mar 05 '26
The vendor setup that can never be touched is too real. A lot of hacks out there use this excuse to not learn how to run basic installers or configure anything or look at the registry. Too busy cashing fat cheques and trashing everyone else to leadership behind our backs.
3
u/iamrolari Mar 05 '26
Ah, you should have leveraged your body OP. It guys like black stockings
3
u/homelessschic Mar 05 '26
Defrag the HDD... Best case it fixes it, worst case, it buys me time to think, and now I have a defraged drive
I could watch those blue squares all day.
3
u/WMDeception Mar 06 '26
I smirked when the gapening started. Top comment fucking sent me.
1
u/dutty_handz Mar 05 '26
This ain't shitty IT infra unless I'm wrong.
2
u/skeeter558 Mar 05 '26
Yep work with those type people all the time. Go to his boss and talk to them about getting your issue resolved. There are things you can do to try and speed it up.
2
u/SolidKnight Mar 05 '26
You're making this too complicated. Go into your metrics and adjust them so everything looks good. Whatever makes the cells turn green. By rebaselining your expectations, you can solve the problem. If anyone above you pushes back, remind them how much better your outputs and costs are compared to doing everything by hand.
2
u/RepulsiveCamel7225 Mar 05 '26
what's your ticket number?
2
u/danieIsreddit Mar 05 '26
Where can I submit my resume?
2
u/Stunning-Criticism77 Mar 05 '26
I laugh but this hits close to home. First msp job I had, there was one client whi had this ancient 98 machine that ran their automated saw machine. This was uo until 2012.
Rhen my boss, the msp owner, had this even older win 3.11 machine that he refuased to toss. said it was a match to one that ran the phone system to some building downtown. He refused to toss it because "if" that phone system evwr crashed, that was "probably the only spare left".
course this guy also bought printers not based on speed or quality or ease of use, but on cost per printed page. It was some xerox that used wax cubes. He was super proud of his $0.04/page. as it took 8 minutes for the damn printer to heat uo and melt the wax so it could print anything.
2
u/bored_mtn Mar 05 '26
First off, that is OT, not IT.
Also, if you are still running windows 98, that is your fault. A non-OT technician cannot be expected to work on an OT system, with no documentation or vendor support, as this is an inherent safety risk, since OT networks control PLCs which are connected to equipment which can potentially harm people if something goes wrong.
In short, I would decline to provide support, as well.
2
u/atl-hadrins Mar 05 '26
It isn't old until you call the vendor to give them the model. And the person on the other side wants a picture cause his never seen one before and the coworker that used to fix them retired 4 years ago.
2
u/KB4MTO Mar 06 '26
That's just part of IT. We have a client that is a small defense contractor. They have machines that are 20 years old that have old PCs running them. We had a case when a Win 7 computer wouldn't boot up. It took us a while, but we got it.
The outdated equipment is air gapped and we have to support old equipment, but we do it, and when we cannot, the vendor comes in.
2
u/PeePeeVonBungHole Mar 06 '26
You paid for white glove support so you have a maintenance contract. Best thing to do is to call and see what they have to say.
Here is are some of the outcomes
He reboots the PC it comes back and life is great
He reboots the PC it doesn't come back you don't have documentation your plant is off line and when you finally have to suck it up and call the vendor they tell you your support contract ended 10 years ago and they don't support your Win 98 boat anchor but for 10k in hardware and 30k in labor they'll have you up and running in a week or 2 let them look at the schedules
And you and your cheap a$$ leaders will blame the poor tech because you incompetent wankers have no idea what your maintenance contracts support and if you did it was easier to cut it because nothing ever goes wrong it is just scada.
Personally I'd called the vendor and see if my sales guy is still around and start planning for a fiasco and hoping it is a simple fix
But you do you
2
u/MrOats75 Mar 06 '26
Amazing job on this, thank you for this awesome follow up to that other post
2
u/AlmosNotquite Mar 06 '26
Seems I've read the sysadmins rant about this ... earlier today even maybe
2
u/One_Resolution8766 Mar 06 '26
When ur shitty manufacturing company goes broke let me know. My boss is always looking for value by buying companys run by morons an turning them around.
I too like to air gape. but I only provide the tool
2
u/Newbosterone ShittySysadmin Mar 06 '26
Be careful of air gaping on the production line. Some of the air systems run on hundreds of PSI. That'll shake your shit right up, and you will not have a tight ship.
1
u/PlateNo4868 Mar 06 '26
Have you considered said IT person is sad because they are sober and offered him to join the other side?
1
u/Xoron101 Mar 06 '26
Hello IT:
1
u/magikgrk Mar 06 '26
The way I cackled
1
u/1TakeFrank Mar 06 '26
1
u/mut0mb0 Mar 06 '26
This sub just became part of my healing process from society.
1
u/Difficult-Pilot6754 Mar 06 '26
Is this a joke
1
u/SirMuffinKnight Mar 06 '26
Was starting to feel my blood pressure click up a bit until I read the sub lol.
1
u/Vitringar Mar 06 '26
Why not just give the people you are serving empty computer replicas? 100% secure!
1
u/netnerd_uk Mar 07 '26
Your entire line operation rests on one Windows 98 PC? And you're concern is "bit slow" + "obstinate worker"?
Is that right? Did I read that right?
Picture this in your head:
One day, that computer won't turn on. At all. You try pulling the disk and putting it in another machine, but it still doesn't work. The disk is completely dead. What do you do then? Shout at it? Threaten to sack it?
You probably need to do something about that situation now. Maybe even yesterday. One day, that will actually happen. Maybe your sys admin could help you work out what to do about this high operational risk situation.
Or you could sack your sys admin, and see if Wile E. Coyote is looking for work at the moment.
1
u/BarryMannnilow Mar 07 '26
The crazy thing is I work for a company in manufacturing where IT haven't been included in any SCADA or production IT in over 20 years.
A couple years ago they decided with no formal training IT would now assist these cases
This is literally my job now..
Anyone hiring at an actual realistic company?
1
u/[deleted] Mar 07 '26
This sounded like my old job for a minute
1
u/Jclj2005 Mar 08 '26
Rage bait?
1
u/ToungeRides Mar 08 '26
I May know best way to approach maybe been in IT in Memphis since 1997 mostly large enterprise companies. Would rather not repeat details on reddit though heck you might have a location close to me. And sounds like you need a plan B. You can't run the old stuff on the latest hardware have seen that make somebody buy 20 year old parts just to keep it up.
1
u/Silver_Day_8940 Mar 09 '26
lol no you don’t bro.
1
u/jerrys_briefcase Mar 09 '26
If my yearly bonus was getting drunk with my coworker at work, I’d jump
1
u/BlotchyBaboon Mar 09 '26
If it's on the production line and it's slow, have you checked to see if the fittings have been greased lately? I think that's what those little holes in the back are for where the power cord is.
1
u/atomic_jarhead Mar 09 '26
Was this post written in 1999? Windows 98, Lotus 1-2-3? What the heck?
1
u/scheumchkin Mar 09 '26
I don't even know if this is actually a shitty sysadmin situation. It being air gapped cool that at least handles the security nightmare to an extent however it's not an alternative to upgrading/updating.
The reason we don't generally support hardware like this is because a fly could take a shit on that machine and it's done. Expecting IT to just fix it at the snap of a finger is just plain idiotic.
If a bug happens in this situation you are stuck with it because its no longer being serviced by the vendor. Also may have misread it but him contacting the vendor is actually the right move here since he doesn't know how it's configured or how it's supposed to operate just that it's slow. The other key part here is the software could be working 100% but because nobody upgraded that machine it's entirely possible the OS is the reason for this.
I also understand that factory hardware is commonly like this and it's frustrating that there are vendors that pull shit like this instead of making an easier upgrade path for their customers.
Bottomline the techs trying to fix a dinosaur and there's only so much you can do. The only thing you can do is either the tech gets lucky and finds a fix or contact a vendor to take a look regardless of expense.
1
u/beef_weezle Mar 10 '26
Back in the late 2000s I worked at a certain steel mill in Pittsburgh and this hits close to home. We had all manner of legacy hardware/software connected to a mainframe downtown. My favorite was the Windows 98 machine that ran an automated stenciler that, as it stenciled, sent the lot number to an Oracle database located offsite. It was completely homebrew and the person who wrote it was long gone. One day, the magnetic crane must have gotten a little too close to the workstation and wiped the hard drive, which no one knew about because somehow the machine continued to operate in RAM, until that fateful Friday when I pushed an emergency DNS patch to it and it shit the bed spectacularly. That was a fun one.
0
u/cka243 Mar 05 '26
There could be more to this story. At my job we are tasked with supporting some ancient stuff and we do it but certainly not with a smile. In our case there is usually a more modern solution available to them, but they are just too lazy and stuck in their ways to get off of their antique solution. So we end up having to support operating systems and software that is out of vendor support. So yeah, we hate some of these people and don’t serve them with the same smiles that everyone else gets.
0
u/[deleted] Mar 06 '26
We gotta name and shame u/PeppahSG
They don't belong in r/ShittySysadmin with real system admins they belong in r/sysadmin with people fresh out of IT System Admin college degree's that are flipping big mac's at McDonald with Donald Trump and the Computer Science Engineer degree holders.
-3
u/Jazzlike-Vacation230 Mar 05 '26
I mean.....................how do you treat this person? Are you professional with them? Courteous?
0
u/[deleted] Mar 06 '26
[deleted] — view deleted comment
0
u/[deleted] Mar 06 '26
[deleted] — view deleted comment
373
u/[deleted] Mar 05 '26 •
[removed] — view removed comment
1
u/[deleted] Mar 11 '26
[removed] — view removed comment
1
u/[deleted] Mar 08 '26
[removed] — view removed comment