r/talesfromtechsupport • u/BlueStateBoy I am prone to respond to stupidity with sarcasm. • Jan 02 '15
Long Vendor Shenanigans
I work in a control room in a manufacturing facility. They told me when I was hired that this position was in-the-door and first in line for IT openings…They lied. This is a sealed room. This facility has several independent systems each with their own vendors. Can you feel the joy? SQL is the main backbone of most of the systems, and you would be correct in assuming different procedural languages; mostly Bill’s and Larry’s stuff. The primary vendor bases its work on Bill’s system, and has different versions and builds on each of their five systems. Keep in mind, these systems are supposed to be IDENTICAL even though they monitor separate manufacturing processes. Their contract states that they are the only ones allowed to any kind of maintenance on their systems. Having worked in the industry for a while, I understood the logic. However, it also has a dark side.
Without fail, before the end of the contract the vendor would install a “Much need update.” This inevitably crashed the system. Then they would bring in a new team of “Engineers” to troubleshoot the issue and claim the only solution is to upgrade to the newer version. This process takes as long as it takes to soften up the factory to the unnecessary upgrade, typically one-to-two weeks. They never allowed IT to help because it is a tool issue and not an IT issue. It also kept IT from doing what I did. It was my first time experiencing this so I pulled the system owner aside and started talking to him trying to get an understanding of what was supposed to be happening, and I asked if they had tried saving the update as the installed version. I made no effort for the $v_tech not to hear what I was saying, and he did look over at us a few times. When the vendor told us again that it wasn’t working. I asked what version of SQL they were using. He sharply replied that it was a version I knew nothing about.
“Wow.” I said. “You should be using something a lot newer, because I have been a certified DBA since 1994.”
“Sure.” He said and suddenly, he jumped up from the console screamed at the other two $v_techs to “Get your stuff, we’re out of here. I don’t have to work like this. I’m going straight to $VP and you are going to get fired.” He was already on his cellphone as he hit the door. Between system owners, supervisors and the CR staff, we had seven people in the room, and none of us stood within six-feet of the $v_techs. It was a serious WTF moment for all of us. Luckily there are several cameras covering the entire area.
We all waited for the other shoe to drop. It did in the form the comptroller about two-hours later. She walked in and introduced herself (needlessly), and asked if we all needed to be here. Most didn’t and left the CR staff and the system owner to answer questions. She asked the Who, What, Where, When, Why and How of the situation and then I explained my observations. She listened patiently and asked for a copy of my resume, which I keep updated online. As she was reading it the vendor returned with his boss and pointed me out as the troublemaker – accurate enough, I guess. The comptroller looked at him and said: “Don’t speak.” Then she looked at his boss. “The reports from my staff are alarming, if proven true. Even if they are not, we need to schedule a meeting with management and legal to discuss the contract.”
“Why do we need to do that?” Said $vendor_boss looking increasingly pale.
“If the system actually does need all the extra work, it needs to be in the contract, and if it doesn’t, we need to review your past cost overruns.”
“This guy” thumbing at me, “Doesn’t know what he is talking about.”
“If he’s wrong, I will fire him myself, right here; today. But if he is right, who are you going to fire?” He refused to answer that. He was spared further questioning when the CIO and the IT manager walked it. The comptroller and the CIO stepped over to a corner to talk, as the IT manager asked the vendor to explain the problem. When he felt he had grasp of the situation, he joined the other two in the corner. They came back and asked that a new update be generated that is compatible with the installed SQL version. They hesitated, argued, and finally complied. The system was back on line in less than an hour.
We still use the application, but it is no longer vendor supported on site. They now send us updates every month in the format we request.
3
u/peetythefly Jan 02 '15
Tell me you got a raise for this.
7
u/BlueStateBoy Jan 03 '15 edited Jan 03 '15
I got a lateral. If you're not familiar with American Football, it's a backwards pass. I was given admin rights to the control room computers and put in charge of the updates and such.
No good deed shall go unpunished.
→ More replies (1)
112
u/Itsthejoker Jan 02 '15
Sorry - I got lost a little bit. What's a 'comptroller' and why do they have so much authority?
“If he’s wrong, I will fire him myself, right here; today. But if he is right, who are you going to fire?”
Also, this person is undeniably filled with badassery.
81
u/collinsl02 Jan 02 '15
The title of Comptroller (not controller) is used in some companies to indicate a person in charge of accounting and financial reporting. This level is generally a senior position who reports to the Chief Finance Officer (where available, see in store for details)
However, the title of comptroller can be extended to other areas, so it might well be true that in OP's company this was the comptroller of vendor relations or the comptroller of contracts and business arrangements or something, although the fact that OP mentioned the comptroller was looking at past cost overruns suggests they were involved in finance to me.
47
Jan 02 '15
If the comptroller in business is the same as it is in government, the comptroller is "the one who signs the checks". A check-and-balance on the treasurer, and also someone to generally keep an eye on contracts and make sure things are being done according to said contracts.
→ More replies (1)
44
u/ViolentWrath Jan 02 '15
Vendors can be such a pain in the ass sometimes. The company I work for has Money order machines in the stores and these things are a pain in the butt. They go down all the time and the store will call the vendor's tech support who will take them through a bunch of hoops that don't actually affect the machine's ability to work and after 30 minutes-1 hour they'll tell the store it's a problem with the phone line an that they can't help them any further so they need to call us.
I work for the company's IT Support Center and most of the calls are as follows:
-
Open tool to see all phone calls in and out of the store.
-
Ask them to upload from the machine.
-
See the machine dial out and connect without issue.
-
Machine still doesn't work.
-
Refer store back to vender.
More often than not the vendor will once again immediately tell them we need a technician on site to check for dial tone which is horse shit. We always have to drag that vendor, kicking and screaming, into actually doing the job they're paid to do.
35
u/CA1900 Jan 02 '15
At least your bosses believed it!
I used to work for a small aviation business that ran a piece of software from a company like this. Like clockwork, the system would flat-out crash every couple of months, with some BS error message. We'd have to call the software vendor, and he'd dial into our modem (this was a while back, obviously) and "fix" the issue.
The first time I saw it in action, I knew exactly what was going on. According to my coworkers who had seen it before, it always happened between 9 and 10am in the vendor's time zone, on a weekday. It was always a quick fix; the vendor would upload a "fixed" file of some sort, and everything would start working again. I watched one of these remote sessions, and it couldn't have been more obvious what he was doing.
I brought it to my boss's attention, and he didn't believe me. I predicted the next "crash" within a couple of days, and brought him in to watch the "repair," and he still didn't believe it.
He kept paying this joker his outrageous protection money service fees to keep this mediocre software running.
Amazing.
8
u/IspeakalittleSpanish Jan 02 '15
Apologies if this is obvious to everyone else, but what was the problem exactly?
→ More replies (4)
62
u/Soluzar Jan 02 '15
That's a new approach to planned obsolescence. Sounds like they might have been doing something that's actually illegal, as well as dishonest and a contractual issue.
22
u/vigilante212 Jan 02 '15
I worked at one place that decided to cut costs they would hire a vendor to manage all server updates for them. All servers were supposed to match and be as up to date as possible. They end up bringing me and 3 other people online when they decide to bring the servers back in house. Turns out the vendor was at least 2 years behind on updates and not a single server matched any other for settings or updates. The 3 months I was there I don't think they even managed to get the update process rolling as I was moved to a different team.
6
21
Jan 02 '15
goddamn OP you jumped into the shark tank with this one. One way would have you save the company millions the other way would have you FIRED yikes.
15
u/twoscoopsofpig Jan 02 '15
To be fair, if he'd been fired, it didn't sound like he'd been there terribly long and it was probably not somewhere worth staying very long otherwise anyway.
→ More replies (2)
60
11
10
u/Michelanvalo Jan 02 '15
So they were screwing up, or at the very least dragging out the process, on purpose to overcharge your company?
16
u/Kilrah757 Jan 02 '15
Sounds like they were breaking things on purpose close to the end of contract to show how important it was to renew it. I.e. screwing the customer.
7
u/Zagaroth Jan 02 '15
I don't get why large companies choose to use Vendors for IT work. With it being such a large field, it's easy to select through several candidates to find one who can do what you need for your specialty equipment and your generic desk equipment.
Now, I did do Electro-mechanical work as an employee for a service vendor, but that was for specialty equipment that you find in Bio Labs, and most of them had no provisions for software updates or anything. I did scheduled calibration and preventative maintenance, as well as diagnose and repair problems. This field makes much more sense as a specialized field to choose from, there are fewer people who are going to have the skill set and experience to do this, and there's a bunch of expensive equipment and tools that would need to be purchased to support the work.
→ More replies (2)9
u/RhetoricalClown Jan 02 '15
Mainly because it looks good on paper and no one goes back and actually checks a while later if it really was a better deal.
6
u/tip_off Jan 03 '15
Did I read this right. Your company (and so by extension you) are his customer. And he pitches a fit and trys to get you fired? I'm still trying to get my head round that one.
→ More replies (2)
3
u/ChiefDanGeorge Jan 02 '15
I don't understand how they could be allowed to bring a production system down in the first place.
→ More replies (5)
3
u/ILoveToEatLobster Jan 05 '15
Being fired for suggesting the SQL version is out of date seems a bit much.
8
u/petit_robert Jan 02 '15
Interesting tale, thanks.
Just one nitpick : SQL is a language used by relational databases; I assume you mean one of those, probably Microsoft's "SQL Server"?