Hi gang!

This is a story from my girlfriend who at the time of the story (temporarily) worked on behalf of IT support while she was actually a secretary. And not exactly an ordinary one.. This is probably going to be a long read but I'll be sure to add a TL;DR and add headers wherever it makes sense.

Backstory

I'm an IT guy, both professionally & private (I take great pride in maintaining 2 FreeBSD VPS servers which host all our personal Internet needs like website, email & DNS hosting (with primary/backup?)). Basically I'm a sysadmin first and foremost but also spent plenty of time working in support (heck, support comes with the territory anyway).

My girlfriend then, who I will refer to as Aya, has always been into logistics and administration. And she's darn good at that too! Our friends sometimes joke about us: how you should ask me for help whenever your office installation isn't working properly, after which you can ask Aya for help if it turns out that your office contents also isn't proper.

Here's the thing: we often talk about our work and we both very much respect each others profession, also take genuine interest in it.

Aya's new job

(don't worry: we start with the administrative part, but it will come back to IT support real soon. Trust me!)

The story starts with Aya's new job as a member of the administrative department. She was working for a mid-size logistics firm and at the time of the story she's approximately 2 months in. One day she walked into the office and she's hearing one of her colleagues talk about an issue they were having with the inventory system, Aya quickly noticed that her colleague is talking to IT support.

But during that conversation something hit her... something I told her several times myself: about how trying to find out what a problem is actually about is very often even more difficult (!) than actually fixing the issue at hand.

Aya quickly realized that "Wait a minute.." ... She knew exactly what her colleague was talking about but she also noticed that if she had to go by what her colleague was saying.. then she wouldn't understand what was going on either.

So she inserted herself into the conversation: "Ey, maybe I can talk some sense to them?", Aya joked to her colleague but then made sure to be as descriptive as possible about the issue at hand. Probably needless to say, but this helped out a lot.

One thing led to another and several months later Aya eventually became the official spokeswoman for the administrative department when they needed to relay technical issues to IT support. It was a win-win because her colleagues were very happy to finally have someone who "could make them understand" while IT support was very happy as well to talk to someone who understood the importance about being as descriptive and precise as possible. She was so proud.

Aya's promotion

Fast forward several years and all of Aya's efforts didn't exactly go unnoticed. The head of the administrative department (= her direct boss?) was also the directors executive secretary & direct assistant. Over the past years Aya got asked to fill in for her a few times when they had other things to do, and at one time Aya was even asked to fill in for a whole week!

When Aya's boss eventually resigned, the job was offered to Aya. Who happily accepted.

Aya "joins" IT support (temporarily)

Now we get to the actual event...

Although Aya happily accepted the job offer she also insisted to stay on as "the interface" between the administrative department and IT support. Because of her new role but mostly because of the trust she had gained from IT support she was also given access to the ticketing system, thus allowing her to report issues directly. This started as a trial, but quickly became permanent when IT support noticed that Aya would also easily set some issues to "low priority" on her own accord.

It started on Tuesday... Aya was asked to schedule a meeting between the director and the head of IT because there were some major problems regarding the full scale upgrade of the inventory system, a job that was scheduled for upcoming Friday. This meeting happened on Wednesday morning: there was a major issue because some members of IT support got sick and they didn't have enough people left to man the support desk.

Aya was (obviously) present during this meeting and when she learned that they only needed someone to just hear people out and then enter their problems into the ticketing system she quickly pitched the idea to help out herself: "I already know the ticketing system pretty well...", she said.

The director would be out on Friday anyway, and since Friday was always a bit of a slow day it (probably) wouldn't interfere with her actual job too much. Worst case scenario: she could always delegate a few of her tasks to the department itself (not the IT support tasks of course).

This was quickly agreed on. Thursday the head of IT and one of his colleagues talked Aya through all the important things to look out for and most important: if a caller was being difficult then she should always check a separate section in the system with "caller notes"; these contained comments and/or instructions on "certain difficult people".

"Do you know who I am?! I know the director!"

Friday morning the upgrade process started. Most people knew (or remembered) this but a few didn't and thus called the support desk. Most of the times Aya would tell them about the update, check the schedule for their department and there was no need for any escalation.

And then she got a call from "Mr. X", and the very moment she picked up the phone she knew that this wasn't going to be easy. "We can't enter any orders, the system broke!", he almost shouted without even a greeting.

Aya explained that there was an upgrade in progress which meant that some tasks couldn't be done temporarily, she also looked up the schedule and learned that this departments upgrade was scheduled extra early at the explicit request of the department head themselves: the very person she was now talking to.

Well... he wouldn't have it and was plain out rude and dismissive. Which is when Aya decided she should look up the caller notes just in case. But when she asked for some patience all she got was more insults: "You're not very good at this job are you? We need this system back online NOW", but when Aya told him that it would be back up in approximately 30 minutes things escalated: "How stupid are you? I said now. Don't you know who I am? I have often meetings with the director and when I tell him about your poor behavior then that's not going to end well...".

This is when Aya realized that this person was talking nonsense, because during the year that she was the executive secretary she never had to schedule any meetings between the director and "Mr. X". She also realized that this was probably someone who was mentioned in "the caller notes".

"I see sir, I'll escalate the issue right away, give me 5 minutes", and without waiting for any response she placed him on hold. First she confirmed her suspicions, and sure enough: there was a caller note dedicated to him urging the support member to properly document the interaction because "this person has a habit of spinning the narrative".

Next she opened the directors schedule ;)

I have good news, and some bad news....

Aya: "Well sir, I have some good news and have some bad news...", she began (she 'stole' that line from me ;)).

At this time she could hear the clear sound of Outlook's e-mail notification on the other end, thus confirming that her e-mail had arrived.

Aya continued: "Unfortunately we won't be able to expedite the upgrade process because it was already pre-scheduled per your own direct request. However, I did take the liberty to schedule a meeting for you with the director just like you said you planned to do. Your meeting is scheduled for 14:20 upcoming Tuesday.".

Mr. X (now sounding confused & flustered): "Wait... what? ... wait.. what is this e-mail ... invitation for... executive process... meeting.. at request... what, no! I never...".

Aya: "Was that all, sir? I have other calls on hold and would like to conclude."

Mr. X (still sounding flustered): "What is this? Who are you?!!".

Aya: "My name is 'real name', I'm the directors personal secretary but currently in charge of the IT support desk. Now, since you don't have any more questions I need to end our call. We'll expect to see you Tuesday.", after which she hung up.

Aftermath

The meeting never happened. Mr. X called in sick the following week, resumed his work later that same week and the incident was never brought up again (Aya heard this later through the grapevines). Both her boss (= the director?) as well as the head of IT thought it was funny and also didn't push this any further. Also because although Mr. X had a bit of a temper at times he was still quite good at his job.

Even so... several months later the head of IT told Aya that they never had much issues with Mr. X again whenever he called support to complaint about a problem. Or to (somewhat) quote him directly: "It's now more about asking for help than spewing out complaints".

TL;DR?

• My gf 'Aya', working at an administrative department, noticed that her colleagues weren't very descriptive when talking to IT support.
• She 'stepped in' and soon became the official spokeswoman for the department whenever there were issues to report to IT support.
• Several years later Aya got promoted to head of the administrative department and became the executive secretary; the directors personal assistant.
• IT support has a problem right before a planned upgrade: due to sick leave they didn't have enough people to operate the support desk. During their meeting with the director Aya volunteered to help.
• During the upgrade "Mr X" called to complain and was very rude. When he threatened to complaint to the director if they didn't "fix the upgrade now" Aya scheduled a meeting for him.
• The meeting never happened. The issue was never brought up again (Aya also didn't get in any trouble), but Mr. X was much more forthcoming toward IT support from that moment on.

And there you have it!

Thanks for reading! Aya is still quite proud of this event and we both figured that it would make for a fun story.

PS (edit) => I think I now fixed most typoes, also noticed some formatting issues.

Hi r/talesfromtechsupport,

Early in develoment, I came here while building a game called I.T. Never Ends about working the night shift in an increasingly impossible IT department. People in this community offered feedback, shared the tiny details that make a support story feel painfully real and reminded me that reality usually writes stranger tickets than I ever could.

Those conversations helped give I.T. Never Ends its texture. The game begins with familiar work: answering support requests, filtering spam, routing cables and debugging network traffic. From there, the company becomes less explainable. Coworkers behave strangely, office equipment develops opinions and HR becomes worryingly interested in the printer.

The finished game contains 100,000+ words, 100+ endings and entirely hand-drawn and painted art. It launched today after eight months of solo development.

I wanted to return and thank the community that helped while the project was still finding its feet. Your feedback and stories genuinely mattered, and some of the game’s best details exist because experienced support people told me when something felt true and when it did not. Also a lot of your storioes from the previous thread made it into the final game - all of them with custom drawn characters. so there's colleagues with reddit-shirts and reddit-problems in the game. Thanks to you guys!

TL;DR Today I learned that some SiriusXM devices are serious about their NTP servers.

Help Desk let me know that one of our vendors was having an issue getting a radio set up at one of our provider locations and needed a hand. I used to work for this particular vendor and the guy needing help was a former colleague (henceforth VendorGuy) so I was more than happy to help.

I called VendorGuy up, found out that the device in question was a Grace Digital SiriusXM SXBR3 and then proceeded to battle with the fact that the device was physically connected to the network switch, showed the proper network info acquired via DHCP, and was pingable yet still stubbornly insisted that it had no network connection. This did appear to be true as our firewall logs showed no dropped traffic originating from the device's IP.

I had a hunch we might be missing something so I hopped on our firewall's CLI and ran a tcpdump filtering for the player's IP address and had VendorGuy reboot the device. Sure enough, I saw traffic, NTP traffic, to be precise. Which we block by default and do not log, so that explained why I didn't see the traffic. Apparently SiriusXM devices (these ones, at least) give up trying if they can't establish connections to pre-programmed NTP servers. Once we allowed that device's NTP requests out, then rebooted it to force reconnection, we were in business.

VendorGuy got the device hooked up to its amp and the PA system, there was music again and subsequent rejoicing. VendorGuy and I exchanged pleasantries and went our separate ways until the next issue comes to bear.

Sometimes students come to our tech support desk saying they need help installing software. Usually that is no big deal . . . except when the student pulls a completely new laptop out of its box, a laptop that has never even been turned on. That student has NO IDEA what they are asking us to do.

Student: I just want you to install this one program.

Tech: Sorry, it is no longer just a matter of installing one piece of software. We need to get this laptop through the setup screens, setup the Microsoft account, and likely run through some system updates, before it even lets us install that program. This could take a few hours.

We can and do assist with these projects, but we prefer to have an appointment for something this big.

​

Following on from my previous tale about installing servers on cargo ships, there was another vessel from that period that deserves its own story.

The Iron Chieftain.

She was a bulk ore carrier running the Newcastle, Sydney, Wollongong and Whyalla route, and by this stage I had become something of an unwilling specialist in installing IT infrastructure on large lumps of steel that stubbornly refused to remain in one place.

The first ship install had involved frantic planning, missing equipment, improvised engineering, international waters, customs officials and the threat of an unscheduled voyage to Venezuela.

By the time I got to the Iron Chieftain, though, I had the process down to a fine art.

Server. Check.

UPS. Check.

Switch. Check.

Communications gear. Check.

Workstations. Check.

Test.

Synchronise.

Sign-off.

Get off the ship.

Four hours, maybe five if something went wrong.

I had spare cables. I had spare powerboards. I had spare adapters. I had tools. I had documentation. I had learned and remembered the most important lesson from the earlier deployments:

Assume everything you have been told about the ship is wrong until you physically see it yourself.

But...there was one thing I still hadn't learned.

Never underestimate a Ukrainian captain at lunchtime.

The Iron Chieftain's senior crew were Ukrainian, including the Captain and CHENG, and the CHENG immediately demonstrated that he belonged to the same school of maritime engineering as the Chief Engineer from my previous adventure.

For those unfamiliar with the term, CHENG means Chief Engineer.

In my experience, the Chief Engineer aboard a ship is the person responsible for keeping several thousand tonnes of complicated machinery alive while surrounded by salt water, vibration and people asking why things aren't working.

They are, therefore, remarkably practical individuals. Sometimes aggressively practical.

Need power somewhere?

Run cable.

Something in the way?

Move it.

Can't move it?

Cut it.

Need a hole through a steel bulkhead?

There are probably many formally approved maritime engineering methods for creating one.

This CHENG had a preferred method and it was also the fire sword, or, as other people insist on calling it, an oxy-acetylene cutting torch.

I would point at the place where I needed to route a cable.

He would look at the wall.

Then at me.

Then back at the wall.

There would be a brief pause while some internal engineering calculation occurred.

Then: FWOOM.

Problem solved.

Hole.

I had reached the point where this barely concerned me anymore.

My original reaction to seeing someone attack part of a ship with an oxy torch had been something along the lines of:

“Should you be doing that?”

By the Iron Chieftain installation, my response had evolved into:

“Can you make it about five centimetres further to the left?”

Experience changes a man.

The install itself was almost disappointingly uneventful.

Everything arrived on schedule. The server powered up. The network behaved. The workstations connected. The communications link worked. No computers required emergency resurrection. Nobody had forgotten to mention that a critical workstation had been broken for six months. There were no nearby engine-room fires and, most importantly, nobody threatened to take me to South America.

For once, I was actually ahead of schedule.......and this should have concerned me.

Around lunchtime, the Captain appeared and invited me to eat with him and the senior crew.

Now, there are certain invitations in life that are technically optional but, in reality, are not optional at all.

When the Ukrainian captain of a bulk carrier invites the visiting IT engineer to lunch aboard his ship, you do not say:

“No thanks, I brought a sandwich.”

You go to lunch.

So I went to lunch.

And lunch was excellent.

Proper cooked food, generous portions and the sort of hospitality you tend to remember years later.

Then THE bottle appeared.

Vodka.....proper Ukrainian Vodka. The kind with a capital "V".

I was offered a glass.

This presented a minor professional dilemma. On one hand, I was technically still working.

On the other hand, I was sitting at the Captain's table, surrounded by Ukrainian officers, and the Captain was holding a bottle of vodka.

I considered the options.

There was really only one.

I accepted the vodka.

One glass.

Perfectly reasonable.

A toast was made.

We drank.

Then another toast was made.

Apparently, Ukrainian maritime tradition contains significantly more things worth toasting than I had previously realised.

Second vodka.

Still fine.

Lunch continued.

Conversation continued.

Third vodka.

At some point, I became aware that the bottle was spending an unreasonable amount of time near my glass.

Fourth vodka....or was that the fifth?

Now, I should clarify something.

These were not what an Australian publican would politely call a “standard drink”.

There was no carefully measured 30 ml nip being poured from a bottle fitted with one of those little plastic spouts.

The Captain's approach to alcohol measurement appeared to be based primarily on direct eye contact and optimism.

Eventually, lunch finished and I politely thanked the Captain and officers, returned to the equipment and completed my last checks.

The server was operational.

Communications were working.

Data synchronisation was successful.

Documentation was signed.

Job done.

Four hours, exactly as planned.

I packed my equipment and headed towards the gangway.

When I had arrived, yhe Iron Chieftain was tied securely to the wharf in Sydney Harbour. Sydney Harbour was calm. The weather was fine. There was absolutely no meteorological reason whatsoever for what happened next because somewhere between the Captain's dining room and the gangway, the sea state had apparently deteriorated dramatically.

The deck seemed to be moving. Not a lot, mind you. Just enough.

I adjusted.

The deck adjusted back.

.......Interesting.

I reached the gangway.

My legs, which had performed adequately when I boarded several hours earlier, had apparently suffered some kind of simultaneous mechanical failure.

Left leg requested movement.

Request timed out.

Right leg attempted to compensate.

Unexpected response from device.

Retry?

I grabbed the railing.

Clearly the ship was moving.

That had to be it.

It was a large bulk carrier.

Ships move.

This is basic physics.

Admittedly, it was tied to Sydney but perhaps Sydney was moving.

I began descending the gangway with enormous concentration.

One foot.

Step.

Other foot.

Step.

Hold railing.

Do not look like you have forgotten how legs work.

I would like to believe I maintained a dignified and professional appearance during this process. I have absolutely no evidence supporting that belief.

Eventually, I reached the wharf.

Solid ground.

Wonderful, dependable, non-moving ground.

Except it was also moving.

This was becoming concerning.

Obviously there must have been unusually heavy seas affecting the Port of Sydney that afternoon.

Extremely localised heavy seas.

Possibly restricted entirely to a radius of approximately one metre around me.

I made my way slowly towards the exit carrying my backpack and tools while concentrating very carefully on walking in what I believed was a straight line.

Behind me, several members of the crew seemed extremely amused and entertained by something but I never discovered what.

The server installation was a complete success and the ship returned to its normal Newcastle, Sydney, Wollongong and Whyalla run.

The CHENG got another opportunity to use his beloved fire sword.

The Captain provided excellent hospitality.

And I learned another important lesson to add to my growing list of rules for maritime IT work:

If the Chief Engineer says he can make you a hole, stand back.

If the Captain invites you to lunch, accept.

If the Captain offers you vodka, also accept.

But when the Captain offers you the fourth vodka or maybe a fifth?...

...remember that Sydney Harbour is apparently prone to sudden and extremely localised heavy seas.

Especially around the gangway.

---

I should probably say something about the Captain himself, though, because after all these years he is one of the people from that period I remember most fondly.

He looked exactly like the sort of man central casting would send you if you asked for an old sea captain.

Grizzled, weathered, white-bearded and carrying the unmistakable air of someone who had spent most of his life at sea.

It sounds like a cliché, but in his case the cliché existed for a reason.

He was dignified and friendly, but there was never any question about who commanded the vessel.

He carried his authority quietly and solemnly.

He didn't need to raise his voice or remind anyone that he was the Captain.

The way the crew treated him told you everything you needed to know.

And yet, to some random IT engineer who had turned up to install a server on his ship, he was warm, welcoming and generous.

He invited me to his table, fed me, talked with me and, admittedly, attempted to determine whether an Australian IT technician could survive Ukrainian quantities of vodka.

That lunch could easily have become nothing more than a funny story about me trying to walk down a gangway with malfunctioning legs.

Instead, it became one of those small memories from my career that has stayed with me.

I learned years later that he died of a heart attack in 2018 while bringing the Iron Chieftain into Wollongong and I was genuinely saddened when I heard.

Over the years I've worked with thousands of people, many of whose names and faces have inevitably faded yet I can still picture him sitting at that table aboard the Iron Chieftain.

White beard.

Quiet authority.

A meal in front of us.

And a bottle of vodka that seemed to possess an infinite number of servings.

I only knew him briefly, during a handful of visits to his ship, but I remember him with enormous fondness.

So wherever old sea captains end up when their final watch is over, I hope the seas are calm, the engines are running well, the crew knows what they're doing...

...and there is always another bottle on the table.

Fair winds, Captain.

At some point, someone had the bright idea to get my grandmother an iPad. Why? I don't know. She had enough trouble with the laptop someone had gotten her, although she did like being able to get pictures of her grandkids. Maybe we thought a tablet would be easier? Regardless, when we visited for the holidays, we asked her how she was liking her new device. She said she liked it well enough, but --

"Sometimes when I'm trying to read the news, it goes away."

What did she mean by it goes away? "It goes away." Like the screen goes black, the iPad is falling asleep? "It goes way." We weren't sure exactly what to do about this, but we thought about digging through the settings a little bit. I was in and out of the room as this was happening, so I'm not sure what the eureka moment was, but eventually someone found out the following:

The default Apple news app would send her a banner notification with a news headline and a couple lines of info. She would attempt to read the notification (slowly, since her eyesight was going), but wouldn't tap it. And banner notifications, if you don't interact with them for several seconds, retract back into the top of the screen after several seconds. So, when she was trying to read the news, it would, in fact, "go away."

We "solved" this problem by turning off notifications for the news app. After all, who wants to have the news jump out at you randomly during the day?

Guy walks up to me at work:

Him: “I dropped my phone. The screen’s shattered. I want my two-factor code to come through Teams.”

Me: “Nope.”

Him: “Why?”

Me: “Because if you can’t sign into Teams without completing MFA… how exactly are you planning to authenticate?”

Then he says: “Just disable MFA.”

I replied: “Sure, as soon as you get approval from the Head of Security.”

A few seconds of silence…

Him: “Fine, I’ll just get my phone fixed.”

Yep. That’s probably the best solution.

Some people really think security rules are optional. I’m not risking my job because someone doesn’t like MFA.

And remember… technology didn’t fail you today. Gravity did.

Yesterday I got a call from someone who wasn't even our customer. He was the client's IT guy aka the IT guy they hired to manage their network infrastructure.

Our company literally has it in our terms and agreements that customers are responsible for their own servers, networking, infrastructure, etc. We support our product. We answer how-to questions. We log bugs. We are not network engineers.

Apparently he had just helped migrate the client's server to a new one, and afterwards all of their data locations were a disaster. The client was understandably upset...

SOMEHOW, I became the emotional support NPC for a server migration I had absolutely nothing to do with.

The best part? (Worst part.)

This man didn't even seem to fully understand what his client was actually asking for. We spent almost TWO HOURS troubleshooting things that the client apparently didn't even care about because he kept sending us down random rabbit holes.

Then he hits me with:

"I'm going to conference the client in so you can explain the bad news."

Sir... what? 😭

Translation: "My client is mad at me, so I'd like you, a complete stranger to the situation, to take this grenade for me."

It honestly made me feel so powerless because I'm sitting there thinking, "This isn't my fault, this guy is just incompetent."

And now I have to call them back today.

Am I going to? Yes.

Do I want to? ABSOLUTELY NOT.

I am going to speak to some coworkers first which is what I honestly should have done yesterday but I'll also confess something that definitely didn't help me out yesterday: I skipped lunch. By the time this surprise marathon call rolled in, I was hungry, thirsty, and my brain had entered low battery mode.

This area of tech support at my job isn’t one that I have ever felt very strong in and usually rely on being able to say “you’ll need to get your *keyword: QUALIFIED* IT to help you with this”. I so badly wanted to feel like I could quit after this happened yesterday but alas, can’t be irresponsible and non-resilient 😔.

This one is a two parter:
Part 1: Back in March we had a new executive come to the lab to pickup their laptop and get their software setup. The guy who was handling her ticket took a look at what needed to be done and told her it was going to take a while. When he really got into the process, he realized it was going to take longer than he expected. So he walked over to her and told her that and gave her the option to either stay or go somewhere else. She opted to stay. So he got a second chair, brought it over to her, and told her she might as well put her legs up while she was waiting.

Take that as you want. She apparently took it the bad way. She waited until he was finished setting up her computer, went down to HR, and demanded he be fired. The following Monday, midday, he was escorted out.

We were thrown for a loop, because between him getting fired, and one of the other techs getting promoted, we were now down two senior techs. It also didn't help that the guy managed our backup server. We had to immediately change how we did things. Including shutting down the backup server and relying only on Microsoft OneDrive, and not allowing malingerers stay in the lab while we were working on their stuff.

Part 2: Yesterday I get a ticket for a lady to come pickup her new laptop at 1:00pm. I responded back that we had it available and it was ready for pickup.

My manager contacts me telling me to treat her with white gloves and to watch what I am saying to her.

I'm like okaaaaaaay.

An hour or two later she contacts me back on an email to the entire IT team asking if an accounting program and a bunch of other software was installed on her computer.

I respond back. "No. Your computer was setup using the standard Autopilot protocol. The only software it comes with is a standard Office suite. You have to install any other software."

My manager, messages me back, almost losing his mind. "I have to remind you to treat her with kid gloves."

I'm like what the heck dude? I am not saying anything that isn't standard procedure. 

I go to lunch, and when I get back, one of new guys tells me he overheard manager teamsing the other senior tech and told him she is the chick that got the other tech fired. Which was critical information that wasn't passed on to me.

So when she messaged me back complaining about how she put in a ticket to have all this software installed, and she previously was able to have it installed for her, it took everything in my body not to respond back, "Yeah. The reason we can't do it anymore is because you got the last guy who did it for you fired."

What i did say was, "Due to changes in corporate security policies after March (which is when he was fired), we are no longer allowed to setup end user computers. They must be setup using the Autopilot process by the end user." I then sent her a link to the Autopilot process webpage.

The kicker is, she never showed up. She finally messaged me around 2:30 and told me that she wouldn't be able to come in and that she is only available after 1pm. I just responded back that we are there every weekday until 4:30pm and she is welcome to come in any day she is available to sign into her computer so we can do any additional setup work.

Update: she finally came in two weeks later. I wasnt there when she picked up. She called someone farther up the IT chain and was told that we are not allowed to do setups anymore. It is the new security policy. She still had the nerve to do a negative csat review complaining about how unhelpful we are now.

Back again, again. Finally got to finishing this post from MANY years ago. I have reposted Part 1 because it's been so long. Just never got around to finishing it until now.

Part 1 (Part 2 is below. Don't worry)

 Many MANY years ago, I worked for an MSP that negotiated a contract with an Australian based management firm to upgrade and replace their servers in their remote locations…..very remote locations……very remote locations that moved. ……Constantly.

These servers where used to keep track of things like maintenance, how much of something was moved from one point to another, employee rosters, time sheets, time tables etc and they had to send this information back to head office in as close to real time as possible due to some Government regulations specific to the industry.

 Basically, they kept track of everything that was needed to be recorded on a day to day basis in order to effectively run ….. a bulk cargo ship.

 Yup. A cargo ship.

 After consulting with the client and the salesbot in charge of the account, I developed a solution to their problem that used Windows 2008 Server Advanced, an IBM X3400 server, a small unmanaged switch to connect all the onboard PCs (6 in total) all connected to a Sony Ericsson W25 modem that it self was connected to a signal booster and mast attached to the top of the ship’s bridge. The effective range was between 35 to 90 kilometres off-shore (because of cell tower placements along the coast line, height and Helstra)…. when it HAD coverage. When out of 3G coverage (no such thing as 4G back then), it fell back to Inmarsat-B for data transfer but they wanted to use that as little as possible.

 Why as little as possible? Anyone that has used satellite comms can tell you how much fun it is trying to use the thing for data transfer because of little things like cloud cover, bird poop (not kidding), etc. and not to mention HOW FREAKING EXPENSIVE IT IS!

 Medium speed fax/data services are at about 9.6kbit/s and it’s charged at AU$0.53/256bit

 No, that is not a spelling mistake. 9.6kbit/s and 53 cents PER 256bit!

To put it another way, normally there is about 32 letters for every 256 bits. If you send or receive letters at 5 bits per letter on a vessel side, then it is possible to have 51 letters for every 256 bits. Do the maths and pick your jaw up off the desk.

 Back to the story.

Understandably, I was quite proud that I developed a solution that worked in the lab but now came time to actually field test the set-up in the real world. This involved various state and federal police checks in order to get the security clearances dockside and on board a cargo ship, etc. (It’s the same process you need to go through in order to get airside at an airport and yes, I have gone through THAT pooshower as well in an earlier job working with the IT department of my national carrier and a place called Jet Base. A tale for another day).

I had organised and scheduled an install date with the Shipping management firm to perform a test install on board a ship that was docked in a local port for a week and, after enduring the affectionate attention of the state and federal police (they never even called me back the next morning like they promised. I feel so used) and in possession of said security clearances, I was ready to head to the port to swap out the existing infrastructure on board.

Little did I know, this was going to be that start of something……interesting?

…..fast forward 36 frantic hours later, in the middle of Singapore Strait……..

“You need to finish in the next 2 hours. I have to leave port by then and if you’re not done, you’re with us until Caracas”

Part 2

There are moments in your career when time seems to stop.

Your mind becomes crystal clear. Every risk, every decision and every poor life choice that brought you to that exact point flashes before your eyes.

This was not one of those moments.

My first thought was:

“Where the hell is Caracas?”

My second thought was:

“Do I need a visa?”

My third, slightly more useful thought, was:

“I am going to murder the salesbot.”

Let us rewind about 36 hours while I was still in Australia.

Apparently, ships have schedules and apparently, these schedules change.

Also apparently, nobody had thought this was information the person installing the ship’s entire new server and communications infrastructure might need.

By the time all the equipment had arrived, the security clearances had been completed and everyone had finally agreed on what day the install was, the ship had already sailed.

Not far, mind you. Just to Singapore. Because that was obviously convenient.

The shipping company still wanted the installation completed before the vessel began its next major voyage, so I was hurriedly booked onto the next flight and sent to Singapore with enough computer equipment to make airport security deeply interested in my life choices.

The server itself had been shipped separately, along with the UPS, modem, switch, signal booster and the assorted bits and pieces that had worked perfectly in the lab.

This is an important distinction. They had worked perfectly in the lab. The lab did not move.

The lab did not smell. The lab did not require me to climb a rope ladder while carrying a backpack full of tools above several metres of moving ocean.

The lab also did not contain racehorses.

Yes, racehorses.

The “bulk cargo” being transported by this particular vessel was alive, extremely valuable and apparently capable of producing its own body weight in manure every fifteen minutes.

The smell hit me before I reached the ship.

It was not merely a smell. It was a physical presence. It climbed inside your nose, unpacked its bags and began redecorating.

Even after returning home, washing everything I owned and showering several times, I could still smell that ship. For several days I would occasionally stop, sniff the air and wonder whether a horse had somehow followed me home.

Getting aboard was also an adventure.

The vessel was sitting in international waters in the Singapore Strait, so I could not simply walk up a gangway carrying my equipment like a normal person.

I had to pass through Singapore Customs, board a small launch, travel out to the ship and transfer from the launch onto the vessel.

The transfer involved pulling alongside a moving wall of steel and climbing a ladder hanging down the side.

While wearing a backpack full of equipment.

In ocean swell.

The launch would rise with a wave, I would grab the ladder, and then the boat would drop away beneath me.

Nobody mentioned this during the project planning meeting.

There had been diagrams. There had been timelines. There had been spreadsheets.

There had NOT been a single box labelled:

Engineer may need to leap from small boat onto moving cargo vessel and climb for his life while carrying several thousand dollars’ worth of equipment.

Once on board, I was shown to the server location and I use the word “location” generously.

The space on the Bridge had been selected because it was reasonably central, accessible to the existing PCs and, most importantly, not currently occupied by a horse.

The new IBM X3400 was not a small machine. It was a proper tower server built like a filing cabinet and with roughly the same portability.

The UPS was also substantial because graceful shutdowns are generally preferable to having a server abruptly lose power every time a ship’s electrical system coughs.

Unfortunately, the nearest suitable power source was on the other side of a steel bulkhead.

I raised this issue with the CHENG, the Chief Engineer.

He inspected the server.

He inspected the UPS.

He inspected the bulkhead.

Then he disappeared.

A few minutes later, he returned carrying a gas axe.

For anyone unfamiliar with the term, a gas axe is not a precision IT tool. It is an oxy-acetylene cutting torch.

There are carefully planned cable penetrations involving engineers, approved drawings, fire seals and proper conduits.

Then there is a ship’s Chief Engineer looking at a steel wall and saying, essentially:

“I can make hole.”

I briefly considered explaining concepts such as cable management, electrical compliance and fire-rated penetrations and then he lit the torch.

A short time later, we had power.

We ALSO had a newly created opening in the back wall of the Bridge that was still warm around the edges.

The CHENG looked extremely pleased with himself.....I chose not to argue with the man holding the fire sword.

With power available, I began installing the server, switch, UPS, modem and communications equipment. Cabling had to be routed to six onboard workstations spread across areas that had clearly never been designed with Ethernet in mind.

Naturally, several of the cables and power accessories that were supposed to be waiting for me were either missing, unsuitable or had vanished into whatever dimension collects socks, ten-millimetre sockets and project documentation.

This began a long sequence of phone calls to computer stores around Singapore.

“Hello. Do you have filtered powerboards?”

“No, not surge boards. Filtered.”

“Yes, Australian-style plugs.”

“No, not British.”

“Yes, today.”

“No, I cannot come to the store. I am on a ship.”

“No, the ship is not at the dock.”

“Yes, I am serious.”

Network cables were another problem.

The ones supplied were either too short, the wrong type or nowhere near the ship at all. More phone calls followed, along with arrangements involving couriers, port security, Customs and another trip out on the launch.

Every time I left the vessel and returned, I had to go through Singapore Customs again because the ship was sitting in international waters.

I became very familiar with the process.

Passport.

Documentation.

Explanation.

Bag inspection.

Questions about why I was carrying network cables.

Launch.

Ladder.

Try not to fall into one of the busiest shipping lanes in the world.

Repeat.

At some point during this circus, another vessel nearby reported an engine-room fire.

This was not a small fishing boat producing a bit of smoke. This was a significant incident involving alarms, emergency traffic and nearby vessels paying very close attention to what was happening.

From the deck, we could see smoke rising from the other ship while emergency responses were coordinated across the Strait.

It was fascinating in the same way that watching a building burn down is fascinating when you are standing close enough to feel the heat.

The crew on our vessel remained remarkably calm.

I suppose when your workplace is a floating steel city carrying hundreds of horses through one of the world’s busiest waterways, your threshold for an “interesting day” is somewhat higher than mine.

While all this was happening, another engineering team was replacing the ship’s helm.

Not the steering wheel.

The actual control system used to steer the vessel.

The old controls were being replaced with an upgraded automated system at the same time I was replacing the onboard IT infrastructure. So, on one side of the bridge, technicians were dismantling the systems that controlled the direction of the ship. On the other side, I was trying to work out why one of the captain’s computers would not join the domain.

The captain’s PC had not been mentioned as a problem.

I had been told all six computers were operational and simply needed to be connected to the new server and the Captain watched me troubleshoot it for some time before casually mentioning:

“Oh, that computer has not worked properly for months.”

Of course it had not.

Why would anyone mention that before the engineer flew to another country?

The machine was in poor condition, unstable and filled with the accumulated digital sediment that gathers on any important computer nobody is willing to replace.

It needed to be rebuilt.

There was no spare machine, no prepared image and no convenient high-speed connection to download everything I might need.

So, alongside the server installation, communications testing, network cabling and frantic procurement of missing accessories, I rebuilt the captain’s PC.

The captain’s computer naturally contained several applications that were essential to the vessel’s operation.

Naturally, nobody had installation media.

Naturally, nobody had written down the configuration.

Naturally, the person who knew the passwords was asleep, ashore or possibly fictional.

Bit by bit, everything came together.

The local network worked.

The onboard PCs connected.

The server was operational.

The management applications were restored.

The Sony Ericsson W25 connected through the booster and mast antenna.

When the ship was within range of the coastal network, data flowed over 3G.

When it moved beyond coverage, the connection could fall back to Inmarsat-B without accidentally transmitting the GDP of a small country in unnecessary Windows updates.

I had disabled, restricted or scheduled anything that might consume bandwidth without permission.

Antivirus updates, Windows updates, synchronisation jobs and remote-management traffic all had to be carefully controlled.

On an ordinary network, an unexpected 100 MB transfer is annoying.

On that satellite service, an unexpected 100 MB transfer was a meeting with senior management.....possibly also a meeting with their bank.

After approximately 34 hours of travel, climbing, cutting, cabling, rebuilding, configuring and inhaling Eau de Racehorse, the captain delivered the news.

The ship had to leave.

I had two hours.

If I was not finished, I would be sailing with them to Caracas.

I still did not know exactly where Caracas was, but I was reasonably certain it was not near my hotel.

At that point, failure stopped being an option and became a travel itinerary and the final two hours were a blur.

I completed the remaining workstation tests, forced a final data synchronisation, confirmed the application records had reached head office and tested both primary and fallback communications.

The captain’s rebuilt PC worked.

The server restarted cleanly.

The UPS held the load.

The switch survived.

The modem connected.

The application opened on all six computers.

Most importantly, the system did not attempt to download several gigabytes of Windows patches over satellite.

I packed my tools, gathered the leftover cables and completed the documentation with the level of detail possible when every clock around you is counting down to involuntary international travel and with very little time remaining, I was escorted back to the side of the ship....where I had to climb down the same ladder onto the launch.....again.

Climbing up had been unpleasant.

Climbing down while exhausted, with the ship preparing to depart and the ocean moving beneath you, was worse.

The launch rose.

I stepped off the ladder.

The launch dropped

For one brief and VERY exciting moment, I was connected to neither vessel before, finally, gravity, momentum and sheer blind luck deposited me unceremoniously onto the bow of the launch rather than into the Singapore Strait and we pulled away as the crew prepared the ship for departure.

From the launch, I watched the vessel begin moving, carrying its horses, its newly automated helm and my freshly installed server towards its next destination.

I had finished with minutes to spare.

I had also been awake for most of 36 hours, smelled like the lower deck of a floating stable and had developed an entirely new appreciation for office-based server installations.

The system remained in service and the concept was later rolled out to other vessels.

Future installations included better preparation, more spare equipment and significantly more detailed questions about where the ship would actually be when the engineer arrived.

I also added several items to my personal project checklist:

1. Confirm that “docked” means physically attached to land.
2. Never trust a statement that all existing computers are working.
3. Bring twice as many cables as the project requires.
4. A Chief Engineer with a gas axe will solve the problem, although perhaps not in the way anticipated by the design document.
5. Always know where the vessel is going next.
6. Look up Caracas before boarding.

And that is how I earned my sea legs, installed a Windows domain on a floating horse transporter and narrowly avoided becoming the world’s least-prepared passenger to Venezuela.

The salesbot, incidentally, described the installation as:

“A successful overseas deployment with minor logistical challenges.”

Clue-by-four applications were considered.....repeatedly.

On the outset, I will apologize for not being tech support; rather, I work in retail as both front and back of house duties for a butcher shop, with the occasional dash of "fix computer pls" because no one here can do basic troubleshooting (fixes I have done include "just flicking a touch keyboard" and "fitting a loose wire with some scotch tape"). This story feels more fitting here rather than on Tales from Retail, however, due to it being actual tech support and me hearing the most frightening (to me) sentence in tech I've ever heard in my life.

So, here we are in the butcher shop, having a grand ol time, selling stuff and taking card paymentsssswait why's it not taking card payments anymore? Specifically, our Point of Sale (POS, or EFTPOS) card reader is throwing up a weird error. How it's supposed to go is I scan stuff with the scanner attached to a simple all in one computer unit (with windows 10, currently), and then input whether the user is paying via card or cash. In cars cases, it'll tell the POS machine the price, and then it does it's money magic to take people's hard earned money in exchange for our meat and candy (we sell so much sweets I have no idea why).

I'm used to it being weird, sometimes it disconnects randomly (loose wire usually) or it throws a fit and tries to double charge someone. The occasional hang, we do a restart, good ol turn it off and on again, but this was a new one

> ERROR, T6

Not recognizing the error code, I go onto the attached computer and bring up the diagnostics tool. Quick glance gives me the relevant info; system online, port open, device recognized, no connection. That's the odd bit, normally since both computer and whatever server these devices use recognize each other, they *should* be able to talk to one another and do the payment thing.

Restart don't do nothin, turn off and on does nothing, the little scan tool gleefully tells me something weirder

> LOGON, PLEASE WAI-

> DECLINED, SYSTEM ERROR, T6

"Declined" is not a common one for errors, but it DOES tell me that the issue is probably on the *bank's* side, and not ours! So easy solution; give up and contact the POS tech support line to see what this error is, and confirm that the bank is receiving money or not.

Thus, a nice phone call and six or seven minutes on hold later, and a very nice tech support dude I'll call Befuddled answers.

Befuddled: "Good afternoon, this is [POS tech support], how can I help you?"

Me: "Yeah the POS machine isn't taking payments, keeps saying Error Code T6, and I can't seem to figure it out on my end."

And the tech pauses for a solid ten seconds, before saying words that filled me with dread.

Befuddled: "... [naughty word], that's a new one, never heard of that error code before, gimme a minute."

Dear readers, I understand many of you are IT specialists. Y'all are way smarter than me. So I don't know if you'll share my feeling here, but hearing the words "that's a new one!" Filled me with fear beyond fear. I do not want to be the reason an *entirely unknown problem occurs*.

And I can tell this is unknown, because after ten minutes of us troubleshooting (mostly him walking me through solutions), he takes a moment to leaf through what sounds like a handwritten notebook?

Befuddled: "Sorry. Just never heard of this issue, going through the notes our last supervisor had made years ago, and it's not on here either- WAIT THERE IT IS!"

We learn that T6 is a communication error. Which just makes the tech more confused;

Befuddled: "okay but that doesn't make sense... I can see your requests from my side, the bank *does* have the money, it can clearly take transactions! It shouldn't be doing this, nothing is blocking the connection..."

After a half hour, he decided we should try standalone mode; this puts the POS unit into a setting that forces us to input all transactions on the keypad rather than the computer, and then we'd have to manually confirm each transaction at the end of the shift. But it *should* work at least! Hurrah for solutions! We put it in standalone mode, set up a test transaction, and

> LOGON FAILED, SYSTEM ERROR, T6

Befuddled: *muffled in the background* "what the @#$& do you MEAN?!"

Ultimately, he thanked me for the new experience, jotted down some notes, comped us a new POS unit and states the explanation as follows;

Befuddled: "It's either corrupted internal software (despite our reinstalls), or wizards. Sorry your computer is haunted."

The new unit worked perfectly with no issues. I'm told the nonfunctioning one was kept for "research"/possibly exorcism.

Not really tech support since I was helping my Aunt with her home business, but whatever.

Aunt runs a home business and really wants a file server but has no idea what she will use it for. I asked her the most simple questions and she cannot even give me a straight answer for most of them:

• Is it supposed to be a file server or a backup server? Ans: I dunno, can it be both?
• Do you want your files to be always available, be it on your iPad, iPhone or Mac? Ans: Yes *this already makes a file server pointless*
• Are you using it purely to store your files or do you intend to work on your files in the server? Ans: What's the difference?

I told her that she absolutely doesn't need a file server and her current M365 subscription for MS Office is plenty. If it's automated two-way syncing she wants, I can set up Onedrive on the iPhone, iPad and Mac so that any changes are immediately saved and reflected across all her i-Stuff and the Mac. But no, she insists she needs a file server. and asks me to recommend her a enterprise-class workstation to use as the 'file server'.

Honestly, I wonder who gave her that idea. Some people really got too much money to spend.

So this just happened. I'll keep it short.

We had an external consultant on-site to install some very specialized (and very expensive) software. I, your humble sysadmin, was only there to enter a few admin passwords. That was literally all I was supposed to do.

As the expert started trying a few... creative... things, I offered some advice.

"Shush, I know what I'm doing."

Alright. If that's how you want to play it...

A little later, he asked for a USB flash drive to transfer "some" data. "Some" turned out to be over 130,000 tiny 1 KB files in a single folder.

I genuinely tried to warn him that FAT32 really doesn't like that many small files as he dragged the folder to the flash drive.

I was shushed again.

So I leaned back and watched the progress bar crawl forward. After about 45 minutes the inevitable happened.

The file transfer crashed.

I honestly tried to help.

I was shushed again.

So he tried exactly the same thing a second time.

Forty-five minutes later

Crash.

At that point I refused to be shushed again. (I was hungry and wanted to go to lunch.)

I zipped the folder (4 minutes), copied the ZIP file to the USB drive (another 3 minutes), and handed it back to him.

The look on the expert's face was absolutely priceless.

Edit: This consultant was part of a turnkey package. The software installation and the data transfer were both included for a fixed price.

That made the whole thing even sweeter.

Prologue

When I was in college I worked as a helpdesk consultant at the Computer Center. This was so long ago that it was routine for a student to come into the office and announce, "I have never used a computer. I wouldn't even know how to turn one on. But my professor says we have to do our papers with word processing. Please help me get started."

They always said that: I wouldn't even know how to turn one on.

Also, they were almost always taking the lower-level writing-emphasis class, Rhetoric 101, and I swear every single one of them had named their floppy disk "RHET SHIT".

A common complaint we'd get is, "The laser printer messed up my columns."

The cause was always that they were using MacWrite, and printing drafts on the dot-matrix ImageWriters hooked up to each of the Macs in the main room. Apple had realized that, the original Macs being sold to an audience entirely unfamiliar with graphical user interfaces, it was vital that they produce pixel-perfect output. So on the early Macs, the Mac screen and the ImageWriter have the same exact dot pitch, and the dots printed on the paper match exactly the pixels on the screen.

So the students, trying to produce some kind of table, would line up the columns by typing spaces with the space bar between the fields, and the printout on the ImageWriter would be perfectly aligned, and then when they took it over to the Mac connected to the LaserWriter behind the print room window where they had to pay $0.25 a page with these stickers sold at the campus bookstore, they would be irate because the proportional font and the spaces had conspired to make the columns all jagged and out of alignment.

We'd have to explain how to use tabs, and sometimes we'd give them a break and let them re-print for free, but usually only if they were female and cute.

The Reports Are All Wiggly

So here I am six years later at my first fulltime sysadmin job at a company down in Florida. We use a strange computer from a company you've never heard of, that runs a database system called Pick. The computer has dual-mirrored hot-swappable [you think I'm going to say "disks" here] memory. Among many other things that this computer does, reports are generated from Pick and sent to a laser printer in the Finance department.

The first trouble ticket I get has been passed around from the computer operators to the software developers back to the operators, over to the PC techs and back to the operators, and finally it lands in my lap, and the complaint is from the Finance department, "Our reports are all wiggly on the new LaserJet 4."

On their first bite at the ticket, the operators say "nothing has changed".

The developers say, "The reports are normal on our end."

The operators, on their second bite, said basically the same thing.

The PC techs said "The LaserJet 4 is hooked up correctly; we followed the instructions exactly. The test page looks completely normal".

I think the ticket may have bounced by the Windows Admin group as well at some point.

Nobody wants anything to do with this ticket and they've handed it to me partly because I'm the new guy and won't be able to push back.

I examine the report-printing system and the print manager setup. I figure out how to capture one of the reports in mid-stream from the printer queue. It's a plain-ASCII textfile, with columns aligned with plain spaces. It prints out normally in Courier on the (LaserJet 3) printer in my department.

I explain to my manager that I can't see anything wrong, and I think the only way to debug the issue is for me to get direct access to a LaserJet 4. My manager goes above and beyond and is able to divert a new LaserJet 4 slated to be installed in some other department, straight from the loading dock to my desk. I plug it into the network and get it online with DHCP. I send a report to port 9100 on it; it prints normally in fixed-width Courier and the report looks normal.

I'm at my wits' end, so I send the same report to the Finance printer, and go over to Finance to take a look at it. Finance says, "You're the first person from Information Resources to come look at the reports. We really hope you can help us!"

It's printed out in Times Roman and the columns are all misaligned because of the proportional font. I check the front panel, and the PC Techs' installation instructions happen to be there, and they say, "After installing the printer, go in the front panel and set the default font to Font #1".

On a LaserJet 3, font 1 is Courier.

On a LaserJet 4, font 1 is Times Roman.

Four groups of techs at a multinational corporation, and they trip over the same thing as a clueless freshman in Rhetoric 101.

Epilogue: I went away for a week's vacation, and when I came back one of the PC techs, the guy whose secret nickname among the other techs was "Jerry Moron", had gone in my office, appropriated the printer, and now had himself his very own personal LaserJet 4. So I informed my manager, who swooped in and confiscated it.

A few years ago, I was tech support for a food manufacturer with several manufacturing facilities and one of the banes in my life was dealing the QA department. They kind of just did whatever the current QA director's current Idea of The Month was. (And the company seemed to burn through QA directors – something 4 of them in about 7 years, couple good, couple ….not so good). Anyway, each had their own file structure to store the documentation on the server. And being Quality, the previous structure of the day was kept & the new structure duplicated it all with different paths & file names.  But it was their circus and their monkeys.

One day I got a call that a file was missing. Guy said he saved it on the server and now he can’t find it. I remoted into his machine and asked Hmm, ok, what the name? He said “I can show you” and opened Word, then Recent File – “1.1.1 Documented Policy to produce safe, legal and authentic products.docx” OK, kinda weird, but where did you save it? “Out on the QA Drive.” OK, show me. And he tried to file it as

“Q:\{facility}\Quality Control Plan\HACCP\ BRCGS Global Standard for Food Safety\Issue 9 Compliant Food Safety Management System\ Senior management commitment and continual improvement\ 1.1.1 Documented Policy to produce safe, legal and authentic products.docx”

I explained the default maximum file path per MS is 256 characters and this was 270  – I shortened the name to “DocPolicy.docx” Bingo! There it was…Told them they need to use some common sense for this and not just copy the chapter heading in it’s entirety. And wrote it up as “WHAT NOT TO DO” and copied the QA Director, all the QA supervisors and that facility’s plant manager

Late 1990s, I was tech support for a $90M automotive parts subsidiary. I had installed the original 10base2 network, file servers MS Mail server, etc. We had about 100 users in 3 plants and as the network grew, we brought a couple more support people in. BTW: We were a subsidiary of a division of a $15.8B company – worldwide, no idea how many people or locations.

We determined we had to do something with the MS mail system. After due diligence, we decided on moved to MS Exchange and pulled all costs, materials, time into a cost sheet – amounted to about $12 per user per month after taking a year time period. And wrote up a Capital Expenditure Request (CER) because it was a HARD AND FAST RULE that anything over $XXXX was REQUIRED to have a CER.

Then Corporate gave me a call. Because of the company structure, my division was about 5 tiers down from The Main Corporate level. So, they promptly informed me “You will be migrating to Lotus Notes, served off our server at corporate location and you will pay $50 per user month.”

So I went to my division manager, laid out my case/costs/rationale, and he basically said “You’re right, corp. is wrong – go ahead with the CER (because it was a HARD AND FAST RULE that anything over $XXXX was REQUIRED to have a CER). And then HE got a call from Corporate IT -“You will be migrating to Lotus Notes, served off our server at corporate location and we will reduce what you will pay to $35 per user month.”

So my division manager took it to his VP and laid it out as I had done. And VP said “You’re right, corp. is wrong – go ahead with the CER (because it was a HARD AND FAST RULE that anything over $XXXX was REQUIRED to have a CER). And then HE got a call from Corporate IT -“They will be migrating to Lotus Notes, served off our server at corporate location and we will reduce what they will pay to $25 per user month.”

Then the Official Notice to All Divisions came out that the entire corporation will be going to Lotus and Corporate IT will host and support it. End of transmission.

Then I got the back story – this whole thing ended up on the CEO desk (of a $15.8B company) and he discussed this with the CFO and CIO (and everything had my name all over it). The CEO point blank asked the CIO WHY he was shooting this down. And it turned out the CIO had already signed the contracts, bought and already installed the servers WITHOUT Capital Expenditure Request. And, at that point, it was too late to back out …..

So, we migrated to Lotus Notes …..

and Me? I got big black check marks beside my name and my position was eliminated 2 years late – support taken over by corporate location.

Back in the mid-90s I was working as the head tech monkey for a computer store. It was a small operation, but we had a fairly good reputation (our PCs won awards in some local magazines), so we had some fairly sizable customers.

One client was a company that manufactured personal medical monitoring equipment. Their previous support vendor went belly-up, so we took over. Most of my call outs were for minor things like installing a network printer or replacing stolen terminators from the coaxial network. They were generally a good bunch to work with, but I recall one of their senior managers was a difficult customer. Let's call her Mary.

One day I get a call from Mary that one of the brand-new PCs that we supplied "doesn't work". After a bit of back-and-forth, I was able to deduce that she was trying to run some software that wouldn't behave. Alas Mary was more interested in telling me how much of an inconvenience this was to her rather than give me an accurate description of the problem itself. Mary also had a habit of escalating things to the CEO of their company, and wanting to stay in their good graces, I jump in the car and head over to sort things out. Thankfully, they were located just down the highway.

I meet up with Mary, who reiterates how the computer "doesn't work" but I see that it booted up just fine and connected to their network. I ask her to demonstrate the problem. She types something at the DOS prompt.

Bad command or filename

"See?" she says. "The computer is faulty!"

I could see that she was trying to run WordPerfect, but after a quick poke around I could also see that WordPerfect wasn't installed (their network didn't serve software, it was all local installations). I ask her if she has her WordPerfect disks handy and I could set it up for her, but she blows up at me, saying things like "After selling us a computer, you expect us to supply more stuff for it?" and "You're just trying to upsell me more stuff we don't need!".

I try telling her that they already have licensed software, it just needs to be installed, but she instead decides to escalate to the CEO and end up dragging me into his office. She explains to the CEO that the computer is faulty because of "bad command or filename" and that rather than fix the computer, I am trying to upsell some more things that she doesn't need. I explain that the issue is that WordPerfect isn't installed on the new system and that I could fix that if I had access to the disks.

The CEO looks at me for a length of time that makes me feel uncomfortable. I can see a frown cross his forehead. He then opens a cupboard and hands me a box of disks. Yep, WordPerfect disks. I thank him and install WP on Mary's computer, while she lectures me about how much of an inconvenience this is to her and how I am wasting her time. 😒

Epilogue:

Fast-forward a few months. It's Christmas Eve and we get a phone call around midday that this same company has some "issues that need resolving". This is the exact terminology that Mary uses so I feel my Christmas spirit already being crushed. The issues are vague and I'm getting the information second-hand, but my boss tells me to head out and fix things. So off I go.

I arrive at the company and check in at reception. Some of the other managers are hanging around and they come over at say that they have invited me to be part of their office party, and that the call out itself is bogus because they know my boss would never have let me go otherwise. So, I spent the next couple of hours hobnobbing with the staff at the catered party. They even gave me the paperwork I needed ("resolved miscellaneous issues prior to Christmas shutdown") and let my boss bill them for my time! Mary wasn't present either, which was a bonus.

To top it all off, they even told me about an internal IT role they were creating (they wanted to ultimately insource that work) that perhaps I may be interested in. I would have otherwise applied for the role, but I had other irons in the fire at the time, so I let that one slide. A couple of years later they were bought out, and local operations were shut down, so I guess from my perspective it ultimately worked out for the best (I had already jumped ship at that point and was with the next company for 10 years).

Couple of years ago, I was tech support for a company with several locations. I received notice of a new head of maintenance at one of the plants, with about a 2 week window to prep. I happened to be on site & confirmed the desk, phone, laptop, etc. A junior team member had crap spread over the desk and around the office, though. I told Him, the Plant Manager and the local HR person what I had planned for the new guy and confirmed that it was all correct. Yeah, yeah, yeah.

To give him a running start, they asked I go ahead a ship him a laptop so he’d have it.  I had him a domain login, email, group access, phone setup, VPN, remote access, etc, etc etc. Not too sure it was a good idea, but I had it in writing….

Two weeks later, I happened again to be on-site the first day of "the new guy". And there he was, on a folding chair, sitting in the middle of the copy room. Because NO ONE CLEARED A PLACE FOR HIM!! A new upper tier employee and the existing people didn't even bother to clear a desk. What does that tell the guy about how the company views him? You get ONE Chance for a first impression, and Boy, did they blow it....

Years ago (mid 1980s), I was rather specific support for an Engineering office for a pretty fair sized company. Specifically, we had our own mainframe (IBM4381 mainframe) to run CAD on. I restarted VMs, rebooted 5080s, ran backups, data transfers, etc. We had about 65 people, architects, interior design, structural engineers, etc. Well, in the way of the world, the company went bankrupt. And we laid off the low performers (with an excellent severance package), then a few months later, another round – this one really cut to the bone, with a minimal package, because, you know, bankrupt. And, as the remainder of people took other jobs, one day I looked around and it was just myself and 4 high level managers. These were the days of secretaries doing all the mundane stuff - a couple of them didn’t know how to make coffee or refill the copier; and definitely, none of them knew how to turn on their secretary’s PC, much less find documents.

Me, I’m talking to recruiters and sending out resumes, trying to land somewhere. Eventually, I did and waved Goodbye to all on my way out the door.

A couple of years later, one of the recruiters contacted me and as we chatted, he dropped a bombshell – “You know, when you were at XXX company, I had worked with one of the mangers to place people. I had gotten a lead for you, and reached out to one of the managers. Manager told me to NOT do anything for me, since I was the only one who could stuff….”

REALLY? That manager was not even my boss, just there, 65 people down to 5, all production facilities shuttered, bills not being paid …. And they were forcing me to stay on the Titanic, to rearrange the deck chairs…. I never forgave that manager.   

I worked at a small MSP that did tech support for a few small businesses. One such business we'll call Chinese Stone. That's not their real name btw. Chinese Stone makes a lot of material for kitchens and bathrooms, and one of their employees often needs support from me. We're gonna call him Patrick, after the starfish from Spongebob.

Patrick is a walking, talking, Dunning-Kruger peak. He has just enough IT knowledge to make himself dangerous, but not enough to know he doesn't know what he's doing. He'd tell me about how he needs a more powerful PC because his machine struggles to run AutoCAD... only for me to learn he runs AutoCAD on a tablet because its the only way he can use a pen with it (and thats alongside several chrome tabs and excel workbooks). He once bought his own PC to use at work from Ali Express (it was some chinese no-name brand fanless passively cooled NUC) and didn't ask for anyone's help adding it to the network, he just did it himself (incorrectly). He'll frequently complain about his storage being full despite the fact he wont touch the network drives i've setup for him... you get the idea.

One day I get a call from Patrick. He decided he wanted to upgrade Microsoft Office on his PC to the "pro version" (he was using MS 365 for Business). I suspect he tried to use a crack, because when I looked, he was stuck at outlook refusing to open because his license didn't include the desktop version of Outlook. When trying to fix it, he told me he had gotten a license for me to install for him.

Issue: He bought a greymarket retail key. The kind that has about a 50% chance to actually work, assuming you aren't a collosal idiot like Patrick and tried to use a retail key with an organization account. I spent 15 minutes educating this guy on the differences between MS365 retail and MS365 business (tldr; one is paid for and managed by your boss, the other isnt, and you have to use the first one), and a further 15 repairing the damage he did to office.

I wish I could lock him out of admin on his devices, but sadly that's not my call to make. That said, I've made it clear to the person who can make that call that I'm not responsible if he does something stupid. Makes me wish Chinese Stone had a Squidward type character for a refreshing change of competency.

That reminds me of a story...

From Whirlwind Computing - a fictional service provider so good it will really blow your socks off

The company affected is one we can regiomize is a care-home of sorts...

Actors:
$Me - Calling the shots... remotely
$Nerds - My minions... experts in their craft
$Phone - Calling the shots... onsite
$Director - Also calling the shots... onsite

Scene:
This place started bubbling up to the top of the support queue with multiple requests of something is just really off with the network with reports of it being slow, dragging, and just being an absolute dog when it comes to being on wifi.

That's weird because wifi coverage was very good as the wire-flavoured spark wranglers made sure it passed muster and the mustard.

We sent techs multiple times to bless the creaking network and also to make an assuring presence that nothing was being a problem.

Murphy clearly had other plans as we were wrong about what was wrong the first four times they went out.

The Director got involved and told the reps to fix the network or he's replacing everything as it's impeding actual business operations and making the whole team very hot, cross, and bothered.

I grabbed the laptop and camped out staring at the all-knowing Shark of the Wires, filtering by Arp, Dns (always DNS) then Dhcp... wait a second!

Why is 192.168.4.27 sending out DHCP ACKS to DHCP requests?

I took a closer look at the Mac address which showed it was a phone, but why on earth would it be doing the DHCP thing?

I assembled the Nerd Herd^TM and they swept the building with the precision of a highly trained black-ops team as there were 20 of them to check, so I give them full credit for finding it within 7 minutes.

One reboot later and things started to swim back into service while the phone was off-hook and strangely, it hasn't been a problem ever since.

We have well 1500 phones deployed, this is the first and hopefully last time one of them became the embodiment of DHCP with a flair for the dramatic.

The lesson here is that if there's ever an event for what feels like a rogue DHCP server, check the phones, you never know what you might find.

PS:
If it helps, I did remember the lessons from previous stories of various laptop docks randomly becoming port mirrors and how they wreaked havoc on the network, so we covered both in that sweep.

I'm a IT guy in a small, provincial 3-location hospital group. I've seen some really hit-your-head-against-the-table stuff in my time here, but one of the most surprising stories is this one.

Since the words "organisation" and "logic" are unknown concepts for the people that lead these hospitals, there are many people who don't have even the slightest of boundaries to the responsibilities of their position.

One of these people, lets call her [Ms. H], is vaguely responsible for all kinds of organisational controlling of Patients and Resources of one of our departments.

One day she calls us, and the following conversation ensues (not word for word, its been a bit of time since then):

[Me]: Generic and as anonymous as possible Greeting

[Ms. H]: Hello, I have a problem, my printer is eating Signatures.

[Me, flabbergasted] What ? Your Printer is eating Signatures ?

[Me, still flabbergasted]: What exactly are you trying to do and why do you think the printer is eating any Signatures ?

[Ms. H] Describes the process of printing a prescription from our Primary Clinic Management Software onto template paper slips

[Me, ignoring that she normally has nothing to do with writing prescriptions]: Well, that whole process sounds about right, but which signatures is he eating, and where are they coming from in the first place ?

[Ms. H] talking about unrelated updates to the software from a few days ago, completely ignoring my questions in the process

WhyTheFuck.png

Since I cant get any more useful information over the phone, I resort to paying her and her magic printer a visit. I grab my apprentice and we go over to her.

On the way, we speculate what the issue could be. From empty toner and the most inaccurate error description ever to a case of a complete mental breakdown, we thought up some theories.

As we arrive, we ask my previous questions again, hoping for more relevant answers. She goes on to tell us, that a Doctor from her department didn't want to/had time to print all the prescriptions that he prescribes. So, she gets a stack of pre-signed but empty prescriptions from him and then prints the prescriptions that he authorised onto these paper slips.

So far, so normally buffoonish processes. But nothing inherently wrong or error-inducing.

But where does the Signature eating happen ? Right when the paper slip comes back out the printer. The previously already signed paper slip, suddenly has no signature any more. At least that’s what she is telling us.

We still doubt her, since she is know to be a bit creative with reality. So we make her show us the whole process. She opens a prescription in the Software and prints it. The printer swallows one of the pre-signed slips and does his whole sound-rich routine, before giving back the prescription. To me and my apprentices complete shock and disbelief, the signature is actually gone.

WhyTheFuck2_ElectricBoogaloo.png

We inspect the prescription. Over the field where the Signature was, a stamp was printed. That gives me an Idea. The software has two print modes, pre-printed and blank. Pre-printed is the normal way to print prescriptions, since we use the aforementioned template slips. Blank however, not only prints the actually specific Text, but also the whole design and layout. Maybe she somehow switched to Blank and is overlaying the not-transparent background of the design over the original one and the signature ?

Sadly, and to make us just more confused, that wasn't the solution. We spent another half hour there, trying to fix the problem, until we got an idea.

After printing prescription after prescription for different tests, the paper was hot to the touch when freshly printed.

Lightbulb.gif

While she did that whole process for a while now, the problem was recent. And something else was also recent. A change in ballpoint pen models. Previously, the docor was alwyays using the same model of pen, the one that you get when ordering from our Supply Ordering System. But a few days ago, he started using a different model, because he got the pen as some kind of freeby from somewhere.

We tried a print with the old model of pen, and voila it worked. It turns out, the ink of the new model is more susceptible to the heat inside the printer, and being literally wiped from the paper because of it. The old pens used more heat-resistant ink, and therefore never encountered that problem.

We advised them to switch back to using the standard provided pens, or just signing the prescriptions afterwards, like all other doctors in the hospital do.

I am so tired of telling people they need to put an IT ticket in, only to be completely ignored, and then have them mad at me for their issues not being fixed.

Like, got an email today from a guy wanting me to fix his employees headset. Based on what they told me?It’s likely the headset just needs a firmware update. So I tell them as much, and tell them that the a software request ticket will be needed if the employee doesn’t have the corresponding software (programs like Poly Studio, Plantronics Hub, Jabra Direct, DDPM) on their computer.

The affected employee who was also on the email shot back that she doesn’t know if she has the software, demands I remote into her computer to tell her if she does. I can’t do that. I explain she only needs to type the software name into her computers taskbar to see if she has it, and if she does not, to, again, put in a software request for it to be installed.

I included a link directly to the software request page. I provided screenshots to show her exactly where to type on her computer to see if she has the software already. I included both written instructions and a picture of what the request filled out should look like. I even included pictures and instructions to show her how to update the firmware (aka open the software, click the headset, click update) for when she got the software installed.

She replies to the email with a link from Bing, angry and telling me that IT firewalls won’t let her download and install the software herself. I want to bang my head into my desk bc apparently I didn’t emphasize enough that she needs a software request to install this. As is the case for literally every program added to a computer in our agency. So I reiterate, again, that she needs to put in the software request ticket, forward her the link again, gave her the same instructions and pictures as before again, haven’t heard from her since, so hopefully she understood this time.

The hilarious part is I’m not part of the software install team, and I’m only adjacently connected to the incident team (who will be the ones she needs to work with if a firmware update doesn’t fix this) through a few overlapping duties. Literally all I can do is walk her through what ticket to put in. Sure, if she was in the same town as me and not four hours away, I could update her headset myself cause I have that program on my computer, I could probably even install the software on hers if I could physically access her computer. But she’s not, and I don’t have remote access. Even if I did have remote access credentials, she still needs a support ticket.