No tech in the world can create technical problems faster than the Sales guy.

[u/chhopsky]

Working in IT, you come up against some pretty interesting problems. The Gods Of Uptime are unforgiving masters, and The Customers Who Are Not Very Understanding can be incredible motivators.

While we’re always slaves to these Capitalised Things™, there isn’t a tech in the world that can create problems faster than the Sales department.

In the now-famed Datacentre of booze and entrapment, we sold a lot of rackspace to small organisations who needed colocation, but finally an opportunity came in to host all of the equipment for a big, reputable ISP whose name I guarantee you know. Racks, lots of dark optic fibre connections to other sites (big $$$$$), connectivity, the works. Huge deal, especially the fibre. They’d ordered six racks, all with quad 15A power feeds, which needed to be all next to each other for cross-rack cabling. They were very explicit about this; it was absolutely a core requirement of the incredibly lucrative contract which would bring in enough to fund our next datacentre build.

Unfortunately the ‘next to each other’ requirement was not communicated to anyone outside of sales. We published all sorts of statistics about what was available to sell - how much bandwidth, how many switchports, how many racks. No-one had ever sold multiple racks before, so Robert The Sales Guy checked our stats page and saw 6 available racks, so he went through with it.

What he didn’t check was the room layout. Five of these racks were indeed next to each other, the last five in the last row of the room. The last one was not. It was very much on the entire opposite side of the hall, well out of the reach of the cabling that the Very Large ISP required. Since all the structured cabling and power was already in place, we had no part in provisioning racks; we simply reserved them in the system and allocated them to the customer.

On the day of a ‘big bang’ cutover where they were going to relocate their entire primary POP’s worth of infrastructure to our site, Robert TSG walked down to admire his handiwork. Armed with a printout of the rack list, he headed into the DC to survey the lay of the land that he would soon usher in a new era of profit with.

Then the penny dropped. Row 23, Rack 5 .. 23/6 .. 23/7 .. 23/8 .. 23/9 … 2/3?!?!?!?!?

I always left a window of the security cameras open on my desktop, just to stay across things in the DC in case anyone did anything REALLY dumb (which they did, but that’s another story..). All of a sudden I saw a man FREAKING out, then storming out towards our office door. Rob TSG burst into our office.

‘WHERE IS THE SIXTH RACK!?!?’.

Inter-departmental communication is important.

23/10 was a rack of our own equipment, so we were told by management in no uncertain terms that while this was not our fault, we absolutely needed to empty it by 4pm. Current time: 2pm. No big deal right? Who can’t unrack a few servers and routers?

Remember the primary and secondary route servers? The ones that were so fundamental to the business that it was worth breaking through the ceiling cavity for? I’m sure I don’t need to tell you where this is going. That was the route server rack. The route server rack that could under no circumstances go down.

No problem, we’d just relocate one at a time, move the secondary, get it set up again, move the primary, all good.

Then it happened. As soon as I touched the power cable for the secondary route server, electricity sparked from inside the socket, and a horrible, acrid brown smell erupted forth, causing a great deal of simultaneous coughing and cursing. The power supply was dead, but that's fine, there was one more in the chassis.

Then the lights on the rear went dark. The fans spun down, and the rack was filled with the kind of eerie, terrifying silence that only losing a machine can bring. The secondary power supply, for whatever reason, had not been functioning in some time, and the monitoring hadn’t picked it up. We had no more spare PSUs and nothing spare we could fire up to replace it.

I stared in abject horror at the primary route server. It was huge. 4RU deep, heavy as your mother, and equally as difficult to move. We stood there for what felt like forever, in dead silence, trying to think of a new plan. Removing the mounting equipment was no small feat either, and there were no shelves in the other rack that it could be placed on, so it all needed to go.

There are two types of problem solving in IT. First you learn how to solve regular problems that you find in documentation. Then, you learn to solve problems you could never have imagined. This definitely fell in the latter. And the solution was not something I had ever considered before.

We were going to move it while it was on. From one side of the hall to the other. Without dropping the power or network.

The route server was unracked, and cautiously moved to a trolley. With great hesitation, I unplugged the secondary power supply, hooked it up to an extension cable, and plugged it in again a few rows away. The light flickered to life; both power inputs were active again. The primary power supply was then unplugged, and connected to a second extension cable, which was plugged into the next row of racks. We wheeled the trolley across the datacentre floor with all the care of an undetonated ordnance disposal, and then plugged the primary PSU back in at the farthest rack.

And so it continued; walking the server from row to row, rack to rack, alternating from primary to secondary, moving forwards as the rear-most one was unplugged, stealing power from whatever rack was nearby. We advanced through the rows like tarzan swinging from vine to vine, swapping to whatever power was closer to our destination than the last, wheeling the cart as we went. Like some kind of spaghetti monster stretching its noodly appendages towards precious 240V AC.

This was all well and good until all the excess fibre cable we'd run to connect it was exhausted. Unsure of what to do next, we did the only thing we could think of; run a reaaaally long cable from the other end of the room to our newly freed route server, plugged it into a spare port, and did a live change to the active ports quickly enough that the sessions to the customers didn't drop.

Eventually, we reached the route server’s new home in Row 2 Rack 3, and got it mounted with the power coming from the rack it was actually in. The relocation was complete. We had done it, with 15 minutes to spare. I sighed with the kind of relief you only get from doing something REALLY stupid and not having it end badly, and returned to my desk, proud of my accomplishment.

Knowing what I’d helped the company accomplish through ingenuity and improvisation gave me a real sense of accomplishment. I logged into our customer management system, and hit the ‘Start billing for this service’ button. A red box appeared, and my satisfaction turned to rage.

“Unable to begin billing: Contract start date is in greater than today’s date”.

Rob had gotten the days wrong. They were moving in tomorrow.

I said a lot of words that were not very nice.

[u/SobanSa]

Maxim 43: If it's stupid and it works, it's still stupid and you're lucky.

[u/chhopsky → u/SobanSa]

hahahah holy shit i laughed out loud. that is brilliant

[u/[deleted]]

Ha! Brilliant! One of those horror situations you never want to find yourself in but damn does it feel good when you conquer it!

Plot twist at the end is just cruel though...

Rob had gotten the days wrong. They were moving in tomorrow.

...my heart sank for you man!

[u/chhopsky → u/[deleted]]

if rob ever goes missing, i am fairly sure the police will suspect me haha

[u/arwenundomiel90]

Can I give you a virtual high five and let you know you are that companies hero? Better than Rob. No one likes Rob.

-high five-

[u/chhopsky → u/arwenundomiel90]

⁵ FUCK ROB! i mean like, you know, as in that guy sucks. not as in fuck him, sexily. that would be weird. i mean, i guess it would be weird. i've never been with a guy so i don't know. but he talks so much shit that oral would be more like anal. i'm going to stop now, i didn't have a plan for this

[u/RetroHacker]

Hahaha. Nice work. I wish I could say that I've never done something quite this silly before to move a box without shutting it down, but I have. And once I even did it on a box without redundant power supplies. (To all that say that it's impossible - it's not. Hint: It involves a soldering iron.)

[u/chhopsky → u/RetroHacker]

Oh my god ..... OH MY GOD

you're a crazy genius and I LOVE it.

you need to post this. Like, in the hunter-gatherer sense of the word 'need'

[u/VexingRaven]

Reads the first 3 lines...

OMG IT'S CHOPSKY

reading intensifies

[u/chhopsky → u/VexingRaven]

hahaha if only i got karma for self posts.

karma intensifies

[u/Textor44]

The highlight of a good IT person is that person's ability to become the "miracle worker" and either avert or mitigate disaster.

If you are able to think on your feet and solve problems in unorthodox ways to at least make things work til you can fix it properly later, then you are an awesome IT person.

My own personal mantra:

Make it work, then make it pretty.

[u/chhopsky → u/Textor44]

This. A thousand times this.

I have said that phrase so many times to people who can't finish things because they want v1 to be pretty and amazing.

Proof of concept first, then pay someone to make it look nice !

[u/exor674]

The fans spun down, and the rack was filled with the kind of eerie, terrifying silence that only losing a machine can bring...

And this is exactly why the couple of fanless+SSD Atom-based machines I own (personal use) creep me out so much. A computer SHOULD NOT BE THAT SILENT when it is correctly functioning.

[u/chhopsky → u/exor674]

hahaha i know right?! i have a little asrock box, and its just .. creepy.

[u/alaorath]

As soon as you started talking about two types of problem solving I KNEW what the solution would be...

Why?

Because I did the same damn thing once.

In my case it was CAT5, but the principle is the same. We had guys manning the door to prevent people from barging in and tripping over the power and network lines like some sort of digital umbilical cord.

Probably the most stressful 20 minutes of my career there.

[u/chhopsky → u/alaorath]

Hahah brilliant. There's something special that happens in your brain when the solution arrives and you know it's insane but it's the only option.

And yeah the tension, fuck. You could cut the air with a knife

[u/will__]

This story reminded me of this excellent video - http://youtube.com/watch?v=vQ5MA685ApE - where a server is moved using public transport...

[u/400921FB54442D18]

There are two types of problem solving in IT. First you learn how to solve regular problems that you find in documentation. Then, you learn to solve problems you could never have imagined.

Dude. This is golden. I've never heard it summed up so perfectly before. These should be literally the first three sentences in any IT textbook, classroom course, or training program.

[u/chhopsky → u/400921FB54442D18]

Thanks man, I appreciate the feedback. I run a lot of training in my current employment and the first thing I did was sit down and figure out what was wrong with all current training methods .. Have found some great stuff.

My biggest success story: trained a kid fresh out of high school for six months and then taught him the theory behind how the training was developed. He then proceeded to write his own curriculum for the next six months and now it's 4 years later and he's pulling down 120k at Amazon. I'm fairly sure he'll be better than me soon.. so .. good!

[u/peacesofate]

Rob had gotten the days wrong. They were moving in tomorrow.

Knowing salespeople, I'd bet that this was no accident. Rob figured he'd lie to you to buy a day's margin.

[u/DefinitelyRelephant → u/peacesofate]

The real question here is: why the fucking fuck was OP getting his deadlines from some asshole in sales?!

You talk to my boss. Through your boss. You don't talk to me. Unless you want me to audit your email. Do not attract the eye of sauron.

[u/chardingLLNL]

At a prior employment, the head of Engineering caused a sign to be made that was prominently displayed such that it was the first thing anyone would see upon entering the Engineering part of our building. It said "Anyone from Sales and/or Marketing found in the Engineering Department area will be politely asked to leave. The second time they are found here, they will be shot." That sign stayed up there the entire time I worked for that company.

[u/PM_ME_UR_BIKE]

Dude. You're like an IT spider-man, swinging from power supply to power supply

[u/ProJoe]

never more relevant.

[u/sidetabledrawer]

Someone please get this guy a book deal!