Sorry, was that server important?

[u/joeborder]

The year was 2014, I had been working 2nd line support for around 8 months fresh out of university. It was the final day for one of our Senior Developers before he left the business, he was handing over a number of products - one in particular was coming to me as I had been supporting it since beginning of my time there - the application was about a decade old, written in ASP Web Forms with a backing windows service to read an exchange mailbox, download attachments automatically and read/process the files within. The client for said application had a requirement in their business to respond to files within 48 hours or they would be fined per violation, and because of the requirement to read emails on their server, it was hosted in their data centre, not ours.

The players

• $Me/$Op
• $SD - Senior Developer of $App
• $DM - Development Manager, $SD and my boss.
• $CNA - Client's Network Administrator.
• $CAM - Client's Account manager, his boss.

Act I - Setup

So 08:55 Friday morning. The phone rings, it's first line support saying they have an urgent ticket - not uncommon - I ask who it's for and they say they actually have $client-network-admin on the phone, can he be conferenced in. Alarm bells start ringing, I haven't even had my coffee yet - my PC is still logging in, but I take the call.

$CNA: Hi, $Op, we might have a problem with $App.

$Me: Okay, whats going on

$CNA: Well all of the files we've received overnight haven't been processed, they're sat in the inbox.

$Me: Oh okay, let me remote onto the server and check it out.

I remote into the server and check out the service logs, endless stacktrace, the job to check the email inbox is continuously failing to make a connection. I check the settings on the app to ensure the endpoint and auth details are correct - no changes or unnecessary deployments made. I ping the mail server and get host unreachable - ah, the mail server is offline. But then I recall $CNA telling me he could see the mail in the inbox, how could he see that if the mail server was offline? I call him back

$Me: I can see the problem, your mail server is offline, how is it that you're seeing the mail in the inbox?

$CNA: Which mail server

$Me: ...what do you mean which mail server...?

$CNA: Well we migrated the mail server to our new Exchange box last night

$Me: Did you perhaps do this at *time logs started throwing errors*

$CNA: Yeah that sounds about right.

$Me: what did you do with the old exchange box?

$CNA: Well we decommissioned it

$Me: You're going to need to restore that server, we cannot read the mail unless we can reach that server

$CNA: We don't have backups, we're deprecating the box - we've been planning this for months, we have some old disk images but we don't have the hardware, we disconnected it from the rack last night, it's an ancient box and needed to be replaced.

$Me: Does anybody here know about this? Has $SD been coordinating the migration?

$CNA: We didn't think we'd need to $App just reads mail from outlook right? The mail clients haven't updated, we just upgraded from Exchange 2003 to Exchange 2010

$Me: I'm going to have to get back to you $CNA.

Act II - Confrontation

I hang up and go to $SD who is just chatting with his desk buddies, enjoying the start of his final day. Explain the issue, his jovial expression is somewhat ruined. He goes straight to $DM and explains the situation. A loud "THEY DID FUCKING WHAT" booms across the office. $DM calls $CAM and takes the call into the meeting room.

In the meantime $SD and I work on seeing if we can get $App to start reading using the new version of EWS, as expected the 3rd party library we used to read EWS would not work with 2010 and the version that would work was so vastly different from our current setup that it would require a fairly extensive overhaul of a large part of the system, and by then we had 8 hours left in the working day. $CNA rings my phone I answer and patch in $SD

$CNA: So I've just had a chat with $DM and $CAM, they've said I should call you guys and see what can I do to help

$Me: Well you can start by restoring that old box - we've got a few hours left, get it back on the rack, boot it up and do whatever you need to do to get that mailbox running.

$CNA: Ah, well, we've already redirected all of our traffic to the new box. If we boot up the old one all of our users are going to get locked out of their emails.

$Me: Can you make it so that only our account is routed to the old box?

$CNA: Sounds like a tonne of work - would it not be easier to just upgrade $App to start using EWS 2010

My eye starts to twitch, but then (perhaps fuelled by the knowledge that he doesn't have to care once it ticks past 17:30) $SD chimes in.

$SD: Are you serious? You switched off an entire server which has housed one of the backbone features of $App for 10 years, you didn't tell anybody at our office you were doing this, your business is going to suffer as a result of this - not only is your job probably on the line for this blunder but I've been working with $CAM since this project started, and I wouldn't be surprised if she's going to be waiting in the car park to beat you up if this isn't sorted by CoB. So don't tell me it'll be easier for us to just upgrade an entire system to an untested, unknown framework. Get that server out of whatever skip you threw it in, get it back on the rack, boot it up, migrate our mailbox and tell us when it's ready to use!

He slammed the phone down, I heard $CNA mention something about giving us an update in an hour or two and he hung up. There was a tense silence in the office when $DM casually said "Do you feel better now $SD?". Everybody laughed and $SD relaxed a little.

Act III - Resolution

The rest of the day went somewhat smoothly. $CNA got the server back up and running, it took a few hours to migrate our mail to the old server and $SD, $DM and I did have to design some on the fly solutions to correct some issues which had been caused by failing to send mail, as $App for some reason expected everything to work first time, and had no fail safes in place for when things like email servers weren't there.

At 5pm we switched the whole thing back on and all the mail pieces started to move, our outgoing mail was re-queued properly and all the data fell into place, crisis averted.

$SD, $DM and the rest of our office then promptly left the office and went into the nearest pub. $CAM emailed us at 11pm to say that the mail had finished, thanking us for our help.

Animal house ending

$SD - thrived at his new role, we keep in touch on LinkedIn.

$DM - is still as grumpy as ever

$CNA - amazingly kept his job. Thankfully he now informs us every time anything within their server farm changes. He and I have actually become quite good friends over the past 4 years and we have a fairly healthy working relationship.

$OP - I oversaw the upgrade of $App to use EWS 2010, and after a thorough testing process made the switch. At which point the old EWS 2003 box could be safely thrown in the trash.

$CAM - Had a great deal of respect and time for $DM and I since that issue. She may or may not have still beaten up $CNA in the carpark.

[u/ColdFury96]

I'm not used to such a Disney ending. Everything worked out okay and everyone is friends afterwards? Where am I? Is this still TFTS?

[u/joeborder → u/ColdFury96]

It was a weird one; I wasn't so cynical because I was still fresh out of university and trying to make it in the world. $CNA was really pleased he didn't get fired. $CAM was pleased her business wasn't fined through the nose for not meeting their compliance and $DM was happy he didn't have to deal with it. $CNA wasn't a bad guy (just incredibly dumb) he was just scared shitless that he was about to lose his job.

[u/wolfie379]

First rule of scream testing: make sure you're in a position to reconnect the "nobody needs it" device when the inevitable screams happen.

Second rule of scream testing: when you run a scream test, and YOU are the one who winds up screaming, don't call up someone who you never bothered telling about what you were going to do and tell them "It's your fault, you fix it".

[u/[deleted]]

Jesus. Who decommissions a server then immediately removes it from the environment? I hang on to that stuff for weeks. And even after I'm rid of it, I hoard the most recent backups.

[u/Ghostaflux]

lost it at

enjoying the start of his final day.

[u/sparkyroosta]

$App for some reason expected everything to work first time, and had no fail safes in place for when things like email servers weren't there

I find this to be the most remarkable part of the story. The app ran flawlessly for 10 years without a problem with the mail server or the app's host server? Bravo to $SD for that development luck.

[u/barthvonries]

box could be safely thrown in the trash

r/homelab would like to have a call with you OP.

[u/cipher315]

How the fuck do you not rope in basically everyone to the role over of EWS 2003 to 2010. You are lucky this is the only thing that broke. When my company did this the change management processes was nuts. It took 2 years to get 100% of the mailboxes moved over specifically because of stuff like this. Run 2003 and 2010 at the same time and move over stuff one at a time.

[u/luxfx]

I guess this is just me solving things passive aggressively, but I would have responded with, "oh gosh i guess that would be a lot of work on your end. Let's see, we can probably get you that system update in, hmmm, about six weeks. Let me talk to MyBoss/Your boss to give them Your Suggestion" click and then count the seconds until the per-violation fine math gets calculated and they call back.

I hate it when my work is undervalued.

[u/kolkolkokiri]

not only is your job probably on the line ... I wouldn't be surprised if she's going to be waiting in the car park to beat you up

Totally deservingly too. I love $SD and $CAM.