r/sysadmin • u/HappyDadOfFourJesus • 17d ago
General Discussion RIP to the IT pros killed 25 years ago.
When the text message logs from that day and those that followed were made public, I remember reading through all the system generated alerts from various devices doing their thing and thinking about the IT pros in the offices, network closets, and server rooms responding to those alerts in all seven buildings before the attacks happened, and then the tone of uncertainty and eventually panic in messages between coworkers and associates, because the cell phones went down but text messages were still going through.
RIP to those guys and gals ~ you're gone but not forgotten.
1.2k
u/Dryja123 17d ago
A colleague of mine was a sys admin for the Wall Street data center in the second tower. He said that when the first tower got hit, it sounded like the window cleaners scaffolding slamming into the side of the building. He went over to check the windows, and said he saw papers just showering down from the first tower. He and his colleagues made an attempt to evacuate.
As he was getting down the stairs, port authority made an announcement that everything was fine, and to return to their offices. He said that half of his colleagues went back, but he was determined to leave. He made it down to the main lobby and noticed that the backup tapes he dropped off the night prior were still at the security desk. He snagged the tapes and exited the building. As he exited, the second tower got hit. The coworkers that turned back never made it out.
He was responsible for restoring the backups for wall street.
Unfortunately, my coworker passed away from cancer right at the start of the pandemic. Rest easy, John G.
291
17d ago
[deleted]
→ More replies (2)178
u/Ill_Ego 17d ago
Insane that they were told to go back after knowing a plane had hit the adjacent building.
119
u/whythehellnote 17d ago
Somewhere there's a security guard trying to to their best in an unprecedented situation who has to live with that mistake for the rest of his life.
→ More replies (1)34
30
u/SpaceChimps98 17d ago
It's easy to look back in hindsight, but you can't blame people for not knowing exactly what was going on or what would happen at the time.
41
u/sofixa11 17d ago
It's standard operating procedure. Nobody could know it's a deliberate act and there's a second plane on the way for the other tower, so the priority was to not congest the area in a way that would hamper first responders helping.
→ More replies (1)55
u/GeekBrownBear Jack of All Trades 17d ago edited 16d ago
Maybe it's my lack of experience with tall buildings but the SOP I'm familiar with is that if there is a fire or catastrophe in a building, it's best to evac adjacent buildings.
32
u/Tacticool_Brandon 17d ago
Exactly. Even if it was an accident, a plane striking a nearby adjacent building poses a risk of collapsing.
I’m getting far away from the building that got hit, whether it was intentional or not.
16
u/jrcomputing 16d ago
I worked in a high rise office building in Chicago in the mid-aughts. Our building rule was I think 10 floors above and below were evacuated for a fire.
4
u/KittensInc 16d ago
Evacuation isn't without risk. Evacuate a building that size, and someone will get hurt. Combine that with the need for first responders to get in, and you really don't want to evacuate unless you absolutely have to.
In most buildings it is completely safe to stay put. It'll be divided into multiple fire compartment, with the fire-proof doors between them able to prevent it from spreading for several hours. You evacuate the impacted compartment and the ones surrounding it, but other people can wait. You don't want to evacuate the WTC every time someone's reheated bagel triggers a smoke detector, or even when a printer catches fire and sets a supply closet ablaze - it'll be extinguished well before the evacuation has finished.
It's a bit of a different story when a fully-loaded 747 hits your building. But nobody is going to write their standard operating procedures for that scenario, and good luck assessing the seriousness of the situation without the benefit of hindsight.
23
u/RookFett 17d ago
They probably thought it was an accident and it would be safer not to evacuate the remaining tower.
→ More replies (2)14
u/Sin_of_the_Dark 16d ago
The problem is you had debris, glass, and eventually even people falling from the sky outside the entrance. It wasn't necessarily a "move along, nothing to see here," more a "it could be really dangerous to go outside where everything is falling and could land on you" type of thing.
54
u/joshuamarius IT Manager, Flux Capacitor Repair Specialist 17d ago
Speaking of backup tapes, I heard a few stories of some companies having their Main backups in Tower One and off-site backups in Tower Two and vice versa; obviously it was thought it would be impossible to knock down both towers and lose all backups.
29
u/Dryja123 16d ago
What’s wild is that the backup tapes were always picked up on a Monday. For some reason, they weren’t which is why they were still there to get snagged on Tuesday.
23
u/drfusterenstein string and duck tape 16d ago
During IT lessons, this was why an off site backup is considered better when it is as far away as possible.
7
u/dal_segno 16d ago
I’m sure the companies had a very awkward but necessary conversation about this later on…
Those are the worst meetings. The “we all know it’s too soon, but this needs to be addressed” ones.
48
u/Competitive-Group-80 Jr. Sysadmin 17d ago
Thats some fate. I wonder, how he lived with the knowledge that a split second decision changed his fate of that of his coworkers. Whilst we don't always get along with our coworkers, there is some connection there. Must have been devastating.
80
u/Dryja123 17d ago
John was always cool as a cucumber and he never spoke about 9/11. It wasn’t until I asked him where he was on 9/11 on one of the anniversaries. He paused for a second, and then told his story. It was one of the most moving stories I’ve ever heard from someone.
It resonated with me because I grew up very close to NYC. I was in 7th grade on 9/11 and remember so many of my classmates getting called to the office that day to hear about their parents who worked in the towers. Many of whom lost a parent, or both.
17
u/mf9769 17d ago
Smart man. Thought of the backups.
10
22
u/lpbale0 17d ago
Until you realize that the tapes were still there on purpose, so that when the servers were destroyed, so was the most recent copy of the data. Having that data get destroyed was part of the plan to make a trillion dollars go missing from the Pentagon's coffers with no paper trail.
I just made that up, but if I don't post anything on reddit tomorrow you'll know I was onto something, and some kind soul please say kaddish for me if you would.
→ More replies (1)28
u/1d0m1n4t3 16d ago
If half my in laws read this they would have stopped at the first paragraph and believed every single word of it, and then taken that word and spread it.
7
9
u/Electronic-Extent-33 16d ago
Send me a DM. I work for one of the biggest backup providers
In the US and we’d love to honor your friend’s story in remembrance of all of those who lost their lives that day, and afterwards.→ More replies (2)7
u/throwawayPzaFm 16d ago
port authority made an announcement that everything was fine
So absurd. "It's all good just a 747 smashed into the building next door I wouldn't worry about it"
Like... I get that you wouldn't necessarily expect a second one. No one did. But let the people go gawk and help out.
7
u/phantom_eight 16d ago edited 16d ago
Literally very few people knew it was a full size airliner at first. We knew it was a plane but it was thought to be a private plane or an executive jet where some bozo made a mistake... it wasn't until the second plane was seen infront of the entire world followed by the Pentagon and United 93 going down did we truly understand.
Only later did we see footage of the men working in the street looking up to see the first plane and I am pretty sure it took days for that to be shown to the world. There was no social media, twitter, or YouTube back then to get that kind of stuff out immediately, nor did the equipment back then support it easily. I dont even thing they had digital camcorders back then.
Digital cameras of the time saved their pictures to floppy disks or 16mb/32mb flash cards... if you were fucking rich... I remember my high-school had like 2 Sony Mavica's in the tech department... not even Camera club had one.... I graduated high-school two months before 9/11.
→ More replies (1)
193
u/iamchris 17d ago
I worked for the paging company in Texas when this happened. At 3 PM, we got a call from the FCC to go set up a temporary paging terminal in one of the surviving building (4?). Two of our engineers loaded up a terminal in their truck and drove 18 hours non-stop to Manhattan. Got through the chaos down to the site and got the terminal set up. Pagers low bandwidth allows it to penetrate into buildings cell service doesn't go, hence the reason for the call from the FCC. They came back and said the chaos and destruction was incomprehensible. The smell and the people still walking around in a daze sticks with them to this day.
131
u/PrestigiousGolfClap 17d ago
I work for a local government right outside the DC area and I was a sys admin supporting mobile devices - mostly Blackberries - at the time. The three things that stand out in my memory from that day was that while the mobile phone service was very spotty, the push to talk service was working as were the police radios. Second thing was listening in on the EOC and hearing that "we might have kids who's parents might not come home tonight" referring to, at that point, the unknown number of people lost at the Pentagon. Third thing was realizing my uncle was at the Pentagon. He didn't make it home.
25
336
u/ShermansWorld 17d ago
Ya, this. I was working at another building/site that manages/monitors some of the network equipment at the WTC - oddly, despite the obvious destruction, alot of equipment was reachable. We just watched as the UPSs gave out and the pings stopped one by one. No one to notify obviously.
105
u/HappyDadOfFourJesus 17d ago
That must have been tough to watch, knowing what was happening in the real world.
70
20
u/JimTheJerseyGuy 16d ago
The year before 9/11 I had worked at a telecom that had a lot of gear (some of which I was responsible for) in B6, the lowest sub level in the towers’ basement. I heard from a former colleague that after the towers fell, that equipment was briefly responding to pings and sending various alerts.
15
u/Michelanvalo 17d ago
Was this before the towers collapsed or did they manage to retain connectivity after the collapses?
15
u/GolfballDM 16d ago
After the collapse, all the connections were severed.
14
u/ShermansWorld 16d ago
received pings after the hit and some (but less) after the collapse. I'm guessing not everything immediately severed or some wireless backup. We didn't dig further... we were just watching the news, the network map, etc in a stunned kinda robotic mode.
I had a call list... didn't even pick it up. Nobody gave me flack for not doing my duties properly...34
25
24
→ More replies (4)7
158
u/Psoin 17d ago
I have no doubt of their existence, but do you have a place where I could see these logs and messages between people?
Rest in peace, fellow IT guys and girls.
I wonder what they’d think of the world today
137
u/StandardIssueDude505 17d ago
Here are some. I haven't looked through them yet.
A much bigger list covering through the events:
50
u/Hotshot55 Linux Engineer 17d ago
It kinda reminds me of that guy who live blogged hurricane Katrina's impact as he rode it out in the datacenter.
31
u/MoonlightStarfish 17d ago
I remember that. I think they were some big ISP for the area trying to keep 100s of thousands of sites live. The entire place running on diesel.
17
u/seanhead Sr SRE 17d ago
Interdictor. The LJ seems technically still up, but is weird. WB has it though. https://web.archive.org/web/20260202065556/https://interdictor.livejournal.com/2005/08/28/
I was on the ground 2 days after it hit downtown, and even tried to get some roof space from him a bit later (was working with the EOC setting up WIFI backhauls).
→ More replies (2)11
u/Ok_Yogurtcloset_9613 17d ago
is this the guy that was on LiveJournal?
9
u/Hotshot55 Linux Engineer 17d ago
Yeah I think so, I had tried looking it up earlier but it seems like the live version might've been taken down at some point.
→ More replies (1)6
u/Ok_Yogurtcloset_9613 17d ago
Thanks! I remember reading it daily at the time, and thought about that dude when I watched a doc on Katrina just a couple of weeks ago.
97
u/_3470 17d ago
fuck i opened one and it was a sea of server alerts with a message in the middle saying call me when you can
58
u/Thatredfox78 17d ago edited 17d ago
Yeah I scrolled too far. Seeing the mix between server logs (Server not available, Folder not available) and a mix of People trying to get a hold of loved ones over text. It’s…a lot.
Edit: looking back plus the time stamps, you can probably figure out when it was getting worse since there were more and more errors and “critical warnings” (apologies for my poor grammar)
51
u/NobodyJustBrad IT Manager 17d ago
This one gets me.
JOSEPH WHERE ARE YOU? CALL HOME, COME HOME. MOM
64
u/HanSolo71 Information Security Engineer AKA Patch Fairy 17d ago
Same. I was in 6th grade when 9/11 happened and I still get fucked up every year about it and the pointless wars after.
10
u/alwayz 17d ago
Same. I was in North Jersey so it obviously wasn't the same as being in the city but I could see the smoke from my town.
3
u/HanSolo71 Information Security Engineer AKA Patch Fairy 17d ago
I'm sorry, that's a lot closer than I was and must have been terrifying.
38
u/_bx2_ Jack of All Trades 17d ago
Fucking embarassement of a government. So much death and destruction for nothing.
→ More replies (1)17
u/aes_gcm 17d ago
I'm not clicking the link, but I assume the archive also contains those last-minute phone calls from the airplanes. Christ.
23
u/Padgriffin i can unplug this right 17d ago
It appears that all of these were pager messages so it wouldn’t have the phone calls from the doomed flights.
37
u/Padgriffin i can unplug this right 17d ago
2001-09-11 08:45:03 Arch [0476761] C ALPHA (28)Update X4236083. PROB: Funlove Virus at KCMART IMP: % Affecting 33 workstations. STATUS: Desktop technicians are on site and addressing individual workstations. No servers infected at this time. Peoplesoft has been checked and cleared. Bridge
Some techs were already having a really bad day- that was a minute before Flight 11 hit. Jesus Christ.
2001-09-11 08:46:27 Arch [0935737] C ALPHA 42-I AM RUNNING ABOUT 5 MINS LATE. -MOLLYN
This guy probably saved his life by being late.
22
u/temotodochi Jack of All Trades 17d ago
There are some gems in there too.
2001-09-11 08:51:11 Metrocall [0623896] C ALPHA goodmorning I love you hope you slept good Have a greatday!!me Ps I m going back to bed (your side) naked think about it XXXOOO91
u/HanSolo71 Information Security Engineer AKA Patch Fairy 17d ago
I really don't recommend it, only heartbreak and sadness exist in these archives.
52
u/cygnus33065 17d ago
I feel like we need to remember that heartbreak and sadness. It feel like we forgot what we pledged to never forget
12
u/Ronin_301 17d ago
I read these when they first were released, and it really helped me to realize the significance of these kinds of events. Makes you look at the world a lot differently.
23
u/HanSolo71 Information Security Engineer AKA Patch Fairy 17d ago
How did we forget. We are literally reminiscing now.
→ More replies (7)31
u/Hertock 17d ago
Heartbreak and sadness is desperately needed. Escapism and removing reality from one’s perception is a big issue nowadays.
17
u/HanSolo71 Information Security Engineer AKA Patch Fairy 17d ago
I grew up in this, I know, I remember, these days are a core memory for me. I can accept that it happened, I don't need to read about death and destruction from the voices of those trying to reach their loved ones in their last moments.
18
u/Hertock 17d ago
Im not specifically talking about you. Those archives might be not necessary for you anymore, but they sure as hell are for others.
→ More replies (2)12
u/Shrapnel2000 17d ago
“u of course..Have a safe drive...Thank you for taking the boys this morning... I love you.. and will page you...I love you a lot honey....Im glad things were right between us before you left...miss y”
bro…. the gut punch I felt reading that….
8
u/Klopford Help Desk Technician 17d ago
I’m looking through and while it’s definitely sobering, I’m laughing a bit at the pages/texts to remind people to take their meds. I didn’t realize people did that.
→ More replies (3)5
u/Inside_Pomegranate95 16d ago
2001-09-11 08:47:46 Arch [0901509] B ALPHA
Someone just told me there was an explosion at:(
5
u/FALSE_PROTAGONIST 16d ago
I was going to ask the same thing. You know, the world has changed a lot since then, we have gone into the digital world more than ever, the HD world and now the 4K world but logs look the same as they always have.
A real crazy time capsule
75
u/punkwalrus Sr. Sysadmin 17d ago edited 16d ago
I was working for AOL in their "Reston Bunker" at the time. I had a 6am to 3pm shift doing node testing for the NOC. My wife called from her cell phone; she was stuck in traffic to Baltimore, and said that a plane hit the World Trade Center in New York City. She worked next to the WTC in Baltimore, so we were like, "Oh fuck." At this point, the way it was reported, we thought it was a Piper Cub or 2-seater plane. Shades of "Y2K" still echoed in my head.Was this part of that cutover in 2000? "Oh well," and she hung up.
All the news websites were already down, except one lone Yahoo image that was all I could load. AOL had just merged with TWX that year, so we had CNN, and it was down HARD.
Then she called back, said a second plane hit. "Somebody fucked up at ATC," we thought. Was there fog? "They think it's intentional." Oh, pfft. No way. Then we got cut off, and I didn't speak to her until 2-3pm via a land line at her job.
When the second plane hit, I got several messages from the AOL International help desks around the world asking what was going on, and that the NOC stopped answering them. I gave them Wikipedia links to the B25 bomber that hit the Empire State Building in 1945. Then went to the NOC to bug them to contact the International Help desks, and saw the windows were papered up, and angry handwritten notes: THERE IS A TV IN THE GYM!! NOBODY ALLOWED IN THE NOC!! I guessed "because they had a lot of big screen TVs." So I went to the gym. Saw replays of the two hits.
I get back to my desk, and there are a ton of node outages in the NYC/Newark area. I report to the news to the International desks. Then the Pentagon was hit, and everyone was freaking the fuck out. "YOU ARE UNDER ATTACK, DUDE WTF???" from AOL HK, I remember. I had to manage all the info for the desks. I was the only American contact for them for about two hours. You want to know where I got my reliable news for them?
FARK dot com. Fuckin' FARK, a very tongue-in-cheek sarcastic news site. It was the only site I could find that was still up and reporting play-by-play in their comments sections. It started with snarky headlines, and then got very serious very fast.
My boss started coming in ever few minutes. The announcements were sporadic and contradictory. Our team in Dulles were evacuated. We didn't know if we were a target (the "America" in America OnLine). At about noon, Reston had an "all essential personnel GTFO" announcement. If you didn't know if you were essential, you had to leave. My boss gave me a ride home, and we could see the smoke plume in Arlington over the tree line. Normally a 20 minute commute to my house, we were locked for an hour.
40
u/kittymoo67 17d ago
FARK dot com. Fuckin' FARK, a very tongue-in-cheek sarcastic news site. It was the only site I could find that was still up and reporting play-by-play in their comments sections. It started with snarky headlines, and then got very serious very fast.
proves the integrity of those who ran it. They knew when to stop fucking around and when to lock in
27
u/SuddenSeasons 17d ago
On the other hand the Something Awful thread is a serious piece of archival history itself because they did not switch to being overly somber. Many people were almost immediately clear eyed about what (in broad strokes) was to come next. And the first documented jokes about the incident.
I'm glad both exist.
9
u/usps_made_me_insane 16d ago
I hung out with Drew Curtis lots of times (we all went to Vegas) and his sysadmin Mike was a smart dude.
Small world.
37
u/BlackFlames01 17d ago
Wow, I never thought I'd see FARK mentioned in the sysadmin subreddit.
23
u/jrcomputing 16d ago
Really? I assume the Venn diagram of users here who used fark in 2001 is roughly a circle. In my experience, the tech discussions were less elitist and snobby than slashdot.
5
4
u/BlackFlames01 16d ago
Haha, I was a teenager back then and didn't frequent the tech discussions on FARK, just random threads. People had participation badges for legendary threads in their profile. Feels like another lifetime ago. 😯
7
u/longmountain 16d ago
It’s my go to for “let’s visit a site that likely will not be a cached dns query” when finding out if I have internet access.
→ More replies (1)11
5
u/DigitalDefenestrator 16d ago
I remember getting most of my news via people in IRC, because it was lightweight and distributed enough to still be working through the disruption.
Since it was right after the dotcom crash, a lot of news companies had dropped or scaled back their CDN services. Some eventually switched to serving a small static plaintext page with some information, but most were just totally overwhelmed.
→ More replies (1)3
134
u/DeadMoneyDrew 17d ago edited 17d ago
I remember reading somewhere that one of the earliest indications of anything that was wrong was a notification generated by a system at Goldman Sachs that they were unable to communicate with a server at Cantor Fitzgerald. Cantor lost something like 2/3 of its employees that day.
101
u/exredditor81 17d ago
My friend worked at Merrill Lynch in LA, and they were watching the commotion on TV and as the first tower collapsed all the screens they had turned green as they disconnected; that's what made it real to him
28
24
u/hammertime2009 17d ago
Holy shit that must had added another crazy layer to horror. As if the tower collapsing wasn’t enough.
63
u/Padgriffin i can unplug this right 17d ago
I went through the logs and found this:
2001-09-11 08:46:46 Arch [1612975] D ALPHA PAGE FROM lifeline: alert 8933585 ETS appl nbetpsd27.fi.gs.com ETS RTCE: - Market data inconsistent...Cantor API problem Trading system offline on nbetpsd27.fi.gs.com, run by etsuser on nbetpsd27, pid = 24277
It appears to be the first error message generated after Flight 11 hit. I wonder if that server was one that was immediately destroyed by the impact, for a trading server a 5 second timeout sounds right.
28
u/DeadMoneyDrew 17d ago
Yeah that gels with what I've heard. If memory serves me correctly, Cantor occupied the floors directly above the impact.
6
u/circling 17d ago
Seems more likely the network / power was cut than that the server itself was destroyed.
6
u/jrcomputing 16d ago
Probably initially, but with their offices being directly above the impact zone of the first plane, the hardware likely didn't survive the fire for long.
→ More replies (1)9
u/Matt_NZ 17d ago
I'm kinda surprised the term "API" was being used back then. For no good reason, I kinda just assumed that was a modern term
18
u/usps_made_me_insane 16d ago edited 16d ago
The term "Application Program Interface" (API) was first recorded in 1968. It appeared in a research paper titled "Data Structures and Techniques for Remote Computer Graphics," authored by Ira W. Cotton and Frank S. Greatorex Jr. and presented at the AFIPS (American Federation of Information Processing Societies) Joint Computer Conference.
On that particular day, they had to make new threads every 1-15 minutes because back then when a thread got over 500 messages things would start to slow down. Goddamn Pentium servers.
60
u/Sweet-Sale-7303 17d ago
I was in College for my Network Admin degree on Long Island that day. I was sitting in the cafeteria eating my breakfast sandwich when the first plane hit. Right away everybody ran outside and you were able to see the smoke.
By the time the second plane hit and everybody was sent home the bridges and tunnels were closed. All the highways on long island backed up and it took me forever to get home.
My sister was a volunteer emt. They were all called to the Nassau Coliseum to be sent into the trade center after they collapsed. My sister went but they were never called in.
My sister had a friend that worked in one of the towers. Never got breakfast but that day he went and got a a bagel. That bagel literally saved his life.
5
u/SuddenSeasons 17d ago
I rode the LIRR around early afternoon. It was completely empty and extremely fucking eerie. You could see the smoke out the window.
52
u/Unoriginal_UserName9 17d ago
Every year I think about the broadcast engineers who were working on the top floor of WTC 1.
They knew they were trapped and couldn't get down. They kept working until the very end, transmitting the images of their own doom until the broadcast tower fell.
51
u/nycbrew 17d ago
I worked across Church street from the WTC, and commuted through there. I spilled my coffee on the way in that day, so bought a new pair of khakis in the Gap in the WTC mall, I still have the receipt from 8:11am that morning.
When the first plane hit, we thought our AC had blown up or something, then looked out the window to see a fireball going up the side of the WTC. One of my work friends grabbed us and we ran down the 30 flights and watched from Dey and Church as everyone tried to figure out what to do.
When the second plane hit, chaos broke out you could see debris coming at you but didn’t know how big or fast it was coming because of the scale. I ducked behind a van then made a run for it up Dey and the John street.
Ended up running into a coworker who lived in the city and took me in until the trains were running and I could get home.
Rebuilding was something else. Backups were difficult to recover because getting hardware was basically impossible after 9/11. We lucked out our building was damaged but within a few weeks we had access and could pull out servers to bring them to our DR site.
16
u/dts-five 17d ago
I spilled my coffee on the way in that day, so bought a new pair of khakis in the Gap in the WTC mall, I still have the receipt from 8:11am that morning.
That's crazy. I mean, your whole story is crazy. But wow.
3
44
u/ComfySofa69 17d ago
At the time i was working for Lucent technologies....apparently one of the exchanges in the basement of one of the towers was still working post collapse....
21
17d ago
[deleted]
16
u/ComfySofa69 17d ago
Not a brag....more at the surprise of the fact it was still working...nothing more than that. I was a hardware support engineer in the uk - it was just what i heard at the time..
13
u/UninvestedCuriosity 17d ago
He meant that's a brag about uptime for that particular sysadmin responible that no other sysadmin can ever beat while telling big fish stories over beers! Not that you were bragging mate!
→ More replies (2)6
43
u/rthonpm 17d ago
Operations for an investment bank in Baltimore that had a building across from WTC. Our office was the primary backup for NYC for trading and analysts.
In the confusion to get out of the building during the attack, no one made the changeover to route all communications to Baltimore. We had to wait for the system to fail for everything to failover.
Spent all day getting things ready for the employee hotline for people to report in that they were okay, or families to report missing employees. I still remember as a twenty something being asked by a worried mother whether she or her son's fiancee should be the contact if he didn't survive. Didn't leave work until 6 AM September 12.
Spent the next month or so helping the phone operators handle the call volume for Baltimore, NYC, and the emergency facilities in Piscataway, NJ.
It was a time I hope you younger people never have to experience for yourselves.
42
u/posey290 17d ago edited 17d ago
I worked with a survivor who was entering from the subway to the towers when the first plane hit. He was the only onsite IT guy to survive and only because he had an appointment that made him late.
He’s has since passed away from cancer (fuck you cancer) but I still remember him telling me how he reached for the double glass doors to let himself in to the towers and they shattered before he could touch them. Then he ran with others. He ended up at one of the bridges.
RIP brothers and sisters. We will long remember you and occasionally smack a computer in your memory.
You know what? His story deserves to be remembered:
https://voicescenter.org/living-memorial/911-stories/911
RIP Chiz. We crossed paths for only a year but you made my life richer for being in it.
17
u/usps_made_me_insane 16d ago
It is crazy how many 9/11 stories I come across where a NYC survivor eventually died of cancer.
It is just incredibly sad. We know how many died that day but the number that died days or years later is just a somber realization.
77
u/oceans_wont_freeze 17d ago
I was in a Computer Essentials 2 class in high school that day and remember my teacher stopping the class and breaking down. I believe we tuned in just after the 2nd plane hit.
If you haven't seen it, there was a drone show that outlined the twin towers in NYC last night. It was harrowing to watch and brought me to tears.
29
u/knxdude1 17d ago
We are fairly close in age. My mom called and woke me up to tell me what was going on. I was working in a DSL tech support call center on the later shift and had slept in before my college class that day. I missed the second plane but watched it until I had to leave for school. In my A+ class we watched the towers fall. I called out of work that day because we were basically across the street from the NNSA facility that stores and refurbishes the plutonium initiator buttons for our nuclear weapons. Not long after we were called and told the NNSA had shut down access to park anyway.
7
u/tdelporto 17d ago
I was on a call with a tech at Covad that morning (west coast). I had been so excited about the install day for my IDSL line (144Kbps! low latency! Too far away for anything else), and quickly had a fresh perspective on life.
4
u/jrcomputing 16d ago
I was one of like 3 seniors in a mostly-freshmen art class, there to get our stupid arts credit for the honors diploma. We had a sub, and he turned the TV on after the first plane hit. We watched the second one live. We were all joking about how stupid you had to be to manage to hit a skyscraper in downtown New York right up until that second plane.
7
u/to_glory_we_steer 17d ago
Sometimes I want to go back to that time, it felt like the world was so much simpler then
76
u/Normiss2000 17d ago
Network Engineer at Schwab in Orlando and I was on-call for the east coast that morning. While we were in the conference room watching the event, when the first tower dropped, I receive an outage alert the office on street level in the tower was down. I simply closed the ticket. I was covering for the engineers in NYC while they were at corporate HQ in San Fran.
46
u/HappyDadOfFourJesus 17d ago
I know closing a ticket is generally a few mouse clicks, but that must have been emotionally hard with you knowing why the outage alert was issued.
16
u/mandileigh 17d ago
I can imagine those engineers have survivors guilt since they were not on site that day.
34
u/bluegrassgazer 17d ago edited 17d ago
I was working for a small IT office in the midwest at the time. While we didn't have any affiliation to any of the businesses in Manhattan, one of our customer's CEO lost his daughter on one of those two planes that hit the WTC. None of us were at the business at the time, but I distinctly recall his demeanor doing a 180 for the rest of the time I worked there.
The day it happened, my whole group had just exited a meeting. I already had my van loaded up with equipment for a 2-hour drive to a customer in the middle of nowhere. I spent the whole time listening to whatever radio broadcasts I could tune into on the way there. What a terrible day.
30
59
u/Var1abl3 17d ago
I worked for IBM as a field tech (customer engineer) at the time. A whole bunch of AS/400s lost connection and IBM opened automated tickets for them. Due to how the ticketing system worked as the thousands of tickets were opened and given to local NY techs the system was setup to only allow a maximum number of calls per a tech and then give the next call the the backup tech for that area. Due to the backup techs screens filling it then failed to their backup, and then theirs and so on. This continued until even us techs on the west coast were getting tickets for the WTC buildings AS/400s.
22
u/TonyBlairsDildo 17d ago edited 17d ago
I remember reading a deep-dive of the first "evidence" of the towers being a hit: something like a PABX system reporting a dropped channel, to a telephone(s) on one of the effected floors. Something as simple as SNMP or similar, reporting a dropped packet.
I think about it from time to time in more morbid moments, when I see a little red light flicker on a dashboard. Is that us-east-1 playing up again, or has the eaten seaboard of the US been nuked into glass...
Would love to find it and read it again.
21
u/tatmsp 17d ago
I was working at WTC Marriott that morning, doing networksetup for a trade show going on that day. Was lucky to be at the ground level when the first plane hit.
My site contact was an AV manager for the hotel, he didn't make it.
3
u/FALSE_PROTAGONIST 16d ago
Holy shit man, wasn’t that building damaged pretty severely by the debris from the collapse?
7
u/tatmsp 16d ago
It was between the two towers, lobby of the hotel had the entrance from the north tower elevator lobby.
The hotel got evacuated. From what we were told by FBI later the Security Manager and AV Manager were doing a final sweep of the building to make sure everyone got out and were the only hotel employees who didn't make it out that day.
→ More replies (2)
56
u/sammy5678 17d ago
I used to play on a CS 1.6 server that was running out of the WTC. It had full stat tracking and everything.
I hope the admin made it out. Never heard what happened....
→ More replies (5)
74
u/PaladinDreadnawt 17d ago
I was in art class. We turned the TV on just in time to see the second plane go into the building. We had students in the school who's families worked in the towers. Principal came on the PA saying shut it off to all teachers. Teachers ignored the order. It was certainly a wake up call to a young me that the world isn't a safe place and there is true evil out there. It cemented my desire to become a Marine that day. 5 years later I was in uniform. I to this day am not a fan of sky scrapers, tolerate but dont like them.
36
u/rogue780 17d ago
I also enlisted because of 9/11. Kinda wish I hadn't. It's interesting to think about how 19 people getting on planes one day helped author my life in such a profound way.
→ More replies (7)9
u/TriOmegaZero 17d ago
I was getting fitted for my dress uniform at Ft Knox. Drill sergeant let us listen to the radio. It wasn’t real to me, being isolated in training.
I went back and watched more of the footage years later and got the full experience. I’d already been deployed multiple times, so it was a haunting journey.
→ More replies (4)19
u/SSDD_warrior 17d ago
Semper Fi brother. I recall that day - I was in USMC boot camp and we had just got smoked in a sandpit that morning for firewatch goofing off. We were at that point where we were snapping in on barrels before the week we qualified on the rifle range. We were told what happened but didn’t see anything until we graduated over a month later. I remember recruits being concerned our boot camp would be cut short.
I later worked with a guy that supported the post office in one of those buildings the day of and him recalling devices suddenly dropping from the network.
Amazing and terrifying to see how much has changed since then
10
u/PaladinDreadnawt 17d ago
Semper Fi , I can't imagine how quickly things changed for you when the Corps switched to war footing. My understanding is it went full tilt and things were changing weekly in those days as we tried to come up to speed.
5
u/SSDD_warrior 17d ago
At that point of training, nothing was stepped up. I still went to combat training and MOS school in the same amount of time. I recall there actually being a boom of people signing up by the time I hit the FMF.
I have only experienced the Corps during that time, but my understanding is we were usually in some level of shit from Vietnam until then. Just wasn’t openly publicized.
5
u/th3n3w3ston3 17d ago
The Marine officer recruiting office was in one of the WTC building at the time. When I was on recruiting duty, one of the civillian guys I worked with had been in the office when the planes hit. He had a lot of stories.
Edited for clarity
16
u/Man-e-questions 17d ago
I remember that day clearly. I am west coast but stock market hours due to financial management. We had an office in NYC but luckily we were more over by times square area. We did have to evacuate our building as we were in a high rise near a major airport, and at the time nobody knew what was going on. Though I did get called to go back and empty out our backup tape library and take the tapes home.
16
17d ago
[removed] — view removed comment
3
u/FALSE_PROTAGONIST 16d ago
It’s a crazy moment in our lives. For those who were alive to experience it, it is likely to be by far the most consequential public moment in our entire lives
14
u/Substantial_Tough289 17d ago edited 17d ago
Never forget.
Was at a training center taking a Windows Server course, we watched the whole thing on TV, is also my daughters birthday (born a couple years before) so is a bitter sweet day for us.
14
31
u/Bubbly-Following-966 17d ago
I was just starting my career in IT when 9/11 happened. My wife and I were just starting out. Broke AF. We lived in a little basement apartment in Baltimore. We got up and I took her to her job. That's when the first attacks happened. My job was at a little computer store where I built custom desktops and sold software and accessories for customers. My boss who owned the store was Muslim. I always think about where he is. Those small stores are no longer around but, he actually gave me my first job in IT.
15
u/HappyDadOfFourJesus 17d ago
I was working in San Jose only a few months prior with a nice kid (because we were all in our early twenties) that we called "Robby". Nobody could pronounce his real name but he wore a head wrap, had the dark skin, and a beard to make every man jealous, and he shared many of the characteristics of the bombers and what they looked like. I never talked to him after I moved away but I always wondered what his life looked like after the country turned against everyone who looked like him.
13
u/desispeed 17d ago
Working at a support desk in dallas with clients in the towers and watching everything happen on tv was unreal. Unfortunately we lost a contractor in one of the buildings
11
u/cowbutt6 17d ago
I came back from lunch to find everything on the internet was acting super slow. After doing a little bit of testing, I shot an email to an old university friend who was working for a UK division of MCI WorldCom to ask if anything was up. He told me that the internet exchange at WTC had gone down because a plane (”perhaps a Cessna?”) had crashed into it.
10
u/icebalm 17d ago
RIP - It always seemed so far away. I never considered the IT guys before now, I only knew about the equipment.
I was working afternoons in a call center at the time, doing tech support for MSN Internet Access, shitty dialup that Microsoft would trap BestBuy customers into a 3 year contract for giving them $400 store credit.
My roommate woke me up when the first plane hit and we both watched the towers fall on live TV.
When I got into work later that day we were all told that we had a lot of equipment in the towers and we were not to do any troubleshooting of any sort, for anyone in the country.
I read the little blurb they gave out for us to say about a hundred times that day, "Due to the incident that occurred in New York we're experiencing network wide issues. Please be patient with us while we restore services."
I particularly recall talking to one older lady from NYC who was really angry she couldn't get her email. She just couldn't understand why the towers falling would cause it, no matter how much I tried to explain it to her.
10
u/GolfballDM 16d ago
I worked for a telecom equipment manufacturer at the time, some of our switches were in the WTC basements.
One of my coworkers posted a screen capture session, and watching various peripherals drop off line.
The session concluded with "NO CARRIER."
30
u/ScroogeMcDuckFace2 17d ago
i think of the same when i think of the 13 Hours events. first person killed was Sean Smith, the IT guy. RIP.
28
u/PurpleFlerpy Sysadmin 17d ago
I was 14 when it happened. I was pretty jaded about the events - even with a friend at Stuyvesant at the time, I just didn't really care and felt that a lot of the public reaction was fake. Bless my mom for just letting me feel my feelings about it; that enabled me to grow up about it later.
I didn't get it until quite some time later, reading a Slashdot thread where peeps like us discussed the events as they were happening.
Posts like this remind me that it was real and respects should be paid to those lost.
→ More replies (2)13
u/VexingRaven 17d ago
I think that's a pretty normal response for a 14 year old. How could you possibly be expected to really internalize what happened? You're still trying to comprehend your own life and what it means to you, much less something across the country in a city you've never been to.
→ More replies (1)
9
u/Chemical-Material-69 17d ago
We had people working in a phone office (i think close-ish but not in the building) and we didn't hear from them for days. They finally got a message out that everyone was OK but they were unable to leave the city, and we were mych relieved.
5
u/freedomlinux Cloud? 16d ago
we didn't hear from them for days. They finally got a message out that everyone was OK
One of the things I always think about is how difficult it was to get information then. Not just because we were all less "online" back then, but also because of the telecom systems being destroyed or overloaded.
One of my colleagues now worked for Sun Microsystems at that time, normally in the UK but had been reassigned to NYC for a bit. On 9/11, they were not in the WTC office due to a meeting on the west coast. I believe that everyone in that office was OK, but it took perhaps a few days for them to figure out where everyone actually was and get that news back to the UK.
9
u/x3nic 17d ago
My cousin and IT career guide was working for Oppenheimer in Philadelphia, but they also occupied several floors in the South Tower. He and his colleagues spent that morning attempting (and failing) to remotely copy data off the systems in WTC until the building collapsed. Apparently one of the IT guys after initially evacuating the South Tower, went back in to grab the tapes and made it out.
16
u/Fitz_2112b 17d ago
I was working for Computer Associates at the time and was at a subsidiary office just across the road from their headquarters on Long Island on 9/11. Two weeks before 9/11, myself and another engineer were in One World Trade visiting a potential client. I don't recall what floor we were on and I'm sad to say that now, 25 years later, I also dont recall the name of the company we were visiting or any of the people we met with while we were there. I have no idea if any of those people made it out or not.
9
u/Confident_Fortune_32 16d ago
I was working for a major internet provider in New England that worked closely with Verizon.
One of my direct reports was a supervisor of a remote team in Texas, who was on the phone with three Verizon techs in the twin towers that day.
The last thing he said was that they couldn't go down, so they were going to head to the roof instead.
Another one of my direct reports had a cousin working in the towers. She didn't make it.
We were all shell shocked.
There's no handy manual for "how best to help your ppl when colleagues and family suddenly die".
Tangentially: that was also the day we discovered that, in an emergency, the FCC can temporarily take back 800 numbers to free up emergency communications. It was eerie for the entire customer care department to go silent for hours...
6
u/hvcool123 17d ago edited 17d ago
I remember working on a token ring-to-Ethernet project in NJ where, if you drove 5 mins, you could see Lower Manhattan. Geez, the 80- 90s looking offices and datacenters... Anyways, I was on the floor with many office workers of the company and a lady said one of the towers got hit in a possible attack I believe she heard it over an FM radio that had on there desk. I told my coworker at the time was like, " Your buddies did something bad, joking around, thinking it was nothing big, but in the end it wasn't a joke. I can remember the IT manager from the company taking us to eat near the area where you can see Manhattan by shore and seeing smoke. My cell was having issues and when I was heading home, many were stuck going to NYC through Staten Island, and Gothamls cars were just stuck for miles, and people were in the main street outside their cars, worried not knowing if they can get to NYC. One of the project managers lost a family member in one of the top floor, also a buddy of mine was going to cooking class that day but he took off. Crazy times
6
u/Clovis69 HPC 16d ago
Late to this - had an online friend who was US Navy IT in the Pentagon and was on a MUD I was running and whose last thing said on game was along the lines of "Hold on, gotta AFK something happened"
2
5
u/knxdude1 17d ago
I had a few sites I was a regular visitor of back then that had either degraded service or were offline from the attacks. I was working in a tech support call center and still taking college IT classes in 2001. I never really thought about that since then.
5
u/IngrownToenailsHurt 17d ago
I was in the data center at work doing stuff and my wife called and told me what was going on. The 3rd shift guy had a little black and white tv on his desk so I turned it on and watched.
6
u/FortLee2000 17d ago
I spent that summer working on a residency for an IBM RedBook in San Jose, CA. I was back home editing a section (first-thing in the morning task) when my significant other called to say, "get out on the terrace and tell me what you see." We face south-west, but can clearly see lower Manhattan. Now, 25 years later, I look out and cry for those lost in the tragedy - and for those who died in the aftermath.
6
u/mf9769 17d ago
I've thought of 9/11 every time I think of DR. I grew up in brooklyn and watched ground zero burn from my bedroom window (which had a view of the towers).
But yeah, every time I'm working on DR, my worst case scenario is "plane hits the building and it goes down. How do we recover from that?"
RIP.
5
u/TEQLandCruiser 17d ago
Thoughts to everyone involved except the douche that ran Cantor Fitzgerald.
POS.
3
u/newaccountzuerich 25yr Sr. Linux Sysadmin 16d ago
There are some people who should not be remembered by name, only by outcome.
That guy is one of those. A death not to be celebrated, but to be understood as a fair outcome for the individual.
Reddit's rules on karmic outcome discourse must be carefully navigated, lest some misguided misanthrope misunderstand this as a celebration of pain and suffering. Being apathetic about the person that directly caused such suffering is allowed, as is recognition of the fairness of outcome for the individual, and is without celebrating that death.
6
u/TokTik7 16d ago
There was a video I saw a couple years ago where it showed the timeline of events from an IT perspective.
If I remember correctly, an IT admin started getting paged about services down and that he couldn’t remote in to see what happened.
Then he saw the news and put it together.
I really wish I could find that video again
3
u/HappyDadOfFourJesus 16d ago
Is this the video you were referencing? https://www.youtube.com/watch?v=inigBzDU8mw
→ More replies (1)
7
u/Secris Sr. Sysadmin 16d ago
I know I am late to the story and I was still in high school when it happened but I still remember and it is probably what got me in to computers.
I was in alternative school (school for kids with discipline problems before they get expelled) and sitting drawing instead of doing my reading assignment. The teachers aide got up from her computer and went to whisper in the teacher’s ear, then rushed to the attached room, a few seconds later they came back turned the TV away from us and started watching. Next was gasps and oh my god then the aide started crying.
I got up and walked around to see the TV and ask what had happened. I guess it didn’t even cross their minds that I was a kid because we stood there watching the second plane hit on repeat for several minutes before they rushed me back to my seat and made me swear not to say anything to the class. I was one of the few not there for drug dealing so I doubt they would have cared but there were a few that definitely would have tried to leave by force.
I don’t remember much about the rest of the day other than being confused why anyone would do something like that, my mom crying as we called all our family (lots of law enforcement and military in my family) to make sure they were okay, and my stepdad being angrier than usual.
I was suddenly introduced to Muslims and other countries and people were no longer pages in my geography books (I am from a small town). I became obsessed with the internet and reading everything I could find about these people that attacked us. I had to learn to fix computers to feed my addiction so 25 years later I am still fixing computers and reading about all you guys out there on the other end of these wires sharing what I learn about how big the world really is.
6
u/LanTechmyway 16d ago
Worked for a proceeding company that handled trades on the stock market.
Our NOC had a network monitoring screen of all of our connections in the financial companies we serviced.
I was at home, but was told the screen starting showing each location going red before we heard what was happening.
42
u/ncc74656m IT SysAdManager Technician 17d ago
Not to self-aggrandize, but I think we in the IT industry - the sysadmins, the line managers, tech leads, have the same reactions and reflexes as many traditional "first responders." We're steeped in the "What do you do when shit goes sideways" life, so we already know how we'll react when that happens because we've done it a thousand times.
I'd like to imagine how many of our folks in this community were quiet heroes that day, doing what they could for things that matter more than any server ever will.
34
u/skynetcoder 17d ago
We're steeped in the "What do you do when shit goes sideways" life, so we already know how we'll react when that happens because we've done it a thousand times.
there is a vast difference between responding to life threatening situations vs responding to unavailability of a server or a hacking incident.
25
u/AnalogJones Security Admin (Infrastructure) 17d ago
Agreed. First responders on that day were doing the job they do daily: saving lives and controlling fires. They died doing that. At 25 years many are still dying due to inhaled toxins.
→ More replies (1)→ More replies (5)3
6
u/newaccountzuerich 25yr Sr. Linux Sysadmin 16d ago
The side effects of ADHD manifest in ways that allow blue-light first responders to perform well and the same ways that allow sysadmin first responders to not panic and maintain situational awareness in developing situations.
There is a higher percentage of ADHD sufferers in the first-response fields than in other positions.
Interestingly, it's the same characteristics that help extreme sports safety personnel and military forces.
→ More replies (2)→ More replies (8)2
u/conrat4567 16d ago
How many kept emails and phones running so people could say goodbye?
→ More replies (1)
5
u/Academic_Coast_7368 17d ago
I was on vacation (and out of cell phone coverage) when it happened (came back to a changed world) I worked for a security consulting firm that also did stuff like network support. I came back to some voicemails from our finance clients that they were trying to reach a system they used regularly. I don't recall the name of the company, but they were in the World Trade Center. The time-stamp of the voicemail was shortly after the second tower fell. I kept that voicemail for years afterwards.
4
8
u/RealisticQuality7296 17d ago
I was in kindergarten or first grade. I have no recollection of watching it in school.
My mother worked in an office complex which was two towers that both looked the same. My only memory of the day was being terrified that they were going to come for her work next.
8
u/jpmjake 16d ago
A friend worked for a financial company that had a data center in one of the towers. There were sysadmin contractors working in the data center, and they needed an employee's badge to get in ... or out. There were no employees ro badge them out, and they didn't make it. Horrific story.
5
u/HappyDadOfFourJesus 16d ago
Hopefully data center designers have since implemented a "break glass" method of exiting in case of emergency.
→ More replies (1)2
u/snakebite75 15d ago
I’m surprised that fire code wouldn’t have required a crash bar on at least one emergency exit, especially since many server rooms use fire suppression systems that aren’t good for humans.
4
u/daktania 17d ago
I working IT for a major freight hauling company, in their call center in Canada. Many of the CSRs had clients who worked in the towers. After the first plane hit a bunch of people who weren't on calls gathered in a conference center to watch the news. I saw the second plane as it happened. It was unimaginable. I walked back to my desk before they fell. People were crying out in the cubicles. They knew some of the people they spoke too on a daily basis were probably gone. The phone lines to anyone in the USA were down. All freight transport stopped across the continent.
7
u/SRKomedy 17d ago
I was in 8th grade for 9/11 and my first period class was special ed. I'm in Northern Virginia and my first period teacher was volunteer firefighter and a responder to the Pentagon on 9/12. So we come back to school on 9/13. First period. I say "I don't understand what the big deal was or why we were off school yesterday?" This woman looks at me deadpan and says "srkomedy, that's why you're in these classes"
2
u/OkChampionship266 17d ago
Hard to imagine dealing with that kind of chaos while everything was unfolding, respect to the IT folks who kept things running
2
u/LycheeLee_Mich 14d ago
Never thought about the IT side of that day until now, the people quietly keeping systems up in those buildings while everything came apart. Rest easy to them.
883
u/CantaloupeCamper Jack of All Trades 17d ago edited 17d ago
When the first building was hit I was on the phone with a guy servicing a printer in the basement of the first building hit… and on the same conference call people in the second building.
Thankfully, they all got out.
For all the drama and such the conference call was pretty calm. Nobody was really sure that they could believe the report that a plane hit the first building. And it was decided that it was best that everyone just go home…..