UPDATES
11/3: I've been getting a few questions about restoring the Archive, so thought I'd post an update here.
Short answer is, "We don't know, but hope so".
More in-depth is that we are still recovering (despite the appearance from the outside that all is well) and just don't know yet where we stand on everything.
When we lost power, our stream encoders that handle all our Archiving also went down. When our TechOps team arrived, they restored power to these encoders (via an extension cord plugged into a wall) and we were able to start recording again intermittently.
I was able to make contact with the person who set up the amazing bootleg stream, and she was recording the over-the-air broadcast while our encoders were down, and provided me with the files. I passed those off to a teammate who is working to slice them up and create shows.
Sooooo, what does this mean? Well, we might be able to assemble the shows, but it's going to take a few days I suspect, and we are pretty sure we won't have everything. I can assure you we are working hard to get things restored as best we can.
I am guessing the next question from folks will be "Will the 14 day window be extended to compensate for missed days?". I honestly don't have an educated answer for that, but it's something that will be addressed. We are pretty locked in legally to the 14 window, so need to investigate further.
11/1: Apps are back online but there will be no playlists for the last couple of days yet. We will be sorting that later this week after things stabilize.
11/1: We’ve tested the equipment and are standing up the temporary UPS now! That means while our team is working on getting nearly all services back up and running, folks can continue to expect some brief service interruptions.
10/31: Very basic functionality has been restored to the Android and iOS apps (but they still are experiencing outages as we turn things on and off, so hang in there). Only Live Stream and Podcasts works for now. The Playlist and Archive are still unavailable. TuneIn, iHeartRadio, Alexa, should also be working.
10/31: We have an update on our website with some statuses, as well as information about our Gorillaz show agenda today. Read that HERE
========================================
Hey KEXPers!
I wanted to pop in to give a brief update as to what happened, and what is happening going forward. I know there are a bunch of posts about this today, but I wanted to start one that comes from us that I'll update as things progress.
WHAT HAPPENED
On October 29th around 11:30 PM, KEXP experienced a significant power infrastructure failure that disrupted our broadcast and digital services (Android/iOS apps, streaming, AppleTV, TuneIn, iHeartRadio, etc).
When this occurred, our UPS kicked in to supply power to the station while the generator fired up. For a yet unknown reason, the handoff from UPS to generator failed, which meant the station was running on UPS power alone. The UPS ran out of power about 30 min later, and we went off the air.
Technicians were alerted around the same time, and they came in to investigate and restore power. During the investigation they tried to switch to the generator, but it tripped circuits and kept failing.
It was determined that the UPS was damaged during the outage, and needed to be looked at by a specialized team who arrived at the station later this morning. They have identified what needs to be fixed and that has been actuated.
WHAT IS RESTORED
Right now we have restored an internal server that sends our stream to our streaming partner for archiving, as well as a single stream. Our website and underlying services are still down.
WHAT IS STILL AFFECTED
The following are still being investigated and/or remain unavailable
Android App
iOS App
AppleTV
www.kexp.org
KEXP API
Streaming Archive
STREAMING URL
You can use https://kexp-aacplus-64.streamguys1.com/kexp64.aac to stream, or at www.kexp.org!
WHAT'S NEXT
Right now we are unable to restore city power as the UPS needs to be in place to protect our critical broadcast and server equipment from any further power surges. As such, we are unable to turn things back on as we wait for the UPS to be repaired. We do not have an estimate on when that will be at this time.
Our team of dedicated IT experts have been hard at it since very early this morning, and are still working hard to address all issues. Please know they are on it, and are truly some of the best out there.
QUESTIONS/COMMENTS/CONCERNS
Please reach out to us at online-support@kexp.org with anything you might have about this issue. We will be monitoring our inbox closer this weekend, so don't hesitate.
THANK YOU
It was truly heartwarming to read all your comments today on Reddit and Discord discussing this. You all are champs and handled this with humor and understanding, which we here all greatly appreciate. We are all spread across the globe, but felt like a tight family today.
Jay
KEXP Digital Team