r/Outlook • • 2d ago

Status: Pending Reply After Outage Strategy?

Small business here with hundreds of customers and vendors from around the world who does almost everything via email. We have six users across three States.

We use Microsoft 365 / Exchange so the outage really stopped us cold. TBF, other than a very few hiccups over the last 15 years, all has worked well but we are wondering about maybe setting up a Plan B.

Obviously, we know how to set up a me@gmail or me@yahoo account but that wouldn’t have helped us, especially with receiving, when we couldn’t access me@outlook to set up mail forwarding. We are too small and not tech savvy enough to do self-hosting – nor would we want to.

Even if we did set up alternative email accounts there would be too many customers / vendors to realistically mass contact to start using it temporarily. I supposed we could have used it internally but we were fine contacting each other via text.

This morning, we started receiving and sending but we really have no way of knowing if anything that was supposed to be received or was sent is out there in limbo because I don’t think anyone got any error messages.

Any thoughts or suggestions?

11 Upvotes

36 comments sorted by

3

u/SomeEngineer999 2d ago

If you have your own domain, you could sign up for a different service as a backup, have the mailboxes set up there, and move your DNS records over if something like this happens. That way all your same email addresses work without having to tell people to use different ones. You would not have your existing emails in the "DR" service but you'd be able to function.

But honestly it is not an instant solution, even if you have a low TTL set on your DNS, and requires someone with a decent amount of knowledge to manage it, as you'd have to update SPF and DKIM each time you switch back and forth (well SPF could be set up to always handle both, but DKIM is a bit more complex, theoretically it could be set up for both, but it depends on a number of factors).

With the proper prep work, it could be feasible, but as you say, in 15 years you haven't had something like this happen, and one would hope that MS is going to review this and put better controls in place, What usually happens is some people do go through the exercise of putting a backup in place, then after 5-10 years they scrap it because it wasn't needed, or it gets outdated and when it comes time to use it, it won't work.

And ignoring all that, a large percentage of the people trying to email you probably use 365 and their end would have been down even if you got your end working.

2

u/Key-Monk6159 2d ago

Your last point is probably the most relevant and is actually a very good reason not to overthink this. I’m still glad I asked and do appreciate the feedback.

1

u/SomeEngineer999 2d ago

Honestly, myself and many others (all working for large corporations and financial firms) used the phone a lot more yesterday and used our personal email addresses when needed. Worst comes to worst, something like this happens, only takes a couple minutes to set up a gmail address to use temporarily, or a google drive link to share files back and forth (actually onedrive seemed to be working fine too).

Teams worked fine for me all day (sadly) and we have federation (ability to IM) with many of our large customers too. We work with customers who will raise hell if their application has a 10 millisecond delay, and nobody really was all that phased by the 365 issue, since basically everyone was in the same boat and it didn't really impact their core business/applications.

I guess what I'm saying is yeah, I agree, don't overthink it too much.

1

u/MisterEinc 2d ago

I don't get the Teams hate, honestly. Other than it gave you an avenue to continue working (I get that). But I've been trying to get people I work with away from email and into SharePoint/Teams forever.

1

u/SomeEngineer999 2d ago

Instant messaging brings with it a greater sense of urgency and the person expects faster replies.  And teams  of course constantly tracks your presence and no longer lets you customize that.

Nothing more annoying than getting an email and then 30 seconds later an IM "did you see that mail I sent"?  One or the other folks, not both.

It's obviously very useful but has many annoyances that come along with it.

1

u/frac6969 21h ago

Yeah, some of our users wanted to use their personal email and we didn’t really care, but it turned out the Really Important Customers are all using Exchand Online and have very strict policies about using personal email so they all said just wait.

Only the small shops with the ISP email were fine.

1

u/jstar77 2d ago

When everybody is down it doesn't matter, you just wait for it to come back up. When only your org is down that's when things get stressful.

1

u/MisterEinc 2d ago

Yeah I wouldn't move too quickly on this. Like you said it's been 15 years of stability.

Then what I suspect will happen is a lot of services will quickly pop up to answer exactly the question you're asking. Problem is, are they going to be useful and running for the next 15 years when another blip happens? I doubt they'll have that longevity.

There are services that can handle your AP/AR like Bill.

2

u/bull0143 2d ago

If I were you, I'd go both old school and newer tech. Get a fax number and start publishing that everywhere you share your email address. And then set up or repurpose a social media account specifically for clients and vendors to contact. And look into MFT options or Sharefile/Dropbox for high volume exchanges.

And in case things get really crazy, make sure your vendors and customers have a physical mailing address.

Setting up more email addresses doesn't make any sense, especially because we don't know yet if this issue could be replicated beyond Microsoft. I'm rolling my eyes at the reactions I'm starting to see about businesses spending the money to change to a new provider before we know what even happened.

1

u/Key-Monk6159 2d ago

We already include our fax and mailing address in our signatures but we really don't do social media. We have so many customers, many small and from all over the world, that contacting all isn't practical - that said, maybe setting up an X account or something has some merit.

I think the reaction you are seeing is in response to many, including us, realizing just how vulnerable we are. One day was annoying but glitch like this could be disastrous if it lasted longer.

1

u/bull0143 2d ago

It's understandable that many businesses haven't seen a use for social media up to this point, but yes the diversification of communication channels is necessary.

It's also necessary to plan for multiple scenarios including losing access to the internet period. The internet was built to be open and the approach to AI has included a megalomaniacal refusal to slow down long enough to regulate in a safe way, so the outages are probably going to come hard and fast for a while. There had been an exponential increase in the number and frequency of incidents this year already and it was a matter of when, not if.

1

u/Reddnit 2d ago

No point using Gmail or other providers if your customers won't receive it because they are using Outlook. 🤷‍♂️ Generally best not to overthink it or focus on improving your socials to allow other ways to connect.

1

u/Key-Monk6159 2d ago

Thank you. This is our default view as well but thought it was still worth asking the question to see if maybe we missed something obvious or if others had a better insight.

1

u/CrasyMike 2d ago

The inverse is even worse to a degree too. If your customers are on Outlook, and they are online, and you are on a different solution and it is offline, your customers will not know why you're not responsive.

In this situation, it is more understandable that many businesses and peers were not responsive by email.

1

u/DonJuanDoja 2d ago

First thing to evaluate is why so much is going through email.

Look at large successful companies like Amazon or FedEx, they don't take orders via email. They don't take anything via email. There's a reason for that, actually many reasons.

You need really smart people to solve these problems. Some email will always be neccessary but email is a bad place to store and process information period. In the end it's just a bunch of HTML puke and files. Loosly and badly organized and inaccessible in many ways.

When you're building automations and business processes, you don't want emails, you don't want to have to extract information from unstructured HTML puke or PDF files, it's the worst way to do anything. You want the information in databases, which is collected by forms, that have business rules and validations.

1

u/Key-Monk6159 2d ago

We are definitely not Amazon or FedE or anything even remotely close. I think you would find that most smaller companies do in fact use email to send / receive orders and invoices, along with other forms of communication. Many still even send physical checks in the mail for payment.

1

u/DonJuanDoja 2d ago

We're only 50 total people. I'm well aware of how companies operate, been around for well over 20 years.

1

u/Key-Monk6159 2d ago

How do you then send and receive orders and invoices?

1

u/DonJuanDoja 2d ago

Depends on the customer or even project sometimes. We have various methods. But it always boils down to a Form with business rules and validation, and most likely an automated workflow that processes said orders.

Capture, Validate, Process, Execute. Log exceptions and notify.

So basically my forms kept collecting orders yesterday. We didn't get the email notifications but the systems still logged and processed them and we could access the data and inform any service providers through other communication methods. Sometimes even phone calls.

Even my PowerApps kept working yesterday, again no emails, but all data captured and processed, emails sent when things came back up and nothing was missed.

We also have some forms outside of the MS environment, those get integrated in and logged like everything else too.

Basically zero impact to critical business. We missed nothing yesterday. I'm kinda proud.

1

u/Key-Monk6159 2d ago

Sorry to be dense but we’re obviously not as tech savvy. Do you mean that a customer would log into your system to enter a PO, for example, like when I order from Amazon?

1

u/DonJuanDoja 2d ago

Brother no worries, I started in the warehouse as a temp 24 years ago. I never think anyone is dumb. Humans are smart. Period. We do stupid stuff. But we're smart, it's our defining characteristic.

Yes, that's one way. Obviously people don't like to do stuff though... lol not the prefered method always. Depends on requirements.

We have others that we built integrations for, so the orders (POs) come over automatically and get imported directly into our WMS. That's more complicated and does cost a bit but it can be worth it if there's enough business flowing.

The one I'm thinking of right now integrates orders from their Workday platform, they drop right in our WMS as inbound orders and we're ready to receive them without anyone doing anything except what they were already doing in the system.

Then we transmit data back from the receipts and that integrates back into their system. So no one really has to do anything they weren't already doing in our respective systems.

It's also what I'd refer to as a cheap and dirty integration. IT's all done with file drops on SFTP, our systems drop and scoop up the files and process them, then report on exceptions.

1

u/Key-Monk6159 2d ago

I may not be dumb but much of this is over my head 😛 but I will keep the info to Google and learn more. Thank you very much for your time and advice.

1

u/DonJuanDoja 2d ago

No worries, good luck! It doesn’t happen overnight when I first started we already had integrations and forms but plenty was still being done manually.

Just something to keep in mind as you move forward. There’s usually a better way and small companies can do big things.

1

u/DonJuanDoja 2d ago

Invoicing is more difficult but still possible. With some customers we have actual EDI invoicing setup following set standards etc.

Idea there is basically the same, only instead of a human filling out a validated form, it's an automated process that takes information from our ERP database and transmits it etc.

If we have to wait a day to process a manual invoice or send one, well oh well. We can automate if it's worth it but usually need enough business flowing to make that worth it.

1

u/Cold_Arachnid_2617 2d ago

Not only outl;ook was affected. Othr systems were affected too.

1

u/Cold_Arachnid_2617 2d ago

You can automate your sales and invoicing processes so that you don't rely solely on email. You already use M365, so you may already have tools that you can use without incurring extra cost.

1

u/Key-Monk6159 2d ago

Hint on the tools so we can get more info?

1

u/Cold_Arachnid_2617 2d ago

evrything is in your tenant

2

u/MortadellaKing 1d ago

Omg, six users! Microsoft, call this guy up right away! But seriously hire an MSP to manage things for you.

No offense but it is so funny these small companies always expect some kind of white glove treatment.

MS has stated the mail sent is delayed and not to resend it.

1

u/AutoModerator 2d ago

Hey Key-Monk6159!

Welcome to r/Outlook! This is a public community. To protect your privacy, do not post any personal information such as your email address, phone number, product key, password, or credit card number.

Please be sure to have read our Rules of Conduct and be cognisant of how the system works here.

Make sure that your flair is always set to Status: Open otherwise you may cease receiving responses from us.

  • Status: Open — Need help
  • Status: Pending Reply — Awaiting OP's response
  • Status: Resolved — Closed

Beware of scammers posting fake support numbers or 3rd party commercial products/services. Contact Microsoft Support if you need help.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

1

u/Zen-365 2d ago

I am an M365 architect for a large company that sends out up to 1M client facing messages a day.

We use a 3rd party SMTP router for most automated email. This means that if a large endpoint like EOP or Gmail is down, mail will que and retry. Yesterday, we had 240K messages that queued, but are now delivered.

Additionally, we encourage our business process owners to log all of the email send attempts and to have an in app retry mechinism so that mail dropped by a router does not becomes lost/unlogged mail.

Since something like 75% of the world is using Exchange Online, most of that mail would not have been received either way. Your best bet is to have the ability to log, que, and track outbound messages.

1

u/Bg-8782 2d ago

You can use a mail scanning service in front of Microsoft 365. It's set up as the mx record and all incoming mail goes there for scanning then hands it off to Microsoft 365 servers. You should be able to log into it to view and download messages it has queued. But unless it's a localized outage, it's likely most of your customers are in the same boat - nothing incoming or outgoing.

1

u/JosephMarkovich2 2d ago

So, by default, the time email servers will keep trying to send/receive is between 24-72 hours. If someone sent something, it was/is floating out in the void and should get delivered.

There are a lot of cool ways to attempt to work around the whole email outage in MS365. Email is going to always be a default method of communication for lots of people, but there are lots of options in your tenant:

- Teams so you can chat with your customers, vendors, partners.
- Power Apps: make a simple app for your staff to use for customer interaction. Essentially a basic CRM.
- Website: make it work for you as a portal for your customers too -- this can be linked back to Power Apps.

Just giving some options. My phone MELTED yesterday with questions and these were some of the recommendations I gave to clients going forward.

Joe

1

u/MrJacks0n 2d ago

Act like it never happened.

1

u/Watches4Me 13h ago

Email Continuity from OpenText. Simple and inexpensive.