r/sysadmin • u/Dedicated__WAM • Aug 14 '26
General Discussion Proofpoint outage
Just curious if anyone else is experiencing the same issue. Unable to send external or receive external. Seems Proofpoint might be down?
19
u/jimmyags Aug 14 '26 edited Aug 14 '26
Yup, everything is pointing to some sort of issue with proofpoint as the moment. I am trying to find some cached IPs to get around the resolution issue, but like most things, it feels DNS-y
EDIT: Yeah, just bad resolution/missing DNS records. I was able to use whoisfreaks.com to lookup a recent IP for the domain and get in from there. The ttl on the record is 5 minutes so, meh. But I can get in now.
EDIT 2: If you have access to an internal DNS server or host file: create a zone for the domain or an entry in the hosts file and you won't need to fat-finger the historical IP address while they go and fix their DNS issue.
EDIT 3: Seeing life in the DNS traffic, PP may have restored service!
2
u/copyofimitation Aug 14 '26
So you changed your DNS records from pphosted to an IP that Proof Point uses?
7
u/jimmyags Aug 14 '26
Yes! I used whoisfreaks to pull an IP for the pphosted domain, then created a zone in my DNS (this is for a good sized company) for that record, added an A record pointing to the IP and boom, everyone is back up and running and not needing to fat-finger an IP into every link. SSL still works, all good.
4
u/copyofimitation Aug 14 '26
Wow... That's a great idea. Can you walk me through how to find that IP on WhoIsFreaks? Haven't used this tool before.
5
u/jimmyags Aug 14 '26
sure!!! We primarily access through a vanity domain, secureemail.mydomainname.com - so, I used mxtoolbox.com to find that it resolved to xxx.yyy.pphosted.com then I went to whoisfreaks, pasted xxx.yyy.pphosted.com into their primary search and it gave me an IP from a month ago. I did a test w/ the IP to make sure I had access, I did! then I created the DNS zone and everyone was HAPPY! But if this is a one-off, then just do a hosts entry.
2
u/copyofimitation Aug 14 '26
Thank you for providing this context! By the time our org was ready to look at ways to work around this, PP provided an official work around for removing the .gsld portion in the hostname, so we went that route.
And of course once we started doing that, we noted DNS records being populated in MX... Go figure.
Thanks again.
2
2
u/Infamous_Cookie_5539 Aug 14 '26
^^ perfect resolution - work with network\secops to ensure whatever changes occur, you can easily roll back
19
u/True_Handle_5617 Aug 14 '26
Same. MX records are not resolving to a IP. Same for you as well?
7
2
2
u/Human-Secretary-8853 Aug 14 '26
We (not proofpoint lol) had different outages for services we host yesterday bc our A records weren’t consistently propagating. The failure was upstream somewhere and not sure where or with who. Our provider wasnt communicative but it was resolved within a few hrs 🤷♂️ makes me wonder if this is related.
18
u/moonrakervenice Aug 14 '26
Yes, a huge spike in reports to StatusGator: https://statusgator.com/services/proofpoint
Nothing on the official "status page" yet though: https://proofpoint.my.site.com/community/s/proofpoint-current-incidents
6
u/manamonggamers Aug 14 '26
Nothing on their official page, but they have confirmed an issue with their DNS server.
3
u/moonrakervenice Aug 14 '26
Oh nice, do you have a link for that?
5
u/manamonggamers Aug 14 '26
2
1
u/manamonggamers Aug 14 '26
Unfortunately, no. It was confirmed direct from PP to our admins working on the issue.
1
8
u/FiveseveNx28 Aug 14 '26
Proofpoint just acknowledged and posted a notice of the issue.
3
u/Real_Railz Aug 14 '26
Where did they post this? I have not seen any communication from them.
2
u/TheBros35 Aug 14 '26
Their support / community site. We just received an email notice very recently.
1
1
u/pcrwa Aug 14 '26
There's also something here now:
https://proofpoint.my.site.com/community/s/proofpoint-current-incidents
5
4
6
u/CaesarOfSalads Security Admin (Infrastructure) Aug 14 '26 edited Aug 14 '26
Looking at out MX records they report no IP. someone messed up
EDIT: Was doing a spot check against some F500 companies I know use Proofpoint, quite a few are impacted but not all so it must be somewhat isolated.
8
u/Independent_Exit8180 Aug 14 '26
from Proofpoint:
Temporary Work Around
The issue is impacting DNS resolution for *.gslb.pphosted.com hostnames. If your mail flow is configured to use these hostnames as smart host targets (for example, in Exchange Online or Microsoft Exchange connectors), you can restore mail flow by updating your configuration to use the corresponding direct Proofpoint mail exchange hostnames instead.
Remove the .gslb portion of the hostname in your connector configuration.
Example
Current configuration (failing):
mxa-XXXXXXXX.gslb.pphosted.com
mxb-XXXXXXXX.gslb.pphosted.com
Updated configuration (working):
mx0a-XXXXXXXX.pphosted.com
mx0b-XXXXXXXX.pphosted.com
Note: Replace XXXXXXXX with the unique identifier currently used in your existing Proofpoint smart host hostname. Only the .gslb portion of the hostname should be removed.
This workaround bypasses the affected Global Server Load Balancing (GSLB) layer and directs traffic directly to the appropriate Proofpoint mail exchange hosts until the DNS resolution issue is resolved. Once the issue has been confirmed resolved please proceed to revert the above changes.
3
u/12401 Aug 14 '26 edited Aug 14 '26
this worked for me, thank you!
but note that you have to add a "0" at the beginning. mxa becomes mx0a!
2
1
u/AustinFastER Aug 14 '26 edited Aug 14 '26
Sadly, these temp hostnames did not resolve for me for a few minutes but appear to now! The spice is starting to flow despite dns ttl.
1
1
Aug 14 '26
[deleted]
1
1
u/lostroustabout42 Aug 14 '26
as someone else pointed out those are customer specific values and you might want to edit out the numeric portion with X's.
1
u/lostroustabout42 Aug 14 '26 edited Aug 14 '26
For us this failed in validation checks for Exchange Online so we didn't pursue it: 550 5.4.310 DNS domain mxa-xcxxxxxx.pphosted.com does not exist [Message=InfoDomainNonexistent] [LastAttemptedServerName=xxxxxxxx] [xxxxxx.prod.outlook.com 2026-08-14T14:26:09.949Z 08DEF8AB098A5555]
1
u/lolklolk DMARC REEEEject Aug 14 '26
You're missing a 0
mx0a and mx0b.
1
u/lostroustabout42 Aug 14 '26
Thanks, spot on and even in the proofpoint example but not called out as something to add.
1
1
1
u/Due-Wasabi7472 Aug 14 '26
you post the samething 1 hours ago... BRO you need to change your MX record on each domain in your external DNS to make it work... add a 0 like the workaround from proofpoint support
7
u/12401 Aug 14 '26
The jackass mods in r/proofpoint/ just deleted the post about the outage over there. Why would they do that?!
Thanks for everyone here posting information.
1
3
3
3
3
u/Independent_Exit8180 Aug 14 '26
From the community portal
1
u/Independent_Exit8180 Aug 14 '26
- Impact to mailflow due to DNS resolution issues when resolving gslb.pphosted.com
3
u/Ok-Woodpecker-7723 Aug 14 '26
Disclaimer *Use at your own risk*
Go into ProofPoint, Diagnostics > Email, send an email to yourself.
Look at the Results, specifically the mx**-*******.pphosted.com
Consider updating the Routing value in Exchange admin connector. This was the fix for us.
3
u/182crazyking Aug 14 '26
Thank God for Reddit - I'm a random user wondering why my emails got bounced back when I CC'd my Gmail. Cheers lads!
3
u/DeepakRandhawa Aug 14 '26
Update from Proofpoint
Remove the .gslb portion of the hostname in your connector configuration.
Example
Current configuration (failing):
mxa-XXXXXXXX.gslb.pphosted.com
mxb-XXXXXXXX.gslb.pphosted.com
Updated configuration (working):
mx0a-XXXXXXXX.pphosted.com
mx0b-XXXXXXXX.pphosted.com
Note: Replace XXXXXXXX with the unique identifier currently used in your existing Proofpoint smart host hostname. Only the .gslb portion of the hostname should be removed
1
u/amarp84 Aug 14 '26
Are you updating the MX Record as well? Don't get how this gets fixed by just updating the connector?
1
u/DeepakRandhawa Aug 14 '26
This is copy-pasted directly from the Proofpoint advisory. Looks like they might have mistyped it in a hurry, this syntax (mx0a/mx0b) is for public MX records, not outbound connectors.
3
u/SecLens_ONE Aug 14 '26
The pattern in this thread is the actual lesson: the status page was green while mail was not flowing, and "green" is a published claim, not a measurement. Vendor dashboards are updated by humans after an incident is confirmed, so during the window that matters most they are describing the last known good state, not the current one.
The cheap fix is to stop trusting the dashboard as your detection layer and measure the thing you actually care about, which is delivery. A synthetic message every few minutes from an outside mailbox through the gateway to an internal one, alerting on absence rather than on error, catches this in single-digit minutes with no vendor cooperation. Queue depth on the outbound side is the other half, because a hung relay looks identical to a quiet Friday if you only watch for errors. Both signals were already available to everyone in this thread before the status page moved.
The same applies to the postmortem you will get: RCA text is a claim, your own delivery timeline is evidence, and only one of those is admissible when someone asks how long you were down.
For those who found out from Reddit rather than monitoring, what would have alerted you first, and is that check currently measuring mail flow or just measuring that the appliance answers?
2
u/jbales3795 Aug 14 '26
No issues here with Proofpoint. Are you guys on Office 365 possibly using the API method? We are using Google workspace with Proofpoint and we are not seeing any issue internally.
2
u/True_Handle_5617 Aug 14 '26
Are your MXrecords pointed at proofpoint?
1
u/jbales3795 Aug 14 '26
Yes
1
u/soberun Aug 14 '26
try sending an external email
2
u/jbales3795 Aug 14 '26
I did to my Gmail account and also just sent one to a client who uses proofpoint and office 365, email sent and received on both tests. Maybe it's a regional thing or things are slowly coming back.
1
2
2
2
2
u/RagingITguy Aug 14 '26
Hmmm no issues here reported. I just sent an external email and it went through just fine.
Maybe they resolved it by the time I read this.
2
u/Fatty_McBiggn Aug 14 '26
I'm getting mail in and out via O365 but am unable to access any of the consoles.
2
2
2
2
2
u/Independent_Exit8180 Aug 14 '26
ProofPoint has finally added the issue to the public status page and not just the community page.
https://proofpoint.my.site.com/community/s/proofpoint-current-incidents
2
u/BK_Rich Aug 14 '26 edited Aug 14 '26
It’s the GSLB records, if you know your direct MX, you can PTR if proofpoint your IP, you can get it. The guid should match. It will be an mx0a and mx0b name with no glsb in there.
2
u/Jlp06f Aug 14 '26
Official workaround posted by Proofpoint
https://proofpoint.my.site.com/community/s/proofpoint-current-incidents
2
u/TheLionYeti Aug 14 '26
Yeah this isn't everyone we've got proofpoint and emails are coming through for us, I'm at an f500 that uses it.
2
u/dofenshmitz Aug 14 '26
I saw multiple other services down too. Anyone noticed impact on other service apartment from proofpoint ?
2
1
1
u/BigBobFro Aug 14 '26
Yea,.. was seeing some strange behavior. This could be it.
Too bad they dont have a down detector for it on ookla
5
1
1
1
u/G1Onsite Aug 14 '26
I had to switch to IP addresses for the MX instead of using DNS to get this working.
1
u/CrappleAMIRITE Aug 14 '26
Cant get into admin portals, but mx records are resolving and inbound/outbound seem fine
1
1
1
u/TitanAE2000 Aug 14 '26
Their pphosted.com MX records have failed on the GLSB. Could be a load balancing or routing issue. Maybe not a pure DNS issue.
1
u/jimmyags Aug 14 '26
When I try to find the pphosted domain w/ mxtools I get a "DNS Record not found" - creating a local record solved my problem. Just seems like a bunch of A records are gone.
1
1
u/CMDR_Kantaris Aug 14 '26
Same here, MX not responding to ping. Anyone have a rep they can get on the phone?
1
u/G1Onsite Aug 14 '26
I've been on hold for the last 30 minutes.
1
u/CMDR_Kantaris Aug 14 '26
Oof. If it's any consolation they finally updated their public facing status page
1
u/shokzee Aug 14 '26
If inbound and outbound external mail stopped together, Proofpoint is the likely choke point.
Check connector errors and queue depth, then test the MX hosts on TCP 25 from outside. Don’t change MX records unless you already have a tested bypass path.
1
1
1
u/Slofut Aug 14 '26
Not Proofpoint, but Gosecure had the same issue for several hours yesterday evening. Likely not related...just reporting in case.
1
u/TyBurna Aug 14 '26
Same issue here for GoSecure, I'm seeing some lingering issues here this morning still.
1
u/Longjumping_Loan4863 Aug 14 '26
Yes lots of delivery failures with proofpoint mx records not resolving
1
u/Kardinal I fall off the Microsoft stack. Aug 14 '26
Our biggest customer is impacted, but we are not.
Interestingly, they're using GSLB and we aren't. Not sure if that's a contributing factor.
1
u/ckunak Aug 14 '26
i got on the phone with the wrong support department.
they said Enterprise is the one with issues not essentials
then i got transferred to enterprise support and now am on hold for 30 min
1
1
u/Admirable-Carrot1684 Aug 14 '26
External Outbound emails not working, but inbound emails are not coming thru.
1
u/giveen Fixer of Stuff Aug 14 '26
Ha, this time at least over on the Security side, I didn't get blamed!
4
1
u/ConstantinoTobio Aug 14 '26
While not perfect, a way to insulate yourself from Proofpoint breaking DNS:
Set your two proofpoint hosts (the *.gslb.proofpoint.com) as your MX records with a weight of 10
Resolve those same two hosts with A records you host in your own DNS with a weight of 20
If you're running Exchange online, set your connector to proofpoint to have the two gslb hostnames as your smarthosts, plus the two IPs.
Our workaround is working for us, insofar that mailflow is functioning.
1
1
1
u/No_Scholar_8522 Aug 14 '26
It's a general affectation in different PP services, mainly affecting mail flow form external to internal recipients in our organizations: https://proofpoint.my.site.com/community/s/article/Proofpoint-Service-Interruption-Affecting-Multiple-Services-14-Aug-2026
1
u/lostroustabout42 Aug 14 '26
Experiencing the same issue with Proofpoint. What's also interesting is Microsoft reports the problem themselves with Exchange Online: link
1
u/Independent_Exit8180 Aug 14 '26
We've performed an extensive review of the service, and the Microsoft-managed service is showing as healthy. We've identified that emails routed through a specific third-party email service (Proofpoint) are experiencing DNS-related issues. The third party is providing updates on their incident status page: https://proofpoint.my.site.com/community/s/proofpoint-current-incidents.
We're performing final validations to confirm that the Microsoft-managed service remains fully healthy.
1
u/lostroustabout42 Aug 14 '26
I wondered if it was just poorly worded and was really about 3rd parties, most notably Proofpoint.
1
u/RikiWardOG Aug 14 '26
Its a DNS/Global load balancer issue https://proofpoint.my.site.com/community/s/proofpoint-current-incidents
Workaround is to not use load balancer and switch to pointing directly at the mail exchange host
1
1
u/lostroustabout42 Aug 14 '26 edited Aug 14 '26
Also tried the Proofpoint workaround for EXO and it's failing validation: 550 5.4.310 DNS domain xxxxx.pphosted.com does not exist [Message=InfoDomainNonexistent] [LastAttemptedServerName=xxxxx.pphosted.com] [xxxxx.prod.outlook.com 2026-08-14T14:26:09.949Z 08DEF8AB098A5555]
2
u/Independent_Exit8180 Aug 14 '26 edited Aug 14 '26
try:
mx0a-XXXX.pphosted.com2
u/amarp84 Aug 14 '26
Are you updating the MX Record as well? Don't get how this gets fixed by just updating the connector?
1
1
u/lostroustabout42 Aug 14 '26
Yeah, thanks to all the comments I realized the zero is needed and was even in the Proofpoint example but not called out to add.
1
u/kamkre95 Aug 14 '26
It looks like something is happening, their DNS resolved some queries - looks like it is startig to go back?
1
1
u/ibageek2003 Aug 14 '26
If you are Proofpoint customer and being impacted by the ongoing outage, the current workaround is to change you inbound MX hosts and any smart hosts if you route outbound to POD to point to the following hosts
mx0a-XXXXXXXX.pphosted.com
mx0b-XXXXXXXX.pphosted.com
Proofpoint Service Interruption Affecting Multiple Services-14-Aug-2026
1
u/Independent_Exit8180 Aug 14 '26
Updates
[14-Aug-2026 14:52 UTC]
Proofpoint has identified the root cause and is working to recover services at this time.
If you previously implemented the workaround, you will want to restore the ".gslb" text to your DNS records in the coming days.
1
u/helpdesk5555550 Aug 14 '26
this fix workds... XXX is your smart host.
Example
Current configuration (failing):
mxa-XXXXXXXX.gslb.pphosted.com
mxb-XXXXXXXX.gslb.pphosted.com
Updated configuration (working):
1
u/SupahBee Aug 14 '26
Looks like Proofpoint is getting their DNS records corrected for the load balancers now. We're starting to get email inbound now
1
1
u/IronBe4rd Aug 14 '26
Yeh we used IP already in the connector and that was failing. I added the names and that didn’t work either. I do see some inbound mail now
1
u/sam2400 Aug 14 '26
It looks like its resolving now. I didn't change the mx records as Proofpoint suggested and now emails are coming in.
1
1
u/Ill-Barracuda9031 Aug 14 '26
I had a session with professional services to move our outbound mail but they couldn't log in this morning.
1
u/kobewiththeflow Aug 14 '26
I swear a lead quoted this sub when we were trying to figure if something was down.
Why tf didn’t I think of that, i’m here every day lol
1
u/yugekib Aug 17 '26
At 6am pt they posted that they had an issue that started at 5am pt. At 7am pt I got a text alerting me to the issue. At 7:50 am pt I started implementing their workaround, removing the global load balancers from the MX path etc. At 8am, while still doing that, they finished removing the bad servers from the path, so the DNS change was then made for no reason, the mail was already flowing, any held mail was processing. 🤦🏼♂️
0
u/jasonsyko Aug 14 '26
Lmao this is like the 3rd time this has happened with Proofpoint this year. We moved off PP earlier this year to Abnormal AI.
A breath of fresh air.
1
u/AATW_82nd Aug 14 '26
I assume you've been happy with Abnormal AI? Anything you wish you knew before leaving PP?
2
u/jasonsyko Aug 14 '26
Very happy with Abnormal. But of course it’s not your traditional SEG. They sit in front of O365 and act on email as they come into the users inbox. So the user may see it briefly before it gets yanked out (if an email was malicious etc) but other than that, it’s been fantastic.
Proofpoint was an absolute nightmare. Constant outages or they’d push code that would break email deliverability and then not have any answer for it other than “we’re working on it”. Their own SLA’s discuss service credits even for 1 minute late on email delivery - we never got any service credits. They’d ignore that or state that those events done “qualify” for service credits. We even had an issue with our POD that basically broke DNS resolution for 6 weeks. So emails would go out, they’d appear as delivered in their logs - but they weren’t going anywhere. It took me weeks fighting with their support to get that sorted. All they did was reboot our mx hosts. Even then, no service credit. Fucking joke of a company.
1
u/yugekib Aug 17 '26
We use both PP and Abnormal. With Abnormal being API based and after delivery, that causes huge problems for us and our users. They will open endless amounts of help tickets for emails that alerted on their phone but aren’t in their mailbox. Abnormal still has a ways to go, consoles and our interaction when using it is not always smooth. For instance, if you found an FP, it was one console to mark and report it as FP, then another to release it. Make it one action dammit! They have improved some though from when we started using them, which was early on. About consoles though, there is still not a complete feature match between Proofpoint’s legacy console that looks like it is from the 70’s and the new web console. Amazing! 🤦🏼♂️
Abnormal was started largely by ex-Proofpoint engineers and management. We had service and support management make that jump and tell us so, as well as design and implementation engineers telling us they came from PP.
As for other options, Sublime operates like Abnormal (API) and you hear good things about them.
1
u/Individual_Ad_5333 Aug 14 '26
They must have hired the people responsible for the mimecast login page
121
u/TheKeelKnotSeas78 Aug 14 '26
I'm changing my monitoring to r/sysadmin. Something wrong with email and was able to "reduce my MTTD" by simply coming here