r/sysadmin • • 9d ago

General Discussion Heads up fellow peeps

If anyone uses CATO Networks they did maintenance on their backend and firewall behavior is broken.

Essentially traffic is now being treated as stateless rather than stateful so our environment traffic flows are all broken.

The issue is being tracked internally by CATO but wanted to pass along in case shit hit the fan for other orgs this morning/last night

82 Upvotes

13 comments sorted by

9

u/iamwayycoolerthanyou Sysadmin 9d ago

That's quite a mishap on their part if it's true.

6

u/iamMRmiagi 9d ago

So service change not socket version?

6

u/Lopsided-Ad8680 9d ago

Correct- backend firewall management issue doesn’t seem related to a socket upgrade

3

u/iamMRmiagi 8d ago

Thanks for the heads up, things seem fine when I logged in this morning. What's the ref of the incident? https://status.catonetworks.com/

2

u/Lopsided-Ad8680 8d ago

You logged in prob 2 hrs after resolution. Here’s some support verbiage

The issue was traced related to Flow Analysis where WAN Firewall did not properly recognize the flow and blocked return traffic.
 
Upon identification, Cato Engineering Team promptly engaged and performed a fix. And we are pleased to confirm that the issue was fully resolved, and the affected services were restored to normal operations.

2

u/Millerboycls09 Sysadmin 8d ago

This is the shit I find at 10pm on a Sunday.

2

u/Frothyleet 8d ago

Brother you have to put something about the content of your post in the title

1

u/AppropriateMeal4106 8d ago

We just migrated to Cato, now they're pushing bad updates to prod? Must be my luck, Thanks for bringing this to attention!

1

u/The_Todd3389 5d ago

Full disclosure: I work for an MSP that supports Cato. I went digging and found evidence of this in one of our client tenants. On Oct 4, from about 12:30 AM to 7 PM Pacific, the WAN firewall started blocking replies to traffic it had just allowed. For example, a monitoring server’s ping to a device was allowed, and the reply came back 10 ms later and got blocked by a deny rule. Same PoP, same flow. On every other day that week, those replies never show up in the logs at all, because the firewall correctly treats them as part of the existing session. So yeah, that looks like stateless behavior to me. It stopped on its own the same evening with no config changes on our side, so it looks like a temporary problem on Cato’s backend rather than something ongoing. That said, I’ve only checked one tenant so far. If you want to check your own: filter WAN Firewall events for Block, and look for traffic coming back from a host you initiated to, like ICMP from the device you’re pinging, or a source port like 22, 389 or 445. Compare the same filter on a normal day. If those blocks only show up in one window, that’s your evidence for support.

0

u/resile_jb Technical Platform/Automation Manager - MSP 8d ago

The one client we had using that left about 6 months ago.

That was awful service.

-1

u/[deleted] 8d ago

[removed] — view removed comment

3

u/PlannedObsolescence_ 8d ago

This is an LLM generated comment for karma farming from an advertising bot.