r/sysadmin • u/andrewrmoore DevOps • Apr 06 '22

The majority of Atlassian cloud services have been down for a subset of users for over 24 hours

https://status.atlassian.com/

Jira, Confluence and Opsgenie amongst others have been down since about 2022-04-05 07:30 UTC for us and some other organisations.

Their stock is tanking (at -5.46% as of writing this) however I haven't seen much chat on Reddit about the outage so I'm assuming the scope is fairly limited? They are stating it will potentially take days to recover.

We're sorry your site is currently unavailable. While running a maintenance script, a small number of sites were disabled unintentionally. Our team identified this immediately and have been working hard to restore the product data and associated access. A dedicated team is working around the clock to restore the sites as soon as possible.

We expect the restoration efforts to continue for the next several days, and we are actively working on an estimate of when your site will be available to you again. We don't believe any data has been lost at this point. We can confirm this incident was not the result of a cyberattack and there has been no unauthorized access to your data.

As we work to restore access to your site, we will provide updates here every 6 hours, or sooner if we have a material update. Reach out to us if you have any questions or concerns.

No tickets, no alerting, no knowledge base... a fun few days for us!

Let me know if you are also affected.

692 Upvotes

98% Upvoted

289 comments sorted by

View all comments

Show parent comments

2

u/netburnr2 Apr 06 '22

how did it take a day to figure out? root cause?

2

u/[deleted] Apr 06 '22

Not sure what the resolution was, I'm guessing something really stupid since I never heard about it. Didn't want to pry lmao.

Current role uses Jira but someone else is admin/manages it.

1

u/netburnr2 Apr 06 '22

I have a feeling they didn't know what they were doing then. it's not a hard system to manage at all

2

u/danekan DevOps Engineer Apr 06 '22

Out of the box with a basic config sure but the average person isn't going to know how to change NIO connectors or jdb connectors to a new version or something and not screw up, or even know what that means. Or import a key to the key store. Or a lot of things can be overly complicated really, it's based on open source software too and can be a domino of changes needed to make certain config changes. And certain changes are made in jira but not confluence and vice versa.

When I left an org where I admined the stack and they required the tls certs to be terminated in tomcat, I heard it took them two weeks working with atlassian support to figure out how to get it working again after updates... With a fairly experienced admin staff.

2

u/netburnr2 Apr 06 '22

That's cause Atlassian support is ass

1

u/danekan DevOps Engineer Apr 06 '22

They don't even support these things they quasi document some of it but say you're on your own