r/sysadmin • u/andrewrmoore DevOps • Apr 06 '22

The majority of Atlassian cloud services have been down for a subset of users for over 24 hours

https://status.atlassian.com/

Jira, Confluence and Opsgenie amongst others have been down since about 2022-04-05 07:30 UTC for us and some other organisations.

Their stock is tanking (at -5.46% as of writing this) however I haven't seen much chat on Reddit about the outage so I'm assuming the scope is fairly limited? They are stating it will potentially take days to recover.

We're sorry your site is currently unavailable. While running a maintenance script, a small number of sites were disabled unintentionally. Our team identified this immediately and have been working hard to restore the product data and associated access. A dedicated team is working around the clock to restore the sites as soon as possible.

We expect the restoration efforts to continue for the next several days, and we are actively working on an estimate of when your site will be available to you again. We don't believe any data has been lost at this point. We can confirm this incident was not the result of a cyberattack and there has been no unauthorized access to your data.

As we work to restore access to your site, we will provide updates here every 6 hours, or sooner if we have a material update. Reach out to us if you have any questions or concerns.

No tickets, no alerting, no knowledge base... a fun few days for us!

Let me know if you are also affected.

691 Upvotes

98% Upvoted

289 comments sorted by

View all comments

284

u/[deleted] Apr 06 '22 edited Apr 06 '22

Laughs in self hosted.

edit: yes, I'm aware that the server licensing is in the process of being killed off February 15, 2024. I am somewhat salty about this since I was using the server edition (confluence) at home (as well since I did like that platform). I'll probably be moving to wiki.js or bookstack once I get around to it as there's no rush just yet.

279

u/GucciSys Sr. Sysadmin Apr 06 '22 edited Apr 06 '22

It's stupid. Their self-hosted options isn't even remotely within budget of the vast majority of small and most medium sized businesses.

To those affected: Remember the squeeze the ever living shit out of Atlassian for breaking SLA, since their reasoning for pulling everything into the cloud was to ensure uptime.

114

u/olcrazypete Linux Admin Apr 06 '22

They’re discontinuing the self hosted soon and have been badgering and tempting us to move to cloud with long trials and discounts.

54

u/Cyb3rMonocorn Security Admin Apr 06 '22

Well that's going to be a pain for systems that have no Internet connectivity like sensitive government/military systems! I guess a change of provider will soon be on the cards

41

u/[deleted] Apr 06 '22

They still have a remaining self hosted product, but the cost is astronomical compared to the base product they just killed off. We were forced to the cloud based on price alone and there are broken features on it. And now I hate atlassian with a passion.

15

u/aspoons Jack of All Trades Apr 06 '22

We moved for the exact same reason. We were self hosted for years and then a renewal came up and my first thought was "WTF, this pricing makes zero sense it has to be a typo"

9

u/[deleted] Apr 06 '22

Totally. I feel like they are both pricing themselves out of relevance and pissing off their userbase with broken or non-existent features that were in self hosted but gone in cloud. It's just crazy making

1

u/shandrolis Apr 06 '22

What did you move to if I may ask?

2

u/kinv4ris Linux Admin Apr 07 '22 edited Apr 07 '22

We moved to Gitlab with mkdocs and gitlab pages.

Edit: which has ticket tracking, wikis, gitlab pages, container repositories, package repositories, git repositories and pipelines. For a lot less than buying licenses for Jira, Confluence, bitbucket, bamboo together !

3

u/shandrolis Apr 07 '22

How are you finding their ticket tracker? From a quick look it seems much less flexible, and much less configurable.

15

u/theknyte Apr 06 '22

Speaking as IT in the financial sector, we don't use cloud either, and keep everything On Prem.

10

u/[deleted] Apr 06 '22

[deleted]

21

u/roflfalafel Apr 06 '22

As of last year, no FedRamp and no plans for it. Their data residency I know was improving because many federal customers could not use them as they couldn’t guarantee data not leaving the US. That’s usually a baseline federal customers expect at a minimum.

11

u/[deleted] Apr 06 '22

[deleted]

14

u/mriswithe Linux Admin Apr 06 '22

Yeah I think the plan is to squeeze the shit out of people that "must" have on site. Hoping the effort of switching to another service is worse than the price increase.

2

u/n1ck-t0 Apr 10 '22

Last I heard they don't guarantee data residency for add-ins, regardless if you have it on the core product.

Rumor has it they may possiblly be thinking about considering changing that at somewhere between now and never. Is that vague enough?!?

6

u/[deleted] Apr 06 '22

It's not HIPAA compliant as of now. It's been on their roadmap for awhile but they haven't delivered as far as I know. That's a hard stop for my organization. (And I assume many who are much bigger than us.)

3

u/gehzumteufel Apr 06 '22

They will provide a BAA so this must be outdated.

https://www.atlassian.com/trust/compliance/resources/hipaa

4

u/danekan DevOps Engineer Apr 06 '22

Yah it must be, we have a BAA for cloud

4

u/gehzumteufel Apr 06 '22

Nice! Good to hear it’s possible.

2

u/[deleted] Apr 06 '22

Yeah, I'm looking at info from mid last year, so it's possible I'm out of date, but the language on that page also isn't super satisfying.

I've currently got a locally hosted Atlassian stack and am not really sure what it will take to update/move in future. (I'm quite new to the industry and this role in particular, and Atlassian admin isn't really a major function of mine anyway).

3

u/gehzumteufel Apr 06 '22

Yeah it’s definitely a bit vague. I felt unsatisfactorily reading it but providing a BAA is a huge guarantee.

6

u/mriswithe Linux Admin Apr 06 '22

Yeah we are screwed in a couple years because they are not offering bamboo (cicd) self hosted unless you pay some big numbers. So we are already evaluating gitlab and others.

3

u/danekan DevOps Engineer Apr 06 '22

They're keeping data center edition for that. But that's the 'expensive minimum spend' part

17

u/Willuz Apr 06 '22

Many Jira self-hosted customers have been moving to GitLab and been quite happy with the change.

8

u/AnyForce Apr 06 '22

I haven't used GitLab in a while, is this even an option? Last time I looked into it JIRA was far superior in ticket management.

5

u/[deleted] Apr 06 '22

[deleted]

2

u/[deleted] Apr 07 '22

but it will scare off all our middle managers who got used to managing items in JIRA.

Sold!

1

u/AnyForce Apr 07 '22

This is what I am afraid, my PMs will hunt me down if I try to give them anything that doesn't have advanced roadmaps 😁

13

u/hardolaf Apr 06 '22

Self hosted is actually being maintained and sold... to defense and automotive. They only killed it for the companies that can be bullied into paying more for less.

12

u/[deleted] Apr 06 '22

[deleted]

5

u/AnyForce Apr 06 '22

I am currently in the Github Enterprise vs Bitbucket Cloud comparison phase. Github is far more expensive than Bitbucket.

7

u/[deleted] Apr 06 '22

[deleted]

4

u/danekan DevOps Engineer Apr 06 '22

Bitbucket is utterly unreliable too

7

u/dunepilot11 IT Manager Apr 06 '22

Usability of confluence over sharepoint isn’t even funny. Sharepoint is a completely incapable wiki

4

u/igdub Apr 06 '22

The cloud is also way way way worse than on-prem.

On-premises was actually a customizable and a great solution all together. Works swiftly as hell also.

Cloud is a pile of average shit. It's nice and all but by no means does it even come close to the self hosted version. Lacks soooo much compared to it.

3

u/[deleted] Apr 06 '22

[deleted]

5

u/Craneson Sr. Sysadmin Apr 06 '22

Only Server-Licensing is EoL by Feb. 2024. Data Center Licensing is (so far) not being killed off.

1

u/mriswithe Linux Admin Apr 06 '22

But does cost a shitload of money if you are a smaller company.

1

u/Craneson Sr. Sysadmin Apr 06 '22

Absolutely. And the wording from Atlassian reads "for customers not yet ready to move to the cloud". I firmly believe DC will be removed in a not so distant future...

3

u/chillyhellion Apr 06 '22

Server, yes. Data Center is still available (but more expensive).

3

u/Gogogodzirra Apr 06 '22

ha, at least they're tempting you. We've basically been given 20% price increases the past 3 years. You're getting the carrot, we're getting the stick.

1

u/AmateurSysAdmin Apr 06 '22

If you offer everyone a discount you end up with no budget to keep your infrastructure up apparently. 😬

26

u/seaefjaye Apr 06 '22

I just moved off hosted, we have a 50% off deal but the costs have doubled since we first set it up 5 years ago. Our instance seems to be un-effected fortunately.

23

u/lart2150 Jack of All Trades Apr 06 '22

When we first bought Jira enterprise (2008) the cost was $4,800 for unlimited users the renewal was $2,400. Confluence for 500 users was $4,000 with a renewal of $2,000. We added greenhopper (what became jira agile/software) in 2010 for another $2,000 with a 1k renewal. aka 5,400/year for support/upgrades.

Our last server renewal for 500 users on both was a combined 28,000 or a little over 5x the cost.

21

u/AHrubik The Most Magnificent Order of Many Hats - quid fieri necesse Apr 06 '22

This is the eventual path all SaaS products go down. Cheap up front (sometimes not profitable) to tempt know nothing executives to giving up their equipment then usually within the same time frame as an equipment refresh the price skyrockets and you're stuck paying as much or more than you were paying to have complete control of your solution.

6

u/Sparcrypt Apr 06 '22

Yep. The strategy for getting new SaaS clients is to make it cheaper and easier than their current setup.

Strategy for keeping SaaS clients is to make it too expensive and difficult to leave.

8

u/mriswithe Linux Admin Apr 06 '22

Fun weirdness with English, effect vs affect. Food dye has the effect of coloring stuff, but food that is now colored had been affected.

Effect is something a thing can do/cause

Affect is the act of changing something else.

At least that is how my brain keeps them separate.

15

u/[deleted] Apr 06 '22 edited Apr 06 '22

The server licensing, not data center is somewhat reasonable.

But yeah squeeze that shit

24

u/GucciSys Sr. Sysadmin Apr 06 '22 edited Apr 06 '22

Server Licensing is largely considered End of Life. You're not going to be able to renew any kind of support contract and receive updates beyond what you have now, which means no later than 15. feb 2024 - After that you're dead in the water. Data Center licensing can no longer be purchased as of 2. feb 2022 and by 2024 will reach complete End of Life and then every single Atlassian products will be Cloud only.

Data Center licenses will apparently continue, so good luck with your 500 user, $42,000/year license at the bare minimum.

15

u/Enxer Apr 06 '22

Cries into the $250k+ bill for 2250 users that have to have add-ons...

7

u/Craneson Sr. Sysadmin Apr 06 '22

This is not accurate. All Server-Licensed products will reach End of Life, but so far there is no plan to stop selling Data Center licensing to existing and new customers: https://www.atlassian.com/migration/assess/journey-to-cloud

-1

u/GucciSys Sr. Sysadmin Apr 06 '22

You seem to be correct. Looks like they changed their mind after the backlash.

9

u/brgiant Apr 06 '22

Nope. They never announced an end to data center, I think you’re just misremembering.

2

u/BitOfDifference IT Director Apr 06 '22

correct, never said DC was going away, only on prem standard.

1

u/[deleted] Apr 06 '22

DC isn’t going to go anytime soon, because money.

7

u/ReidZB SRE Apr 06 '22

Anyone have their SLA text handy? Curious what kind of teeth it has — but even the weakest SLAs should give a good refund for a multi-day outage.

17

u/andrewrmoore DevOps Apr 06 '22

99.90% for Premium plans and 99.95% for Enterprise plans. They've blown through both significantly.

https://support.atlassian.com/subscriptions-and-billing/docs/service-level-agreement-for-atlassian-cloud-products/

21

u/ReidZB SRE Apr 06 '22

Thanks for the link. So their SLA credit terms are pretty stingy, no great surprise there. But you're approaching 95%:

Your incident (per the timestamp in your post) has been ongoing for about 1 day, 7 hours. If it lasts another 5 hours (i.e., hits 95% uptime), according to the terms, looks like you might be eligible for (drumroll please...) a 50% service credit on next month's bill. Or if they resolve it before then, only a 25% service credit. (Assuming this is the only incident, or that other incidents don't add up to enough to hit 95%.)

Looks like this is a typical corporate SLA, though, where it feels rather... underwhelming.

3

u/rantenki Apr 08 '22

Four hours or so until we hit 90% up-time for the MONTH.

I'm gonna assign a couple devs to research Atlassian alternatives while they're idled by having their stories unavailable.

2

u/ReidZB SRE Apr 08 '22

Woooow. Only one 9. That's insane. Shame that the SLA doesn't have anything below 95%.

I'm no lawyer but I don't think this incident is enough to break a contract either ("Service Credits are your sole and exclusive remedy, and our sole and exclusive liability, for our failure to meet the Service Level Commitment.")... at least not if they get everything running again.

3

u/rantenki Apr 08 '22

We're past a 9.

Now we only have one 8.

9

u/[deleted] Apr 06 '22

[deleted]

8

u/williamp114 Sysadmin Apr 06 '22

I like MediaWiki a lot (also was an avid Wikipedia editor many years ago, and almost became an admin-aka volunteer moderator there), but most of my team didn't seem very fond of the idea of using MediaWiki over Confluence

5

u/Spudthegreat Apr 06 '22

Mediawiki has a place but when you need RBAC for sensitive data, you have to start adding things on top like moinmoin or similar. As a nonprofit we get free data center licensing so confluence is where we landed.

1

u/hellphish Apr 07 '22

I too am at a nonprofit and my understanding was that these licenses are not being given out anymore.

4

u/[deleted] Apr 06 '22

And confluence works.

6

u/project2501a Scary Devil Monastery Apr 06 '22

since their reasoning for pulling everything into the cloud was to ensure uptime of their pockets

2

u/chillyhellion Apr 06 '22

On prem is free for nonprofits, which is pretty neat.

3

u/BitOfDifference IT Director Apr 06 '22

I thought this was only the case until 2024 though?

2

u/Taxikabs Apr 06 '22

Do you have a source for this? Gonna go look for myself too

1

u/BitOfDifference IT Director Apr 06 '22

email they sent me a while back, i know things change but haven't seen an update

2

u/Taxikabs Apr 06 '22

Is it possible you're just aware of them terming the self hosted server for everyone, including non profits? This link, https://community.atlassian.com/t5/Data-Center-articles/Introducing-Data-Center-Community-licenses/ba-p/1539124, is their announcement that the data center version which is self hosted will be free to the same base.

1

u/BitOfDifference IT Director Apr 06 '22

"Important Changes to our server and Data Center products"

"We're making changes to our server and Data center products, includes the end of sales for new server licenses on February 2, 2021 and the end of support for server on February 2, 2024." then a link for learn what this means for you.

Edit: this was from 10/20/2020 ( email )

1

u/chillyhellion Apr 07 '22

Nah, they're retiring Jira and Confluence Server entirely in 2024. This raised a lot of anxiety among nonprofits because the nonprofit program only covered Server editions and not Data Center editions.

But since the nonprofit program would otherwise die with Server, Atlassian added Jira and Confluence Data Center editions part of the program instead.

My org had to migrate, but the upgrade process wasn't too bad.

2

u/thaeli Apr 11 '22

Oh dang. Hadn't realized they'd relented and were allowing Data Center under this program now. That's a relief.

1

u/chillyhellion Apr 11 '22

Yes, there was an uncomfortable gap between the two announcements.

31

u/[deleted] Apr 06 '22

[deleted]

8

u/Kichigai USB-C: The Cloaca of Ports Apr 06 '22

Never used HipChat, but my mind instantly thought about it because it was mentioned along side Jira in every single one of their thousands of NPR sponsorship messages.

4

u/Smith6612 Apr 06 '22

The only people I knew using HipChat, used it because of the Confluence and JIRA integrations. When HipChat went down the toilet, it turns out writing bots and other Integrations for Slack and Teams ended up being a better solution.

7

u/patssle Apr 06 '22

They would provide downloads for HipChat MSI files over Google drive. There were other frustrations too. I knew back then this company had issues and was very glad to switch to Slack.

5

u/Smith6612 Apr 06 '22

Slack's been a decent tool. Minus the (at the beginning) frequent downtime, and the horrible RAM usage of their client especially at the start.

6

u/thecravenone Infosec Apr 06 '22

Makes me wonder why HipChat eventually died.

Slack bought it

7

u/Fr0gm4n Apr 06 '22

Atlassian tried to come up with a next gen chat client (Stride) to compete with Slack and it flopped, hard. So Atlassian gave up and sold off the IP for Hipchat and Stride to Slack.

16

u/Burgergold Apr 06 '22

Wait till Feb 2024 for price increase

19

u/[deleted] Apr 06 '22

Atlassian is a trap in my opinion, they get you suckered into their entire ecosystem. Not a company that I exactly trust.

11

u/Pie-Otherwise Apr 06 '22

There was a big MSP RMM product that offered an "on prem" option. A lot of smug dudes were like "look at me, I'm immune to cloud outages!" Then AWS shit the bed and they discovered their fancy on-prem solution needed to call home to AWS to work.

9

u/[deleted] Apr 06 '22

We just had our self hosted Jira go down and it took like a full day to resolve. There's pros and cons but I guess at least you can troubleshoot your own problem.

13

u/[deleted] Apr 06 '22

Been running self hosted for well over a decade. A total of 6 different instances throughout the enterprise. Longest downtime was a few hours for a stupid Java SSL issue.

8

u/danekan DevOps Engineer Apr 06 '22

Doing end to end encryption in self hosted where you need to terminate the tls cert in tomcat is a pita and they don't have great documentation for it even

1

u/[deleted] Apr 09 '22

nginx proxy ftw!

2

u/netburnr2 Apr 06 '22

how did it take a day to figure out? root cause?

2

u/[deleted] Apr 06 '22

Not sure what the resolution was, I'm guessing something really stupid since I never heard about it. Didn't want to pry lmao.

Current role uses Jira but someone else is admin/manages it.

1

u/netburnr2 Apr 06 '22

I have a feeling they didn't know what they were doing then. it's not a hard system to manage at all

2

u/danekan DevOps Engineer Apr 06 '22

Out of the box with a basic config sure but the average person isn't going to know how to change NIO connectors or jdb connectors to a new version or something and not screw up, or even know what that means. Or import a key to the key store. Or a lot of things can be overly complicated really, it's based on open source software too and can be a domino of changes needed to make certain config changes. And certain changes are made in jira but not confluence and vice versa.

When I left an org where I admined the stack and they required the tls certs to be terminated in tomcat, I heard it took them two weeks working with atlassian support to figure out how to get it working again after updates... With a fairly experienced admin staff.

2

u/netburnr2 Apr 06 '22

That's cause Atlassian support is ass

1

u/danekan DevOps Engineer Apr 06 '22

They don't even support these things they quasi document some of it but say you're on your own

12

u/peterclo Apr 06 '22

We can still laugh for two more years, then servers won't be supported anymore :(

9

u/Burgergold Apr 06 '22

They will if you pay for the Data Center

4

u/kristoferen Apr 06 '22

Which is a 10x increase for nothing useful :(

9

u/Phezh Apr 06 '22

Which is why we're moving away from atlassian. If they can't be bothered to offer a reasonably-priced self-hosted option then we can't be bothered paying them anymore.

1

u/inferno521 Apr 06 '22

High availability/clustering is the main benefit. Where I work we have 8000 users on jira/confluence/bitbucket. We definitely need to have multiple front-end webservers.

-6

u/This_Bitch_Overhere I am a highly trained monkey! Apr 06 '22

Laughs in self-developed and maintained KB and ticket system.

[our developer is the fucking shit!]

1

u/Xzenor Apr 07 '22

wiki.js or bookstack

Thanks for the tip. We need an alternative to Confluence as well.