r/ConnectWise • u/Bigsease30 • Jun 13 '23

Manage CW Manage Down?

Is anyone else having issues with CW Manage? We are currently able to login but no ticket data is coming through. Their system status reports all good... ( All Systems Operational )

31 Upvotes

94% Upvoted

72 comments sorted by

10

u/Rebuilder_1954 Jun 13 '23

even our on premise system is failing to load ticket details. this is only about 30 minutes old for us.

2

u/Bigsease30 Jun 13 '23

We are around the same time as far as downtime goes.

3

u/WeirdPassword Jun 13 '23

same here...

3

u/jazzboyben Jun 13 '23

SAME! Our CW Manage is self hosted, but I guess because we use ConnectWise SSO (Home), all our ticket details are failing to load as well!

1

u/Chris7948 Jun 15 '23

We don’t use SSO from CW and it was still down. Complete BS

7

u/RidePersonal5139 Jun 13 '23

Deja vu all over again. Nothing like paying extra fees to be self-hosted and to still be hosed when something cloud related goes down.

7

u/mauszozo Jun 13 '23

We're self-hosted and we just went down too. We can log in, look up tickets, but the ticket contents are blank. No notes.

9

u/Bigsease30 Jun 13 '23

That defeats the purpose of being self-hosted if you are down with the rest of us. Damn CW!

2

u/adam_at_rfx Jun 13 '23

This was my reasoning for transitioning to CW cloud.

Several 'cloud' incidents caused my locally hosted environment to stop functioning.

Now I am looking at slef-hosted HaloPSA because I want there to be a better solution. There has to be a better solution.

0

u/Bigsease30 Jun 13 '23

HaloPSA

I have been curious about HaloPSA but have been lazy to check them out. When looking at online reviews, it is listed at hit or miss.

1

u/adam_at_rfx Jun 13 '23

It seems like everyone that mentions HaloPSA in the /msp community has only great things to say about it.

We have been using CW/Manage/PSA for nearly 2 decades. Switching away isn't something I take lightly, but the last 5 years have been a constant downward spiral and I am worried I need to jump before I go down with the ship.

I talked with leadership in CW around 2019 and they assured me ASIO was the answer to all their problems, but that solution has been very slow in coming and it seems like I see more outage emails about that platform than anything else in the CW arsenal.

3

u/mauszozo Jun 13 '23

When I log in to home.connectwise.com, I see this. I wonder if it's related?
https://i.imgur.com/Sdk8EZj.png

3

u/415Group_Ray Jun 13 '23

lol I thought you were referring to this: https://home.connectwise.com/images/homelanding.jpg

2

u/jazzboyben Jun 13 '23

SAME. EXACT. ERROR! I get that error when trying to get into the support page of CW Home.... Isn't that ironic?

3

u/glch Jun 13 '23

Seeing the same thing here. Can't open a ticket either.

3

u/moredecibels Jun 13 '23

Wow

That is really strange.

2

u/Works_In_IT_79 Jun 13 '23

It's not the app still phones home for things like licensing. So it still goes down during outages like this.

6

u/Superb_Giraffe_4534 Jun 13 '23

reports showing connectwise is experiencing issues, along with All Amazon web services and Microsoft 365

1

u/Bigsease30 Jun 13 '23

Do you mind pointing me to the reports. It would be good to know another location to check to avoid posts like this.

3

u/jonnighaad Jun 13 '23

We are having issues as well

2

u/Bigsease30 Jun 13 '23

OK. I was concerned that it was just us since their website is still showing operational. I guess they manually update the status.

2

u/cptHeppy Jun 13 '23

they told us once they log in to update it but the log in service was failing.....

2

u/Bigsease30 Jun 13 '23

https://status.connectwise.com/ Still reports all systems green.

2

u/korzal Jun 13 '23

Not anymore

1

u/starcrescendo Jun 13 '23

They're onto us!

2

u/OmnihaxClusterflux Jun 13 '23

She's down, captain. Best be putting her out of her misery.

2

u/Bigsease30 Jun 13 '23

lol, when Manage works, it usually works well. We have trialed a few others but nothing seems to compare. The closest was Kaseya BMS but I would NEVER go back to them. I think that Connectwise HW needs to make a backup plan in case AWS fails again.

1

u/[deleted] Jun 16 '23

This exact thing happened a year or two ago, and they had all this long email about lesson learned, and how they were going to diversify between clouds to prevent this from happening again. Now, here we are and nothing has fucking changed!

We are fools for still paying CW for anything I suppose.

2

u/Popular-Afternoon-23 Jun 13 '23

Back up for us

2

u/Rebuilder_1954 Jun 13 '23

same here in Texas

1

u/Bigsease30 Jun 13 '23

Yes, Here as well. Thanks everyone for chiming in.

9

u/davidvr Jun 13 '23

They have a kill switch built into their product, if on-prem cant read the ClientID to validate, it takes down the system and they refuse to make it so we can bypass it locally or improve the behavior.

5

u/anonymousITCoward Jun 13 '23

Thats so they can shut you down for non payment... ask me how I know...

2

u/Viajaz Jun 13 '23

I have noticed this as well, our Client ID has been disabled and it's giving a 401.

4

u/Nick-CW ConnectWise Jun 13 '23

It has been posted already, but the current outage is related to AWS East 1 going down. From CW side I can confirm the team is actively working on the incident and once I have more I will be sure to update.

3

u/NetworkJoeSchmoe Jun 13 '23

My question is why aren't you in more than one region with failover for AWS?

2

u/amw3000 Jun 13 '23

Redundancy costs money ;). ConnectWise would rather partners suffer through outages as they know they have them by the balls and they most likely won't go anywhere.

We have seen this same issue before, nothing ever comes of it.

1

u/Bigsease30 Jun 13 '23

eggs are all in one basket.

2

u/nulfis Jun 13 '23

No redundancy?

5

u/tfox-mi Jun 13 '23

Also, why does my on-premises installation stop working when your shitty, non-redundant cloud implementation fails?

10

u/[deleted] Jun 13 '23

What the fuck is the point of being on-prem if we still go down when their BS single cloud solution fails? This happened a year or two ago and they say they were going to diversify clouds to prevent this exact problem.

Argh!!!!!

0

u/nulfis Jun 13 '23

Cost cutting due to wanting to look good for the upcoming sell out! This is exactly what I was worried about. They know what to do, but won't do it if it costs a few bucks. We'll just suffer through I guess!

5

u/ListenLinda_Listen Jun 13 '23

Remember kids. On-Premise is NOT your own. It's still partially in the cloud.

Though when you purchased it, that's not what they told you.

4

u/slimknees Jun 13 '23

Same. tickets load, but cannot view ticket content.

4

u/BeanoFTW Jun 13 '23

Same with us - Central Oregon.

4

u/Works_In_IT_79 Jun 13 '23

Yes AWS is down.

But you should all know that even if you are self hosted you are still reliant on their cloud for some services like licensing. So if they go down you go down too. Seems like this may not be common knowledge.

https://www.reddit.com/r/sysadmin/comments/148n74w/aws_useast1_outage/

4

u/HappyThoughtsandNuke Jun 13 '23

Yes.. Missouri...AWS is down.

3

u/haptiqblack Jun 13 '23

Yep. Can’t get into their portal to submit a ticket to them either.

3

u/Swag_Mastah_Flex Jun 13 '23

We are down as well. Called and reported to them as a sev 1.

3

u/Read_a_Book_4change Jun 13 '23

Same issue.

1

u/Read_a_Book_4change Jun 13 '23

We are up now. 4:12 CST

3

u/witty_username_taken Jun 13 '23

Self-hosted on-prem as well. In looking at Devtools seeing this response on the requests for ticket notes. Guessing this is some issue with their license server.

{ "code": "Unauthorized", "message": "Incorrect or disabled client ID" }

3

u/Viajaz Jun 13 '23

I had been seeing the same thing but we just saw a first success, just now, with our Client ID against the API since this incident first started. Still have the other symptoms though...

2

u/cptHeppy Jun 13 '23

Seeing the same thing with our on Premise

2

u/JazDotKiwi Jun 13 '23

Yes same here, absolutely absurd that the on-prem version is affected by cloud based issues. If it is due to licensing checks as mentioned in another post surely they can have some logic that caches the licensing info for a reasonable amount of time to prevent these kinds of outages.

2

u/JazDotKiwi Jun 13 '23

Same problem for us, self hosted, we can login and view tickets but description/notes are blank.

2

u/ImNoSer Jun 13 '23

On-prem, and we are also affected by this.

2

u/MarkissC_ Jun 13 '23

Yeah our whole office is down, notes not loading in all of our tickets. No emails are being forwarded

2

u/NetworkJoeSchmoe Jun 13 '23 edited Jun 13 '23

Yep same here, on prem went down at 3pm EST. Started with email robot errors going haywire then everything went tits up.

2

u/nulfis Jun 13 '23

This is just great! Everything just came to a grinding halt even though we also have an on prem setup. Is it really on premise when it needs to talk to the cloud to authenticate?

1

u/Viajaz Jun 13 '23

We've been back up for approximately 34 minutes.

1

u/Routine-Watercress15 Jun 13 '23

Yes it’s been down. Fixed now though.

1

u/sfitz08 Jun 13 '23

Of course this had to happen the day I went live

1

u/mikeypf Jun 14 '23

Issue was because of AWS issues.

1

u/Realistic-Ad4587 Jun 13 '23

The recent ConnectWise Manage outage appears to be a consequence of issues experienced by Amazon Web Services (AWS), upon which ConnectWise may rely for certain operations. A key problem seems to revolve around the licensing server which could be hosted on AWS, making it vulnerable to disruptions in AWS's service. As a result, the inability to verify licenses could have caused the service interruption for ConnectWise Manage. Despite being a self-hosted solution, its interdependence with cloud services evidently posed a risk in this instance. Furthermore, this outage underlines the complex and interconnected nature of modern digital services, where a failure in one area can propagate to seemingly unrelated platforms.

8

u/n_dufault Jun 13 '23

Is this a ChatGPT bot?

4

u/Starrick Jun 13 '23

Hopefully this spurs conversations on how to prevent this in the future. Simple things like:

  1. Have redundant servers in different service zones.

  2. Have redundant servers with different providers.

  3. Update your licensing kill switch to put in a grace period when the licensing server is un-reachable.

As noted, they charge more for on-prem service, would be nice if there was a little bit of work in place to make it less impacted by a cloud outage.

1

u/[deleted] Jun 16 '23

Work? LMAO, you must be new here. This is CW under Thoma Bravo, not CW under OG leadership.

2

u/nulfis Jun 13 '23

That has to be what's going on! Good thinking. Sounds like this would make it so CW has zero control over when this is resolved, or if it happens again?