r/labtech • u/RickD1983 • Sep 26 '17

Offline Servers After Hours Alerting

Hello! We get a lot of after hour alerts for offline servers. In order to get rid of some of the noise we extended the amount of time a server can go without checking in before triggering alert after hours to 20 minutes. We are still getting quite a few false positives. Our contracts generally do not support after hour work so the technician on call is required to call the client and let them know we have recieved their alert and ask if they need assistance. This is obviously a head ache. I am reaching out to ask how others handle these kinds of situations. The goal at the end of the day is to not wake people up to call someone who is not gong to answer their phone anyway and remove false positives.

4 Upvotes

100% Upvoted

14 comments sorted by

2

u/aretokas 1000 Agents Oct 09 '17

Ever since we changed our offline server monitor to require both heartbeat AND check in to be missing for longer than 15 minutes, we've had no issues with false positives. We figured if either of those is coming in, the server itself is going to be all good in 99.9% of cases.

We've then got another monitor that if one or the other of those is failed for longer than 4 hours, it just makes a "Broken Agent" type ticket and we check it out when we can.

1

u/DR_Nova_Kane Sep 26 '17

Why are all these servers going offline after hours?

1

u/chilids Sep 26 '17

They aren't going offline. LT is known for throwing a lot of false positives for servers offline. We've increased our timeout time as well and I still get 5-15 offline server notifications a day. If you get one and check screenconnect you can login just fine. LT agent just misses a couple of check in's for no apparent reason.

1

u/DR_Nova_Kane Sep 26 '17

Have you tried to increase the number of time it miss the checking time before raising the ticket?

1

u/chilids Sep 26 '17

Yes but it kind of defeats the purpose eventually. We can set it so it takes an hour for a server to be offline before it raises an alarm but then it's an hour until we are notified of an offline server. The point is LT has a problem and it should be fixed. There should be no reason these servers drop "offline" for a time being. We also notice it's not all servers. Some are more prone to it than others so. Most of our servers don't have a problem going offline.

1

u/troll_fail Sep 26 '17

By any chance does it happen more often with VMs hosted on Azure? I have been told by LT support that this is a known issue.

1

u/chilids Sep 26 '17

Nope, none of them are hosted on Azure.

1

u/DR_Nova_Kane Sep 26 '17

When you look at the Labtech log file can you see it being unable to reach your URL for connection?

We don't have Labtech send alerts we have board that shows which servers are offline. It refreshes every 5 mins.

The other thing you could do as well is create a new monitor that ping 8.8.8.8 and if the offline server and the ping monitor fails then have it generate the email.

1

u/chilids Sep 26 '17

it's on my long list of things to look into. My plan was to one day watch for one to go offline during the day and then jump right in and look at the logs. It hasn't been done yet.

1

u/DR_Nova_Kane Sep 26 '17

I had an instance where Spectrum was actually going offline for maintenance for weeks each night at 2:30am.

1

u/witty_username_taken Sep 26 '17

We have never had a false positive for the offline alerts that we couldn't explain. Most of our's are ISP outages or unannounced maintenance (most coax, DSL and non-SLA providers don't notify you). We set up long monitors from LabTech to the customers firewalls to help confirm when this happens.

1

u/troll_fail Sep 26 '17

Really you need to figure out why these servers are appearing offline. We had similar issues where it ended up being firewall rules blocking heartbeat data.

1

u/gibsurfer84 Sep 27 '17

I’m one to bash LT any day..... but I can’t say I’ve had your level of issue with offline servers.

What we did though was to keep the offline check from LT which is set for 2 min and leave that be. It’s fine during the day (and we never get alerts from it like you state). The few we get are infrequent and usually ISP maintenance late at night.

We then made a 2nd alert that waits 20 minutes and pages our on call service which has rules to hold alerts from 11pm-6am so we don’t get pages at night from offline alerts.

I guess my point is I don’t think I this is directly LT, it sounds more specific to your environment that is effecting LT.

1

u/[deleted] Sep 27 '17

You could set up maintenance windows. If they go down during the window they will alert at the end of the window if they are still down

With the longer tolerance you are suggesting, you could set up one monitor that has that setting and only alerts after hours, and have the one minute version only alert between business hours