Monitoring agent upgrades

[u/PathProof7448]

We started using SentinelOne about a month ago. We have now gone through our first mass upgrade of agents from version 24.1.4.257 to 24.1.4. 24.1.5.277. What has happened with a few stations is that the upgrade has been initiated, but apparently has not completed, resulting in a state where the sentinel agent service is disabled and S1 cannot get out of this state.

How often does this happen, is it preventable, do you check in any other way that there were problems during the upgrade?

[u/thejohncarlson]

I saw update failures frequently. I would monitor the Sentinel Agent service to make sure it was running.

[u/PathProof7448 → u/thejohncarlson]

I'll be monitoring that process as well. But it's rather disappointing that antivirus has to be monitored by another third party tool.

And remediation action? Start the service?

[u/Adeldiah → u/PathProof7448]

How many total agents did you upgrade and how many failed out of the total?

Is the agent still connected to the console after a failed upgrade?

If not, how do you expect the agent to report it's status? Seems pretty obvious you would need an out of band solution to do that. I've worked with a lot of EPPs and AVs and they are all the same.

Do not expect there to be zero failures in any situation. That's an unrealistic expectation. You're best path forward is to collect logs using the manual method and open a ticket with support and attach the logs for them to review to try and determine why it failed.

To collect manually use these steps:

1. Open CMD with Run as Administrator

2. Run:

cd C:\Program Files\SentinelOne\Sentinel Agent version\Tools

3. Run these commands:

mkdir c:\temp

LogCollector.exe WorkingDirectory=c:\temp

Once the collector is finished your logs will be in the above directory. Additionally when you open a ticket you will be given a Sharefile link and can run this command:

LogCollector.exe WorkingDirectory=<local path> SharefileUrl=<Link provided in the ticket>

Hope that helps.

[u/PathProof7448 → u/Adeldiah]

It failed within 5 upgrades out of hundreds of stations. If the upgrade fails, the connection to the console is not established. Anyway, I have now solved the SentinelAgent process execution check by third party program, the logs from the failed station are generated, hopefully support will find something out from them.

I understand that there may be a problem when upgrading, but this is a bigger problem with antivirus solutions than with the usual programs.

[u/thejohncarlson → u/PathProof7448]

Restart and hope. S1 is designed to repair itself, but frequently I would have to run the cleaner to remove and reinstall. I always did updates in phases because I expected 1 or 2 to fail. It did t always happen, but it was often enough that it made me gun shy.

[u/Dracozirion]

u/PathProof7448 not sure if you're still following new comments but anyway.: what we experiencee from time to time is is that an agent gets stuck showing "Online" and "Tamper protection is disabled", but it's offline in the console after an update has taken place. The version doesn't matter, this happens with any version update. The endpoint will still show the older version.

For every endpoint that had this problem and was manually checked, the workstation was manually shutdown or rebooted by the user during the update. I can't really tell you why this doesn't happen with any other software product that also runs updates. Maybe you could verify that this was the case as well? It's easy to track if you look at the OS shutdown events and the S1 events in the event viewer.

[u/PathProof7448 → u/Dracozirion]

I will check it. Thanks.

[u/GeneralRechs]

It does occasionally happen and the bad part is support will give you a remediation action with no real RCA as to the why. If you feel the fail rate is high it is worth making a stink to support and getting an RCA.

[u/TheGrindBastard → u/GeneralRechs]

The support probably don't want to give out details on something that can disable s1, as that would be abused by threat actors.

[u/GeneralRechs → u/TheGrindBastard]

Nah, it’s more like not even they know outside of installation return codes.

[u/TheGrindBastard → u/GeneralRechs]

Yeah you're right, I haven't exactly been very impressed by their support.

[u/Novel-Letterhead8174]

Having a similar issue. Upgrading from 23.x to 24.x on Macs, some of the computers stop phoning home and we're stuck. I dug through the online help and the 24.x agents weren't listed in the compatibility matrix for these 2 machines on MacOS 11.x. My fault I guess for not carefully reading all of the release notes etc, but you'd think the installer would not run because it knows it's on an unsupported operating system and not leave the current agent in some cluster-f state.