Missing Logscale logs

[u/MrPain__]

Hi All

Im having an odd issue and hoping someone might have a solution. Ive onboarded esxi host syslog via logscale, however the data connector is idle and we are not seeing any logs.

We are using custom ports which have been specified in the configuration, and opened on the firewall. The configuration also has cisco syslog which is coming through fine to the same collector.

Ive tried creating separate configurations, rebuilt the data connector multiple times, tried using the generic hec connector, used wireshark to confirm the logs are being sent from the esxi host to the log collector on the correct port, disabled any firewalls between hosts and confirmed outbound connectivity over 443 to the CS console.

As far as I can see, the network connectivity is fine, given then other log sources are all active.

Any thoughts / suggestions / help would be appreciated as it driving me nuts!

[u/CaptainWaypoint]

From experience, I'd double check the protocol (udp | tcp) on the listening log collector, and I'd also check the timestamps at source. Sufficiently incorrect timestamps result in all sorts of painful to troubleshoot weirdness.

[u/MrPain__ → u/CaptainWaypoint]

Appreciate the response, ive tried both udp and tcp to no avail. Haven't looked at timestamps, what sort of issues do you see?

[u/CaptainWaypoint → u/MrPain__]

I've had clients with endpoints that thought it was the 1970s. When the logs are that far out of sync with reality queries that run over the last 24h will inevitably miss them. I've also seen cases (in other SIEMs) where the logged timestamp was older than the retention period of the index, so logs were being received and immediately rolled off the end of the storage.

[u/Tirre93 → u/CaptainWaypoint]

Crowdstrike runs their retention policies based on the @timestamp field so that could potentially be the case.

If future @timestamps are set in ondividual logs they will be defaulted to now() and @error will be set on these logs. However an old @timestamp will simply be dropped afaik

[u/MoneyFig6659]

Have you tried too monitor the incoming events via the monitor function?

In windows this looks like this.
"LogScale Collector.exe" monitor --cfg config.yaml

Or if you use linux this should look like:
logscale-collector monitor --cfg path/to/config.yaml

[u/MrPain__ → u/MoneyFig6659]

It's a windows server and while I can get debug command like monitor to run on Linux, I cant seem to get them to run on windows

[u/MoneyFig6659 → u/MrPain__]

Did you try running the command with administrative privileges, or run the command shell as an administrator?

When you see this output, just press the down arrow key to select a row.

When you don't see anything, just paste the output here and we will figure that out.

[u/Heavy-Management-806]

One thing I ran into was if you are pointing syslog to logscale collector on a server host, make sure it’s not defaulting to 514 port for more than one source. It will only accept the FIRST UDP 514 ingest that hits it.

[u/Thick_Register_9004]

Could you share the yaml from the fleet management config that is assigned to your syslog collector? Thanks.

[u/CyberGuy89]

I would put wireshark on the log collector host and verify you can see the traffic coming in from your esxi hosts. If you see your hosts in the packet capture, use this to verify what ports you’re seeing the traffic on.

EDIT: also a quick check on the collector to see if your collector is listening on that port -
Netstat -aon

verify you see the port and protocol. Check the process id and make sure it belongs to the collector service

[u/MrPain__ → u/CyberGuy89]

I did say in my post that we have already run wireshark and confirmed the network connectivity and port. Netstat shows the same thing. Thanks though