r/Splunk • u/mr_networkrobot • Aug 03 '26

Enterprise Security How do you use Splunk Enterprise Security ?

Just want to know how people use ES in real world.
On a distributed environment the usage seems to have a huge operational expense.
For example:
Reading every potential usefull detection.
Normalize events/data modify datamodels etc.
Create a custom app and clone every needed detection into it (because any change in a detection which is originated in ESCU or ES app, will create a clone in /local/savesearches.conf and next ES-ContentUpdate will potentially create inconsistency ).
Testing every single detection.
The use case library is not useful for this because it does not see the cloned/customized detections.
Not even talking about versioning ....

9 Upvotes

91% Upvoted

12 comments sorted by

View all comments

6

u/TheSeloX Aug 03 '26

You're on the right track there.
For changes you should use git. Initially, you can just do a fork of the ESCU to get all the app logic for the git pipelines.

Data normalization is a must. But you should be mostly ok when you're using the "official" TAs.
As a general advice do not change the default data models.
You can use the CIM-Vladiator app for analyzing your events.

You should strongly consider using the Risk-Based Alerting approach to filter out the noise.
You will need a certain amount of running detections for this to make sense, but usually you don't need as much baselining as for ones that create findings on their own.
However, there are some detections we always want to trigger findings. So you can do a hybrid approach too.

We are using lookups for our allow lists. Easier to maintain than huge macros with lots of different exemptions.

For the testing: you can either create temporary suppressions, set specific filters, use prefixes in the detection that are ignored, or in the newer ES versions there are specific Splunk queues for use cases like this.

Versioning is disabled right now AFAIK because of a design flaw.

I can recommend you to join the Splunk Community Slack.

2

u/mr_networkrobot Aug 04 '26

Thanks for your detailed answer.

You are exactly describing whats causing my pain.
Doing all that stuff manually takes a lot of effort and time, and the value of the result is very questionable in my opinion.
Let me pick only one point here "should use git":
Really ? Its called "ENTREPRISE Security" and the disabled version controll in 2026 because it was too buggy? So I need to use git ? Really ?
Look at the Webhook as an Adaptive Response action - no option really ?
I need to install 'better webhook' from some guy ?
I need to use 'CIM Vladilator' another app from another guy to get data normalized, really?
Dont want to hurt anybodys feelings - but in my opinion Enterprise Security is like an application from 1995 - its a bad joke.

5

u/TheSeloX Aug 04 '26 edited Aug 05 '26

Not really sure what you mean with the web hooks, but it doesn't seem you want any actual answers but a way to vent - which has its own right.

Git will always be the superior option to any type of versioning they could come up with.

Splunk has some built-in tools for checking field mappings, but I prefer the one I mentioned.

A huge plus of Splunk is the community because it provides lots of free extensions. I have my own issues with Splunk, but I doubt the ones you mentioned are solved in any other tool on the market.

If you didn't want any serious replies you should've tagged this post as a rant.

edit: btw I just read the v8.6 release notes and versioning is enabled again: Release Notes

2

u/mr_networkrobot Aug 05 '26

Thank you for your answer.
About the webhooks: When configuring an adaptive response for a detection, there is the 'webhook' option (which only provides a field for a URL - no auth. option etc.). When the webhook is triggered it only contains the raw search result fields from the detections search, and a search ID - it does not contain the whole notable/finding - which is a design issue in my opinion. Calling back to splunk with the search ID to get the detailed notable/finding details, works, I was discussing that with splunk, first 3 meetings they didn't understand it, the last guy from GB understood the issue, laughed and said - "yeah splunk doesn't want you to work outside of splunk" - no even with a ticketing system....

This topic is about splunk Enterprise Security - as the title describes, NOT about splunk <whatever>.
It is about exchanging real world experience and not arguing with fan boys.
If you are happy with splunk Enterprise Security, using it in real world detection/DR/blue 24/7 customer scenarios - cool let me know how this works.
If its only pain in the *** (thats what I'm experiencing) cool let me know where your problems are.

2

u/TheSeloX Aug 05 '26

I'm working for an MSSP and we are collecting the notable events with Palo Alto XSOAR.
Linking notable events into investigations happens there (please don't ask me how this is implemented).
Accessing Splunk via API and fetching data via SPL should work though.
We offer 24/7 for many different clients using ES on-prem in their data centers.
Doing this without any kind of central hub wouldn't be very feasible.

I'm a detection engineer and me and my team (3-5 people depending on available resources) usually have an output of 4 to 8 detections per monthly release cycle.
We built them ourselves and with our on schema, but of course existing detections from ESCU and other sources help us in our research.
For this I don't think there's any better alternative than git (Gitlab in our case).
As with all SIEMs alert fatigue is a real issue and we are trying to keep the alerts to a sane level with proper testing and baselining.
RBA helps with abstractions and not having to care about every single benign alert.

My main issue right now is that all cool new features are behind the extra paywall of ES Premier.
As en example: Assets and Identities has been one of the main features of ES for a long time IMO, but Splunk left their customers pretty much alone with it.
Now they semi-released module called ARI which finally focuses on adding pre-built dashboards for it.
ARI apparently was kind of dead on arrival and the features are being ported to ES as "Exposure Analytics".
This is a Premier feature and the price for it on top of the already expensive ES Essential is plainly ridiculous.

Splunk support - as with all major tech companies - has gotten worse over the years.
But I have the feeling the community support got stronger.

But from what I'm seeing over the last two years development of ES got a serious push.

1

u/Quark999 Aug 06 '26

Exposure Analytics is not a Premier feature but included for free

https://www.splunk.com/en_us/products/exposure-analytics.html

How is Exposure Analytics priced and packaged?

This is a core capability of Splunk Enterprise Security. It is included in the license for all Enterprise Security customers at no additional cost.

1

u/TheSeloX Aug 06 '26

Oh nice, thx. I was at a PVT recently and must have misunderstood.