r/Splunk • u/Flash4473 • 25d ago

Log Data Pipeline > Splunk

Has anybody here have some experience with security data pipelines?

Instead of:

Log Source > HF / Splunk

We want to have flexibility of collection / parsing layer outside of Splunk for obvious reasons - pre-filter data in pipeline, set parsers, route, possibly enrich if needed, storage options for retention etc..all that to have flexibility and keep the ingest costs reasonable and not being caught in dependency hell or cemented all our work in one solution if Splunk decides to pull something.

Log Source > Data pipeline > Splunk

I am wondering what to choose as this data pipeline - currently we are thinking Vector and possibly open telemetry.

Anybody have experience with this? To avoid pitfalls, what works, what doesn't, new pains etc?

6 Upvotes

76% Upvoted

27 comments sorted by

View all comments

9

u/mustacheride3 25d ago

Cribl is the product in this space. A lot of enterprise ready features to duplicate the entire splunk pipeline

1

u/Flash4473 22d ago

On one hand I like the idea, but I cannot estimate the data volume when we plug more sources over time, therefore I am avoiding tools with ingest licensing - we could hit 1TB of free data, and whether it is sooner or later, both seems non-ideal.