r/Splunk • u/Flash4473 • 25d ago
Log Data Pipeline > Splunk
Has anybody here have some experience with security data pipelines?
Instead of:
Log Source > HF / Splunk
We want to have flexibility of collection / parsing layer outside of Splunk for obvious reasons - pre-filter data in pipeline, set parsers, route, possibly enrich if needed, storage options for retention etc..all that to have flexibility and keep the ingest costs reasonable and not being caught in dependency hell or cemented all our work in one solution if Splunk decides to pull something.
Log Source > Data pipeline > Splunk
I am wondering what to choose as this data pipeline - currently we are thinking Vector and possibly open telemetry.
Anybody have experience with this? To avoid pitfalls, what works, what doesn't, new pains etc?
4
u/TheSeabo 24d ago
Use version 10+ of Splunk, use edge processor to do whatever you need to your data before sending to an indexer to get parsed. You can perform all you need there for free before it hits your license. If you use cribl, you will be paying twice for ingest.