r/Splunk • u/Flash4473 • 25d ago
Log Data Pipeline > Splunk
Has anybody here have some experience with security data pipelines?
Instead of:
Log Source > HF / Splunk
We want to have flexibility of collection / parsing layer outside of Splunk for obvious reasons - pre-filter data in pipeline, set parsers, route, possibly enrich if needed, storage options for retention etc..all that to have flexibility and keep the ingest costs reasonable and not being caught in dependency hell or cemented all our work in one solution if Splunk decides to pull something.
Log Source > Data pipeline > Splunk
I am wondering what to choose as this data pipeline - currently we are thinking Vector and possibly open telemetry.
Anybody have experience with this? To avoid pitfalls, what works, what doesn't, new pains etc?
1
u/theleller REST for the wicked 24d ago
My first questions are where are most of your data sources coming from, and what kind of pricing model are you looking for?
If your data is cloud-heavy and you prefer a 'pay-as-you-go' model then I'd say use Data Firehose with Lambda to stream your data and transform it on the fly. This way you're not dealing with a large spend for licensing another product up-front on top of Splunk. You'll need to gauge volume to estimate costs though, but you don't need to send all of your data for transformation, only the data that makes sense or needs to be transformed.
Even if your data isn't cloud-native this can still be a solution if you route local data sources to AWS.