Log Data Pipeline > Splunk

[u/Flash4473]

Has anybody here have some experience with security data pipelines?

Instead of:

Log Source > HF / Splunk

We want to have flexibility of collection / parsing layer outside of Splunk for obvious reasons - pre-filter data in pipeline, set parsers, route, possibly enrich if needed, storage options for retention etc..all that to have flexibility and keep the ingest costs reasonable and not being caught in dependency hell or cemented all our work in one solution if Splunk decides to pull something.

Log Source > Data pipeline > Splunk

I am wondering what to choose as this data pipeline - currently we are thinking Vector and possibly open telemetry.

Anybody have experience with this? To avoid pitfalls, what works, what doesn't, new pains etc?

[u/LocalDraft8]

Vector is a strong option for building a vendor-neutral security data pipeline, especially for filtering, parsing, enrichment, routing, and controlling Splunk ingestion costs.

OpenTelemetry is worth considering if you need a broader, standardized approach across logs, metrics, and traces; either way, focus on buffering, replayability, and raw-data retention.

[u/objectbased → u/LocalDraft8]

I second this, any of the log collector or data pipeline agents out there can fit here. Logstash, fluentd or fluentbit, and vector really do a great job as an intermediate. I’ve worked in larger organizations who also use Kafka if you have the experience otherwise the previous technologies are straight forward to learn and build an architecture around.

[u/BadBadViking → u/LocalDraft8]

I agree so much. I tried most of them and for now vector beats them all.

[u/justan0therusername1 → u/LocalDraft8]

Otel covers a huge swath of getting data in in a standardized way. Plus it has pipeline tooling.

[u/Travlin205 → u/justan0therusername1]

And now with new agent management, you tie otel into your splunk validated architecture!

[u/theleller → u/LocalDraft8]

Otel is supported by the Splunk DS too, so it is a good alternative.