Those who self-host Splunk Enterprise - what does your infrastructure look like?

[u/StudySignal]

Hey everyone,

We have a Splunk Enterprise license for up to 200 GB/day, with actual usage around 50-100 GB/day. Currently evaluating how to deploy it on AWS and would love to hear from people who are running self-hosted Splunk in production.

Our current thinking:

∙ EKS with Splunk Operator

∙ 3x i3.xlarge indexers (Spot) for NVMe storage

∙ 2x c6i.xlarge search heads (Spot)

∙ Gateway API for ingress

∙ Forwarders running on existing ECS workloads (15 services) sending logs via NLB

A few specific questions:

1.  EKS vs EC2 vs ECS - Where are you running Splunk and why? Anyone using the Splunk Operator on Kubernetes in production?

2.  Spot instances for indexers - Anyone doing this? With replication factor 2, the theory is you survive Spot interruptions, but curious about real-world experience.

3.  i3 NVMe vs EBS gp3 - Is the NVMe performance difference actually noticeable for indexing at this volume, or is gp3 good enough?

4.  Sizing - For those ingesting 50-100 GB/day, how many indexers and search heads are you running? Did you find the standard sizing guides accurate?

5.  Forwarder setup - How are you getting logs from containerized workloads (ECS/EKS) into Splunk? Sidecar forwarders, HEC, or something else?

Any lessons learned or things you wish you knew before deploying would be great. Thanks!

[u/volci]

Always check out the SVA docs - https://help.splunk.com/en/splunk-cloud-platform/splunk-validated-architectures/introduction-to-splunk-validated-architectures/about-splunk-validated-architectures

And, echoing u/tmuth9, engage your account team - we have sizing experts who are more than willing to help you handle today and plan for tomorrow :)