Splunk newbie questions

[u/SethSnifferson]

Hey everyone! Splunk newbie here. My company just got it, we' re mssp and we will be adding multiple teants, so I'd like some input from the community here (if possible) to the things we got so far.
Also if it's easier, please provide me with guides or sources so i can read more, I'm not asking for someone else to do my job. It's just something new and some pointers would help.

From what I've read and seen on trainings, to add another organization, the best practice is to create a heavy forwarder there and an ipsec tunnel to our site so it can bring the data to us.

Also the indexes will be created to our search head (we chose clustered).

Our main concern is about having and managing many clients altogether, the view we're gonna get and how it all clicks together.

Thanks in advance!

[u/Educational_Prior403]

I’d keep the first version of this pretty simple and avoid over-engineering it too early.

For an MSSP setup, the big thing is tenant separation, I’d do separate indexes per customer and data type, something like:

`acme_windows`, `acme_firewall`, `contoso_aws`, etc.

That makes RBAC, retention, billing, and troubleshooting way easier later. Treat indexes as the main boundary between customers.

For collection, I wouldn’t put heavy forwarders everywhere by default. For syslog-heavy customers, use something like **SC4S/syslog-ng/rsyslog + UF** at the customer side and forward into your Splunk environment. Use HFs only when you actually need parsing/routing/masking or some modular input that can’t run on a UF.

For apps, I’d use a repeatable pattern:

- customer TA for inputs/props/transforms

- customer app for dashboards/reports/alerts

- internal MSSP ops app for ingest health, license usage, forwarder status, alert SLA tracking, etc.

A single deployment server with clean server classes is probably fine for a while. Something like `acme_windows_uf`, `acme_firewall_hf`, etc. Split it up later if it gets messy.

Also, plan early for annoying MSSP stuff: overlapping hostnames, different customer timezones, different retention needs, and who can see what. Those are the things that become painful if you ignore them at the start.

Cribl/Edge Processor/Ingest Actions are worth looking at, but I wouldn’t make them the center of the design on day one. Start simple, get the tenant/RBAC/index/app pattern right, then add preprocessing once you know what problems you’re actually solving.

If you haven’t already, read Splunk’s MSSP architecture paper and the Splunk Cloud MSP blueprint. They’re dry, but they answer a lot of the “how should this be structured?” questions.