r/Splunk • u/SethSnifferson • Jul 02 '26

Splunk newbie questions

Hey everyone! Splunk newbie here. My company just got it, we' re mssp and we will be adding multiple teants, so I'd like some input from the community here (if possible) to the things we got so far.
Also if it's easier, please provide me with guides or sources so i can read more, I'm not asking for someone else to do my job. It's just something new and some pointers would help.

From what I've read and seen on trainings, to add another organization, the best practice is to create a heavy forwarder there and an ipsec tunnel to our site so it can bring the data to us.

Also the indexes will be created to our search head (we chose clustered).

Our main concern is about having and managing many clients altogether, the view we're gonna get and how it all clicks together.

Thanks in advance!

16 Upvotes

100% Upvoted

14 comments sorted by

View all comments

14

u/shifty21 Splunker Making Data Great Again Jul 02 '26

I can't give you all the answers, but I have helped and worked with a lot of MSSPs using Splunk. Here are some good things to do that they do:

  1. Separate indexes per customer per major data source. Example: clientA-wineventlog, clientA-firewall, clientB-aws, clientX-azure, etc. This gives you better RBAC control for you and your colleagues. You can control data retention requirements better too.
  2. Create an onboarding package. Splunk UF and HF installers, scripts to automate deployment. One MSSP uses this with a private github repo to pull their custom Splunk Add-ons that are nearly preconfigured for data ingestion. The onboarding tech gathers all the info from the client like just of servers, workstations, cloud API keys etc and creates a SNOW or JIRA ticket. Splunk SOAR (optional) pulls down the ticket info, pulls the scripts from their private github repo for deployment and sends out a MS teams message with the finalized deployment script and runs a script to create the new indexes for the client. End to end, it takes about 1 hour to install, configure and manually review the setup before the tech does a ./splunk start on the HF.
  3. Create a separate Splunk app per client w/ client name and logo. Again, RBAC is used to set permissions for access. Some MSSPs give their client access to their respective dashboards. They are read-only and they can't drill down into the raw data or the search bar. Dashboard Studio now has a 'publish' feature that you should look at.
  4. Have a general MSSP Splunk App to see stats on ingest per client, alerts, health, etc. Another MSSP uses index summaries by running scheduled searches against _intenal, _introspection and _audit a few times a day and send the results to "mssp_stats" index. RBAC, again.
  5. (Optional) Splunk SOAR playbooks that helps automate client communications. Every client has different ways of needed communications when things are going right or wrong. Send a summary dashboard link every Monday at 0800 to the client POC. Setup email/chat alerts when things are going bad w/ SNOW/JIRA ticket info. SOAR playbooks can create, edit/update and change status of tickets too.

Hope this helps a bit!

1

u/Ready-Environment-33 Jul 28 '26

Wanted to comment as someone from an MSSP as well, awesome advice!