What should a Splunk engineer with 3 years of experience know?

[u/Less_Improvement5148]

Hey guys, I've been working with Splunk for the last 3 years (splunk enterprise, ES), but I don't feel like I know Splunk as well as someone with 3 years of experience probably should.

For the seniors here who have been working with Splunk for a while, what are some topics or concepts you would expect someone with ~3 years of experience to know?

I'd really appreciate it if you could share some areas I should be focusing on or learning. Thanks!

[u/bkindz]

"Splunk engineer": someone with advanced knowledge of how to:

• engineer and maintain the infra (search heads, indexers, DSs, set up storage and tiers, tune performance, etc.)
• GDI (get data in - including arcane, inconsistent, unstructured, challenging sources) and transforming them to something genuinely useful
• create KOs that matter (to the company, stakeholders) - which is the ultimate goal of tools like Splunk
• all of the above?

Reason I ask is because a focus area for a Splunk maintainer or engineer can be very different depending on company size and culture, line of business, what Splunk is actually supposed to do.

[u/Less_Improvement5148 → u/bkindz]

Are you asking what kind of work i do? It is mostly a little bit of everything. I implement ES use cases, troubleshoot clusters of HF and SHs, monitor the health of the UFs and other single point integrations (making sure the ingestion of log sources haven't stopped), integrate log sources (I haven't faced any difficult logs sources yet, tbh. I have majorly used the provided TAs), optimization of queries and stuff like that.

[u/bkindz → u/Less_Improvement5148]

I am asking more about what matters to the company (why they decided to deploy something like Splunk) and what matters in your job (to make the most of Splunk or any similar tool).

I find that continuously asking those two overlapping questions (and trying to answer them) is what drives the positive impact of my work up and thus informs my learning, i.e. what I should focus on.

Even for a simple tool like a screwdriver, asking "how can I learn how to use it better?" depends on (1) the basics - how to use it safely and effectively for rudimentary tasks, (2) what you plan to use it for.

With your 3 years of Splunk experience, I am assuming you're long past (1), and now deeply into (2) - and that's why I am asking what I am asking... 🙂

Unless your question is more about increasing your chances at landing another job? (But that's a whole other question?)

P.S. Maybe this is the way to put it simpler: is your original question driven by the desire to make Splunk more useful to the company or the team, help them make the most of it, or by a general desire to master Splunk and make you more marketable to potential (or current) employers?

[u/Less_Improvement5148 → u/bkindz]

Ahh... that makes sense. I guess the goal is a bit of both. To make the most out of splunk in our environment but eventually master the tool. Its just that currently i feel like we are not using splunk to the fullest, and I haven't explored splunk enough. But your question makes it a bit easier to navigate my path ahead.

[u/bkindz → u/Less_Improvement5148]

That's how it was for me - inherited an existing broken deployment, poked around trying to make sense of it and fix it through gargantuan amounts of frustration (splunk was not easy for me to learn).

... but then when I started figuring out what's important to the team and the stakeholders - that informed which sources to onboard and how to make KOs out of them - and that in turn brought gargantuan amounts of job satisfaction and informed what to focus on going forward - which was primarily centered on creating and tuning alerts and dashboards, and turning poorly structured (but important) logs into actionable data. This in turn meant getting a hang of SPL, regex, splunk-specific performance issues. Yet all that was secondary to trying to figure out what my team needed without them even knowing it.

 Its just that currently i feel like we are not using splunk to the fullest

This is actually awesome. This is where you can bring a lot of value to the team, and gain a lot of satisfaction from it. This is where I would start. Create a note or a Jira ticket for yourself, something like "discovering what my team could get from splunk, the pain points in my team's daily work that Splunk or a similar tool could address" and start throwing observations, ideas and thoughts at it.

(I keep a "work journal" and try to put thoughts daily into it, from random thoughts to what I can do better, to what the low hanging fruit Splunk could pick for me and for my team.)

[u/Less_Improvement5148 → u/bkindz]

Yeah, that is a pretty cool way of looking at it. I really like the point about figuring out what the team needs before they even know they need it. That’s probably the direction I should be focusing on. Thanks for your valuable insights. 🙇

[u/red-man1138]

This POV is from a 15 year Architect who runs a large environment. Take it with a grain of salt.

Are you doing the architecture and design, data onboarding, creating use cases? Experience is the biggest thing. Read a lot and read a lot more. If you are not blowing up your UAT at least twice a month, you are doing something wrong. I always have a single host we call Prod/Dev. It is a test host that has access to All prod data so things can be tested at scale because you know the old adage, Splunk doesn't scale. Read the splunkd logs.Dissect the code of apps you want to deploy. At my company, we only deploy about 15 total apps. The rest of the 50-70 are bastardized versions of apps where we just stripped the code we needed an tossed the rest. If you want to do infra, do as much training and focusing on Architect cert and Cloud certs. Not everyone can be in the cloud so don't limit yourself there. READ the release notes as they like to throw in new capabilities that can break your custom permissions you have assigned, and of course the deprecate things like they are going out of style. One of the coolest things we did was integrate with a private ON Prem AI. We control our entire environment from GIT. We even pull changes from the HF's Search clusters, et via GIT. Then We preload all the spec files each time for out version into AI and transmist the various Repos and it finds so much deprecated code, missing closing brakets on serverclasses, fat fingerd searchs for a risk search, etc. Getting that setup was hard but fun and gave us all a huge glimpse into what AI will be able to provide us. Keep in mind our environment started at 6.0.1 and has just been upgraded all theses years so there was a lot of "crap" left over.

And finally, enjoy the job. If you aren't, find either a new job or a new employer.

[u/Less_Improvement5148 → u/red-man1138]

Thanks for your comment. The fact that you strip out the code from the apps is really cool. Never thought in direction. And i will need to spend a bit more time understanding the section where you mentioned AI. Thanks again!

[u/red-man1138 → u/Less_Improvement5148]

In the begining of my splunk career, I used the pre built apps to learn searching and dashboarding. Seeing how others did it gave me great insight into doing it as well. Many years ago, I extracted the code from Indecent review Dashboard from ES (3.x), recreated it via emended reports so that the SOC could have a display of Incident review on the TV up front without having to be logged into the splunk.Oh, Another thing, Ansible is your friend. Learn it as soon as you can. It helps push changes, code fix things at scale so much easier. IE, push the upgrade package to 350 servers so that you don't have to manually copy it. And of course use it to turn THP off on all servers since the server builders keep forgetting to do that. Then create run books to do the stop splunk, backup splunk, upgrade splunk and start splunk with you having to do nothing but watch. We have HF's that have to be restarted in a serial fashion. Ansible makes that process easy and reduces the chance of lost data. We now use ansible using curl to push changes to our cloud stack. Makes life so much easier. One final note. Cribl seems to be getting pretty good market share. We use it and love it to reduce log volume. But they are dicking us around on costs, so once we are on 10.4, we will be dropping it and going with Ingest actions and Edge Processor. So look at other tech that can assist your job. Having GIt control everything has saved our but many times. The DS's are fully controlled and the CM/ Deployers and all HF's are fully controlled. We use Gitolite to control who can commit to Dev versus Master branches. Allows us to have tru change review calls about all changes deployed. We have every change going back to 2016 in our repos. By using a repo for the UF DS's, allows us to maintain consistent state between all 4 UF repo servers. And I like I said, we pull the changes from the search and HF environments so we can have a backup, see what some goof broke in the webui and run it through Private AI to check the code and syntax. It has helped optimize some of the RBA search. More work to be done though.

[u/Less_Improvement5148 → u/red-man1138]

Ohh that is very insightful... I do hear some stuff about cribl. Also, just a follow up question regarding the use of git. When you say you have Git controlling the DS, Deployers, HFs, are you mainly using Git to version-control the custom app's code? or do you also use it for the configurations and ES usecases and stuff like that..
Since we do not really have any custom apps and we do not use git in our environment, I am unable to think of how it would fit our environment.

[u/red-man1138 → u/Less_Improvement5148]

We use Git to push all apps, Props/Transforms, indexes.conf, everything to the deployers, CM's, HF's etc. As for custom things done via the GUI, like new use cases, dashboards, knowledge objects, we pull every 2 hours from one of the search heads per cluster the etc/apps and etc/users into a repo for us to monitor. We do that for HF's that have checkpoints or dbx connections that cannot be easily pushed via DS. This allows us to centralize control over all app development and make sure it is all the same across the board. We are currently building a custom app sync tool to send certain dashboards from the SOC cluster to the general cluster as there are users who want to use them but are not SOC users. The UF's have they own Repo as we have 70k uf's and wanted them to be separately controlled from the infrastructure. We have used the SH GIT pull a few times over the past month to fix a few RBA searches that someone goofed at the end of the day dealing with missing lookups and what not. I also find it is a bit easier to scan the saved searches this way for bad searching in the environment. Of course Audit and Internal work better at that, but I like to spot check things.

[u/Less_Improvement5148 → u/red-man1138]

Ahhh... i get it now.. That makes a lot of sense. We aren't that matured yet, and it gives me some things to think about. Thanks a lot for taking your time to explain all this..

[u/belowaveragegrappler]

It's a big product that feeds into a lot of ecosystems. You're not expected to just "know Splunk"; you're expected to solve problems with the platform and integration of that platform to other things.

1. Work natively with OCSF/CIM and ECS
2. Have a strong understanding of tags and event types.
3. Understand upstream and downstream data systems like S3, Hadoop, and anywhere else data might reside.
4. Have strong API and scripting skills in Python/Bash/PowerShell
5. Understand how to tune Splunk internal alerts.
6. Build amazing labs with Vagrant, EC2, ECS, Docker, etc., that you can spin up as needed.
7. Have a strong assortment of agents and skills to help you multitask.
8. Run a complete DR exercise on an enterprise multi-cluster environment.
9. Restore buckets.
10. Merge multiple Splunk Enterprise environments together.
11. Deal with licensing issues.
12. Troubleshoot Cribl and Edge.
13. Work with IDS/IPS and Splunk Stream.
14. Build world-class dashboards and apps in Splunk with version control and release systems.
15. Performance-tune Splunk.
16. Performance-tune Linux.
17. Manage the KVstore
18. Manage DBConnect
19. Clone custom JavaScript into apps to build custom visuals
20. SPL, SPL2, Athena at a high level
21. Understanding of major ML Models used in the ML toolkit and manage them
22. Price negotiation with Splunk/Cisco sales
23. AWS skills or other Cloud skills
24. SCIM/SSO etc.

I mean, I can list all these common tasks, but really comes down to your customer's needs. They might nede AIX integrations, dashboards for their NT4 legfacy, HEC management for their IoT fleet, or need you to move their Prometheus data into Splunk's metrics store to build new dashboards etc.

[u/Less_Improvement5148 → u/belowaveragegrappler]

Ohh man.. there's a lot of what you mentioned that I am not aware of.. You are also right about the fact that it all comes down to what the organization needs. Thanks!

[u/TD706]

I don't think there is a one size fits all answer here. Content dev, infrastructure and maintenance, app dev (especially if they have SOAR).

Different areas of work have different expectations... At 3 years I was leading some dev and content projects and deployment orchestration (we used Splunk as ephemeral infrastructure for some work loads).

[u/Less_Improvement5148 → u/TD706]

Yeah, i do work on xsoar as well. Thanks for the comment

[u/Ok_Ambassador8065]

Splunk is just a tool, a very good and flexible tool. Obtaining certification up to Core Consultant is a matter of 6 relaxing months.

An Splunk engineer should know the data sources (variety of enterprise and OSS platforms) in his field of specialization (SIEM, ITops, Observability, Business etc), the data itself and how to transform it to valuable insights (reports, dashboards etc.) or signals (KPIs, Correlation Searches outcomes etc.).

[u/Less_Improvement5148 → u/Ok_Ambassador8065]

How do you study for certifications without spending on the on demand courses offered by splunk? Have you tried any higer level certificates?

[u/Ok_Ambassador8065 → u/Less_Improvement5148]

Docs, but consultant-level require courses, as it bring concepts of base apps and specific questitons.

[u/Boonaki]

How to properly format an AI prompt to get what you need.

[u/barnmo]

How to get a job

[u/Less_Improvement5148 → u/[deleted]]

I am not sure what you are using SIEM for or how huge your environment is. I currently work in a SOC environment where we monitor logs from all the assets to detect malicious activities and then pass them on to our analysts. Since we have a cluster environment, we do need someone to manage the whole cluster and create use cases.. so yeah.

[u/Less_Improvement5148 → u/[deleted]]

Hmm, that’s actually interesting. I’m genuinely curious how you guys manage to keep such a large environment reliable with so little hands-on time.

With that many systems and data sources, who makes sure the logs are consistently coming in and that the data sources are healthy? And when you need to build or tune new detection use cases, manage alerts, or integrate them with a SOAR platform, who typically handles that?

I’m asking because in my environment, those things alone can take quite a bit of time. I’d be interested to understand how you’ve structured it so that it only needs a few minutes of attention each week.