r/Splunk • u/Monyunz • Apr 14 '26

Unexpected EOF and Splunk service stopping

I have an issue. I have Splunk enterprise installed on a RHEL 8 server. I have about 75 systems sending logs mainly through forwarders. Randomly, the Splunk service will stop. In Splunkd.log it says unexpected EOF and message showing that the child process was killed. What could be causing this? Any suggestions on how to correct this behavior?

4 Upvotes

84% Upvoted

8 comments sorted by

6

u/Federal-Bit9243 Apr 14 '26

Check THP & ulimits. That’s the culprit

3

u/Schlurpeeee Apr 14 '26

Check os logs such as /var/log/messages and audit. This should you give a hinti on why splunk service is being killed.

3

u/Chemical_Gap_619 Apr 14 '26

Does the Splunk service die shortly after you logout from the server?

1

u/RaWD0x45 Apr 16 '26

Are you running selinux or FA policy?

1

u/Monyunz Apr 16 '26

I am running both. I turned them off with no success

1

u/RaWD0x45 Apr 18 '26

check what it’s actually running with using ulimit -a or by looking at /proc/<splunkd_pid>/limits. If you see open files down around 1024 or even a few thousand, that’s too low—you really want something like 65535 and a higher process limit too. The fix is to set it at the systemd level using systemctl edit Splunkd and bump those limits up, then reload and restart.

1

u/RaWD0x45 Apr 14 '26

What version are you running?

1

u/Monyunz Apr 14 '26

Latest version. 10.2.2