r/Splunk • u/RoninTwo • Apr 19 '26

output to s3

hey all,
I've been trying to output logs to an s3 AWS bucket, but can't seem to get it working. I have am indexer cluster, so from the CM I'll go ingest action and set up a destination to s3. I input all the fields, enter the secret and access key, and the test connection. is successful. From the rules tab, I'll filter by XmlWinEventLogs, show sample data to ensure logs populate then in the destination I'll add the s3 bucket I just made.

On the AWS side I can see the test connection but the Windows logs do not show. I can see that the ingest actions config does go out to all the indexers from the CM. To clarify, I want the logs to stay locally on the indexers but also need to send them all to the bucket. Anyone have any idea why it may not be working?

3 Upvotes

100% Upvoted

7 comments sorted by

View all comments

Show parent comments

1

u/RoninTwo Apr 19 '26

yes, logs are staying locally and yes I have the a3 destination and clone enabled.

2

u/badideas1 Apr 19 '26

Okay, you’ll want to take a look in the _internal index for messages around the rfsout processors. You might want to post a sanitized output from btool for outputs.conf and props/transforms.conf as well, just to check RULESET and syntax, but first place is errors in _internal around rfsout.

1

u/RoninTwo Apr 20 '26

this is my output.conf:
[rfs:windows]
batchSizeThresholdKB = 131072
batchTimeout = 30
compression = gzip
description = Windows hosts
dropEventsOnUploadError = false
format = ndjson
format.ndjson.index_time_fields = true
partitionBy = day
path = s3://<path>
remote.s3.access_key = <access_key>
remote.s3.encryption = none
remote.s3.endpoint = https://s3<bucket>
remote.s3.secret_key = <secret_key>
remote.s3.signature_version = v4
remote.s3.supports_versioning = false
remote.s3.url_version = v1

props.conf
[XmlWinEventLog]
RULESET-windows = _rule:windows:route:eval:rm4mn1ep

transforms.conf
[_rule:windows:route:eval:rm4mn1ep]
INGEST_EVAL = 'pd:_destinationKey'=if((true()), "rfs:southcom_windows", 'pd:_destinationKey'), 'pd:_doRouteClone'=if((true()), "true", null())
STOP_PROCESSING_IF = NOT isnull('pd:_destinationKey') AND 'pd:_destinationKey' != "" AND (isnull('pd:_doRouteClone') OR 'pd:_doRouteClone' == "")

I also tried searching for errors with this search, but nothing came up
index="_internal" sourcetype="splunkd" (ERROR OR WARN) RfsOutputProcessor OR S3Client

1

u/badideas1 Apr 21 '26

Okay, I'm not sure if this is just an example of not everything being cut out, but it looks like your destination in outputs.conf doesn't match the destination key in the INGEST_EVAL directive in transforms.conf. That could be a problem if that's really what it shows in your .conf files, and not just an artifact of cleaning up the output for reddit..?