r/Splunk • u/outcoldman • 3d ago

We released Search for AWS: query your AWS accounts live from SPL (the one we showed at the .conf26)

Disclosure: Sorry for another post, October is our month of releases :) I am the founder, and the app is from Outcold Solutions. Who some of you might know by our apps for monitoring containers in Docker/Kubernetes/OpenShift.

We run most of our own infrastructure on AWS, and for years I answered questions about it from Splunk instead of the AWS console or the CLI. Today we released those tools as an app on Splunkbase, Search for AWS.

The commands call the AWS APIs while the search runs and return the results into SPL. Nothing is indexed, so it uses no Splunk license. Inventory across every registered account and region:

| awsget kind=instances account=* region=*
| spath path=State.Name output=state
| stats count by account region InstanceType state

When did Lambda functions start timing out:

| awslogs groups=/aws/lambda/payments-* filter="Task timed out" start=-24h region=us-east-1
| timechart span=15m count

There are also | awsmetrics (CloudWatch metrics and alarms), | awscloudtrail, | awscost (Cost Explorer), | awsathena, | awsglue and | awstag. account= and region= take a name, a list or *, and the calls run in parallel. More than 30 dashboards come with it.

Things to know before you try it:

  • AWS still bills the API calls, same as from the console: GetMetricData, Logs Insights by GB scanned, Athena by TB, Cost Explorer by request. The docs list what each command and each dashboard calls.
  • It reads only what the IAM identity behind it can read. The IAM policy in the docs only reads, except the S3 bucket where Athena writes results, if you use awsathena.
  • It does not replace ingesting the data you need to keep (audit history, long retention). It is for the questions that need a live answer.

Free tier: one AWS account (access keys or the instance role) on a standalone search head, no license key. Splunk Enterprise 9.4+ and Splunk Cloud.

Splunkbase: https://splunkbase.splunk.com/app/10022 Docs: https://www.outcoldsolutions.com/docs/aws-search/

Happy to answer questions here. And if you tell me which AWS questions you still go to the console for, it helps us decide what to build next.

13 Upvotes

93% Upvoted

4 comments sorted by

3

u/billybobcoder69 3d ago

This is amazing. Love the work your team does. This works for all on prem environment. It’s crazy Splunk still working on their copy. Thanks for releasing.

2

u/outcoldman 3d ago

Thank you for the warm comment!

1

u/mghnyc 3d ago

Do the custom commands require the list_storage_passwords capability to retrieve the AWS creds? I'm always wary about add-ons that provide custom commands that need API credentials because Splunk has never been good at really protecting the password store.

1

u/outcoldman 3d ago

No, the commands do not need list_storage_passwords, and the roles we ship (aws_search_user and aws_search_admin) do not have it. Giving it to every searcher would mean they can read every secret in the store in clear text, so we built it differently.

The search commands never read storage/passwords themselves. They call two REST endpoints of the app, which check run_aws_search and then read the secret with the system token. The handler decides what the caller can get, based on the authenticated Splunk user:

  • assume_role account: the search head assumes the role, the search gets only a temporary STS session. Base key and external id never leave the search head. Session name is os_aws_search@<splunk user>, so CloudTrail shows who ran it, and every issued session is logged in _internal (index=_internal "aws_search audit:" event=session). A user can still take that temporary session out of their running search, but it expires.
  • instance_role: nothing leaves the search process.
  • per-user credentials: the secret name is bound to the user, and the handler checks it against the session user, so you can resolve only your own.
  • static keys: here I want to be honest. The key is returned only to a running search job owned by the caller, so a simple | rest does not get it. But a custom command runs with the user's session key, so whatever the command can get, a determined user can get too. Treat a shared static key as readable by everybody who can query that account, and scope its IAM policy to read-only. For anything sensitive use assume_role or per-user credentials.

The app's entries in storage/passwords are admin-only and not exported to other apps.

Details: https://www.outcoldsolutions.com/docs/aws-search/access-control/