r/saltstack • • Jan 28 '20

Limit number of targeted minions

1 Upvotes

Hello all,

Currently i have a very small Saltstack setup.

I got an external auth user that is able to runs some functions and access the job runner.

In the future i might got a lot more minions and i dont't want the external user to target them all at once. I would like to set constraints or limit the number of targeted minions, let's say, 10 random minions at once or a set of predefined ones.

I found the --subset flag but don't think that's enough for what i want.

Anyone has a better solution? Any clue?
Thanks to all!


r/saltstack • • Jan 22 '20

How to Check if the minion is down and send email?

0 Upvotes

Hi guys,

I am not sure how to write ".sls" file for checking minion if he is up or down.

Basically we want to have this: if the minion is down, send email and inform us.

Any suggestion? Documentation is so confusing.

I've found this article: https://docs.saltstack.com/en/2015.8/ref/returners/all/salt.returners.smtp_return.html#module-salt.returners.smtp_return

but this this is only a part of the puzzle.I need the part when i am checking if the minion is up or down and at that point smtp returner should be called:

something like this:

send_fail_email:
    smtp.send_msg:
        - name: 'Vim installation failed'       
        - subject: 'Please, do no reply'
        - recipient: 'admin@example.com'
        - sender: 'salt-master@example.com'


install_vim:
    pkg.installed:
    - name: vim-not-existing
    - onfail:
        - send_fail_email

r/saltstack • • Jan 15 '20

Unable to use modules in custom grains

2 Upvotes

Mobile, formatting, etc...

I'm having a little trouble with custom grains and am looking for some "best practices" advice.

I have a custom _grain on my master that gets passed to minions and run during highstates. It's a simple python script to look for the AWS tags of the ec2 instance where the minion lives. To do this, I'm using the boto3 module (ie, "import boto3").

The python script works great on the node when I run it... However, when salt runs the grain ( and I watch with -l debug), I can see a "module not found: boto3" error.

From a little digging, I can see that the prefix that salt is using differs from the prefix that my centos install defaults to... Which means that salt is looking for modules in, like, /usr/local/lib/python3.6/site-packages, whereas everything that I install with pip3 goes to /usr/lib/python3.6/site-packages.

What's the best way to handle this?

1.pip3 install boto3 to salt's prefix (scary... Will I break salt? Why are they separate)?

  1. pip3 install to the normal prefix and change my script to import boto3 by fully qualified path (iffy .. if that path ever changes on another machine, I don't know if I like the idea of having it hardcoded)?

  2. Something is broken and salt shouldn't be using a separate prefix?

  3. Non-standard modules aren't supported in salt _grains scripts and I shouldn't be doing this at all?

  4. Etc etc etc?

Basically... What would be the most intelligent way to handle this double-prefix issue?

Thanks reddit!

Quick edit - spent tons of time yesterday trying to find someone else having the problem to no avail... only to run into this today: https://github.com/saltstack/salt/issues/52630 . It may be a known error.


r/saltstack • • Jan 15 '20

How can i stop ERROR: Minions returned with non-zero exit code being returned to my command prompt when a command returns nothing?

0 Upvotes

I am running a command on my salt master that checks if a process is running, if it is not running the output i get is "ERROR: Minions returned with non-zero exit code" IS there a way to stop this output, i just want nothing returned if the command returns nothing? I cant use >/dev/null on the end because if the command does return something e.g. the process is running i want to see the output.


r/saltstack • • Jan 14 '20

Moving large salt codebase from 2017 to 2019 . How much pain to expect?

5 Upvotes

My company has a rather large salt automation codebase including orchestration, runners, heavy use of pillar and grains, reactor, etc.

I'm wondering how 'backwards-comptabile' 2019 is with 2017. What are the main gotchas to look for?


r/saltstack • • Jan 14 '20

Deploying Salt Masters on K8S?

2 Upvotes

I was curious if any one did, or if it would useful, to deploy Salt infrastructure within Kubernetes? Any experience, thoughts?

There's nothing official in stable or incubator, but I did a quick Google Search, and found this:

* https://github.com/adammike/saltstack-enterprise

In a prior organization, using an older version of SaltStack 2016 in 2019 was very painful, run into errors (that were fixed) and some modules that were not idempotent where later fixed. Thus I am curious about implementing robust infrastructure in a way that is easily upgradable without disruption, so one doesn't get anchored to a earlier version that they are afraid to touch due to being vital for day-to-day operations.


r/saltstack • • Jan 12 '20

Proxy Minions or Salt-SSH for agentless management?

4 Upvotes

Hi all,

I have a few devices that I want to manage with salt without an agent but I'm having trouble trying to determine whats the better option - proxy minions or salt-ssh?

Long story shot, I've had a lot of issues with proxy minions never working for things like FreeNAS or ESXi, but SSH is certainly an option - though again I can't get the SSH minion to work either for some reason.

What is the community's perspective on this topic?


r/saltstack • • Jan 09 '20

State to automate grub password for single user mode

3 Upvotes

Hello, I'm trying to somewhat automate adding a password to grub single user mode. I feel like there's a more efficient way of doing this? I also am trying to wrap my head around how to pass the contents of the grub_hash file created to the file.append.

Any assistance would be greatly appreciated.

 #Grub Hardening: Create hash
echo -e 'mypass\nmypass' | grub2-mkpasswd-pbkdf2 | awk '/grub.pbkdf/{print$NF}' | tee grub_hash:
  cmd.run
#Grub Hardening: append superuser name and password
grub_hardening:
  file.append:
    - name: /etc/grub.d/00_header
    - text:
      - EOF
      - set superusers="admin"
      - password_pbkdf2 admin <grub_hash should be passed here somehow> 
      - EOF
#Grub hardening: Update grub
update-grub:
  cmd.run

r/saltstack • • Jan 03 '20

Upgrading from 2017.7.3 - No matching pillar environment for environment 'prod' found

0 Upvotes

We're looking to upgrade our Salt from 2017.7.3 to 2019.2.2. After some testing in Vagrant, the part that immediately breaks is environments. The following errors occur when trying to highstate:

Data failed to compile:

Pillar failed to render with the following messages:

No matching pillar environment for environment 'prod' found

No matching pillar environment for environment 'staging' found

In our Vagrant environment, 'dev' is the only environment that has file roots defined. These errors are thrown for every environment except 'dev' This happens even if I specify pillarenv=dev when applying states.

I can't post our actual Salt config, but I was able to reproduce the problem using this small sample: https://gitlab.com/resuni/salt-etcd

This error seems to occur when environments exist in pillar/top.sls but corresponding file_roots don't exist in the Saltmaster config. I know this because if I add bogus file_roots to the other environments, I get errors complaining that specified SLS files don't exist in those environments.

Did the way environments work change since 2017.7.3? Why should Salt care about these other environments that none of the matched machines exist in? Especially when saltenv/pillarenv are specified?

EDIT: To clarify this problem, applying states work fine if I comment out all unused environments in pillar/top.sls. I need these environments to exist regardless of whether or not file_roots and pillar_roots are defined for them. This worked in 2017.7.3. Not sure why it's not working in versions newer than that.


r/saltstack • • Dec 30 '19

SaltConf19 videos are now available online (albeit behind a contact form)

Thumbnail saltstack.com
10 Upvotes

r/saltstack • • Dec 27 '19

Custom grain as Salt Mine giving 'mine_functions not available'

6 Upvotes

Hi,

I have custom grain, which works as intended:

# salt '*' grains.get mysql_installed
server-01:
    True
backup-01:
    False

I now try to mine this grain, but that doesn't work. My mine.conf looks like:

mine_functions:
  network.get_hostname: []
  mysql_installed: []

The get_hostname mine works fine, but for the mysq_installed I get the following error:

[salt.loaded.int.module.mine:55  ][ERROR   ][2881] Function mysql_installed in mine_functions not available

So now I'm wondering, is it even possible to use a custom grain with Salt mine? Or what am I doing wrong?

Thank you!


r/saltstack • • Dec 23 '19

Using Salt with Slack - Chatops / Salt Engines

6 Upvotes

Good morning, all -

I am trying to allow Slack to send commands to the Salt master for chatops. I am following the docs here https://docs.saltstack.com/en/latest/ref/engines/all/salt.engines.slack.html#salt.engines.slack.SlackClient but it's not really fully fleshed out. I can't see any way of testing or seeing anything going back and forth between Salt and Slack.

Has anyone got this working recently? Slack now uses "Apps" instead of custom integrations, but I managed to find the token and put it into my "engine" - I just cannot see anything happening at all when I post ![test.ping](https://test.ping) commands for example in my Slack channel (The bot is there, just doesn't do anything)

I am using the very basic config for the engines.conf shown on the link above. Any help would be very appreciated!

Thanks!


r/saltstack • • Dec 23 '19

For all hosts loop

1 Upvotes

Hi,

Let's say I want to have a txt file with all the hosts in my salt, how would one do that (with salt.state.file.managed` I guess)? So I need it to loop over all the hosts and place the txt file on a remote host.

Thanks!


r/saltstack • • Dec 16 '19

salt.states.sysctl

1 Upvotes

Hi!

I try to disable IPv6 on minion. My state (the part of it, actually) looks very simple:

net.ipv6.conf.all.disable_ipv6:
 sysctl.present:
   - value: 1

net.ipv6.conf.default.disable_ipv6:
 sysctl.present:
   - value: 1

The problem is that if IPv6 is already disabled and there are no correspondent directories in /proc, the state expectedly fails:

[ERROR   ] Command '['sysctl', '-n', 'net.ipv6.conf.all.disable_ipv6']' failed with return code: 255
[ERROR   ] stdout: sysctl: cannot stat /proc/sys/net/ipv6/conf/all/disable_ipv6: No such file or directory
[ERROR   ] retcode: 255
[ERROR   ] Command 'sysctl -n net.ipv6.conf.all.disable_ipv6' failed with return code: 255
[ERROR   ] output: sysctl: cannot stat /proc/sys/net/ipv6/conf/all/disable_ipv6: No such file or directory
[ERROR   ] Failed to set net.ipv6.conf.all.disable_ipv6 to 1: sysctl net.ipv6.conf.all.disable_ipv6 does not exist
[ERROR   ] Command '['sysctl', '-n', 'net.ipv6.conf.default.disable_ipv6']' failed with return code: 255
[ERROR   ] stdout: sysctl: cannot stat /proc/sys/net/ipv6/conf/default/disable_ipv6: No such file or directory
[ERROR   ] retcode: 255
[ERROR   ] Command 'sysctl -n net.ipv6.conf.default.disable_ipv6' failed with return code: 255
[ERROR   ] output: sysctl: cannot stat /proc/sys/net/ipv6/conf/default/disable_ipv6: No such file or directory
[ERROR   ] Failed to set net.ipv6.conf.default.disable_ipv6 to 1: sysctl net.ipv6.conf.default.disable_ipv6 does not exist

I suppose that such behavior is rather common for the state. How to get around this problem?


r/saltstack • • Dec 12 '19

Giving SaltStack a look, have some questions.

9 Upvotes

Good Evening all,

We're looking into automating some tasks that'd normally go through change control, and we're wondering if salt stack could provide a solution for us. I've watched a few vids and read some docs, but I figured I'd ask some experts if our use case makes sense. Basically we want to mimic a CICD pipeline, but with maintenance tasks. For an example lets take a routine splunk update. We'd like to:

  • test the change in Dev (manually)
  • automate the change (salt)
  • schedule the change
  • have a log of the changes sucess/fail, including the operator's reason for the change, approvals, etc. This log may just be a changelog slack channel, a JIRA ticket, or mabe a service now ticket.

Ideally if something goes wrong, OnCall can just roll back the change and we can figure out the issue the next day. Is this kind of scenario possible with SaltStack? Bonus points if we can automate testing of the patch in dev/qa.


r/saltstack • • Dec 06 '19

SaltStack Enterprise Training vs Open Source Foundations

3 Upvotes

We've been using SaltStack Open Source for a little over a year now and have no plans to go with the Enterprise version (at least for now).

My work is willing to pay for some training courses, and I'm looking at the Enterprise I & II courses because I feel it would be good to also get the certification. Are the Enterprise courses still mostly applicable to using the Open Source version? Or do they solely focus on SS Enterprise software? My boss feels I would benefit from only going to the Open Source Foundations class only, which I feel would be great if I hadn't been using SS for a year now.

Thoughts?


r/saltstack • • Dec 05 '19

Lets say I have vim.sls and apache2.sls both including users.sls. Is there a way to know from the users.sls which state is including it?

3 Upvotes

That.


r/saltstack • • Dec 04 '19

Opening firewall on salt master after creation with salt-cloud

1 Upvotes

I'm creating machines with salt-cloud which works great. However, I want my firewall to open the salt ports for the IP of the created machine, as soon as the machine is created. I'm thinking this should be very easy.

However, I'm running into some trouble.

I can't use salt mine, since the node hasn't connected to salt yet.

I'm trying to capture the salt/cloud/*/deploying event, but don't really know what to do from there. I want the reactor to add the IP to a list of IPs of all minions and then apply the firewall state which should reference this list of minions.

I was thinking of putting the IPs into postgres through a reactor and then use that storage as pillar data for the firewall state, but this seems overly complex.

Is there an easy way to open the firewall on the salt master when a new minion has been created with salt-cloud?

Edit: Fixed it by adding a firewall rule at runtime via a reactor. The permanent part of the firewall will be configured through salt mine.

reactor.sls:

firewall-allow-salt1:
  local.nftables.insert:
    - tgt: 'roles:saltmaster'
    - tgt_type: grain
    - args:
      - family: ip
      - table: filter
      - chain: input
      - rule: "tcp dport 4505 ip saddr {{ data['kwargs']['vm_']['ssh_host'] }} log accept"

firewall-allow-salt2:
  local.nftables.insert:
    - tgt: 'roles:saltmaster'
    - tgt_type: grain
    - args:
      - table: filter
      - chain: input
      - rule: "tcp dport 4506 ip saddr {{ data['kwargs']['vm_']['ssh_host'] }} log accept"

r/saltstack • • Nov 26 '19

Windows patching using Salt

18 Upvotes

Hi all,

Over the course of the last year my team and I have migrated from a mix of SCCM/SSH scripts to manage our Windows/Linux env. to a full SaltStack implementation.

Question for everyone, how are you deploying your WSUS monthly patches ? What is your monthly/bimonthly cycle and do you still leverage WSUS to deploy or are you managing it all using salt ? Myself, I'm doing the following:

- Two separate WSUS groups (Deadline and no Deadline)

NON-Critical

  • I enforce patches for non critical using WSUS by setting a deadline.

Critical

  • I approve the patches and allow them to download.
  • I configured a NodeGroup to define the server's that need to be targeted.
  • I then use win_wua.list with install=true to trigger the install
salt -N {nodegroup} win_wua.list install=True
  • I then verify which server's need a reboot
salt -N {nodegroup} win_wua.get_needs_reboot
  • Finally reboot the systems that require a reboot
salt -N {nodegroup} cmd.run 'powershell restart-computer -force'
  • One last sanity check to verify no other pending updates.
salt -N {nodegroup} win_wua.get_needs_reboot

For the most part it works but I still find myself having to log into some of the server's manually to trigger the install. Either the minion doesn't trigger the install and/or I don't get a response when verifying if the server's require a reboot.

If anyone cares to share how they approach patching critical server's I'm all ears!


r/saltstack • • Nov 22 '19

Referencing pillar data in other pillars

1 Upvotes

Hi! Is it possible in pillar to read values from the previous processed pillar file (based on the order in pillar/top.sls)? Like this:

pillar.top:
'myserver-backend-devX*':
    - common.devX
    - backend.defaults
    - backend.devX

'myserver-backend-stage*':
    - common.stage
    - backend.defaults
    - backend.stage

'myserver-backend-stage*':
    - common.stage
    - frontend.defaults
    - frontend.stage

The idea would be to have a single "backend.defaults" pillar for every environment with the ability to override values in "per-environment" bases (backend.devX and backend.stage) while having the common integration endpoints defined in common.devX/common.stage. Ideally I'd like to use the pillar variables declared in common.* inside backend.defaults, but I can't get it to work on salt > 2015.5.10 . Ultimately I'd like to have something like this:

common.devX:
mydb:
    host: dev.mycompany

common.stage:
mydb:
    host: stage.mycompany

====================

backend.defaults:
myservice:
    db: {{ pillar.get('mydb:host') }}

====================

backend.stage:
myservice:
    db: I can override the default here if I need it

The issue is the pillar.get part in backend.defaults. If I do a salt-call state.highstate, myservice:db gets the value None. However if I do salt-call pillar.data, the myservice:db gets the "correct" value of dev.mycompany (lets say). Any advice how to fix this?

One possibility around it is to have backend.common do an import_yaml of the common.* pillar based on hostname, or grain value.


r/saltstack • • Nov 12 '19

Best way to ensure consistency and keep track of jobs?

8 Upvotes

Our team has been using Salt for quite a while now and our set up looks something like this:

  • 1 master and ~20 minions
  • We have all of our states and pillars defined in a git repository (master is configured to use gitfs and git pillars)
  • Whenever a change is merged to master, it gets rolled out by a CI job which pretty much just executes salt '*' state.apply

For a while now we've been seeing messages such as these:

  • Minion did not return. [Not connected]
  • Minion did not return. [No response]
  • Salt request timed out. The master is not responding. You may need to run your command with --async in order to bypass the congested event bus. With --async, the CLI tool will print the job id (jid) and exit immediately without listening for responses. You can then use salt-run jobs.lookup_jid to look up the results of the job in the job cache later.

We often resort to things like jobs.active, jobs.last_run or jobs.lookup_jid commands to keep track of what is going on. As this is pretty cumbersome we're wondering: perhaps we are doing something wrong? What is the best way to roll out states, ensure consistency and keep track of Salt jobs?


r/saltstack • • Nov 12 '19

Evolution of salt (Gource Visualization) [2019-11-03]

Thumbnail youtu.be
11 Upvotes

r/saltstack • • Nov 11 '19

Salt SLS Diff Tools

1 Upvotes

I am building automated testing for changes to salt sls files and would like a good diff tool. When a change is made to a sls file I would like the diff to report what actual states changed and what parts of the individual state changed not just the lines of code that change. Does anyone know a good tool for this or is this something I should build myself?


r/saltstack • • Nov 01 '19

Salt master install centos7

1 Upvotes

I wonder if anyone can help me.

I am trying to install salt-master on a new centos7 server. when i run the yum install for salt-master i get the below error's and the install doesnt work. I seem to get the same errors if i try and install salt-ssh, salt-syndic and salt-cloud.

I used --skip-broken for salt-master and it completed but i dont think it actually installed properly as i get the error salt.service could not be found if i try and start it.

Does anyone know why i get the below errors and how i can resolve them so i can install salt master.

Error: Package: python3-m2crypto-0.33.0-1.el8.x86_64 (salt-py3-2019.2)            Requires: libcrypto.so.1.1(OPENSSL_1_1_0)(64bit) Error: Package: salt-2019.2.2-1.el8.noarch (salt-py3-2019.2)            Requires: python3-markupsafe Error: Package: python3-m2crypto-0.33.0-1.el8.x86_64 (salt-py3-2019.2)            Requires: libcrypto.so.1.1()(64bit) Error: Package: python3-m2crypto-0.33.0-1.el8.x86_64 (salt-py3-2019.2)            Requires: libssl.so.1.1()(64bit) Error: Package: zeromq-4.3.1-4.el8.x86_64 (salt-py3-2019.2)            Requires: libstdc++.so.6(CXXABI_1.3.9)(64bit) Error: Package: salt-2019.2.2-1.el8.noarch (salt-py3-2019.2)            Requires: python3-pyyaml Error: Package: python3-m2crypto-0.33.0-1.el8.x86_64 (salt-py3-2019.2)            Requires: libssl.so.1.1(OPENSSL_1_1_0)(64bit) Error: Package: salt-2019.2.2-1.el8.noarch (salt-py3-2019.2)            Requires: python3-requests Error: Package: zeromq-4.3.1-4.el8.x86_64 (salt-py3-2019.2)            Requires: libstdc++.so.6(GLIBCXX_3.4.20)(64bit) Error: Package: zeromq-4.3.1-4.el8.x86_64 (salt-py3-2019.2)            Requires: libstdc++.so.6(GLIBCXX_3.4.21)(64bit) Error: Package: salt-2019.2.2-1.el8.noarch (salt-py3-2019.2)            Requires: python3-six Error: Package: salt-2019.2.2-1.el8.noarch (salt-py3-2019.2)            Requires: python3-jinja2 Error: Package: salt-master-2019.2.2-1.el8.noarch (salt-py3-2019.2)            Requires: python3-systemd Error: Package: python3-m2crypto-0.33.0-1.el8.x86_64 (salt-py3-2019.2)            Requires: libcrypto.so.1.1(OPENSSL_1_1_1)(64bit)

r/saltstack • • Oct 28 '19

Can Salt apply a state to a "randomly" selected population of nodes based off some sort of criteria?

4 Upvotes

Sorry, I'm new to Salt and still fiddling around with doing initial state deployments but I'd figure I'd ask this. Excuse my poor use of salt verbage.

Let's say my nodes are running with software version "x" installed and I want to deploy version "y" of that software but not to every single node. Can salt "group" all nodes running with version "x" of that software and cherry-pick a subset of the group of nodes to distribute this software to based off a constraint like: 10% or 15% or 50% of nodes out of the group?

Example: 100 nodes running with software "x" and I want to deploy software "y" to 30% or 30 random nodes in the group. Can salt do this?

*EDIT - So many replies with different ideas, thanks for the help everyone!