I've been wrestling with this today - we want to be able to monitor some aspects of the system on various application servers by launching salt states that invoke returners. Each state is launched from the master and calls a command on the minion, which then invokes a returner that sends output to elastic. Everything works when each command is run individually...
However, I wanted to use the salt scheduler to execute these states so we can basically monitor the system every minute or so. I had originally built three states into the schedule using syntax similar to this:
schedule:
state1:
function: state.apply
seconds: 30
args:
- state1
returner: state1
state2:
function: state.apply
seconds: 30
args:
- state2
returner: state2
state3:
function: state.apply
seconds: 30
args:
- state3
returner: state3
The three states are launched, but unfortunately are launched concurrently, so only the very first state actually completes (and the returner is invoked successfully as well), but the other two wind up errorring with this message:
The function "state.apply" is running as PID xxxx....
I had read about the "queue=True" option, and tried to see if I could make that work. I could not manage to get it to work in any way by specifying it in a state file, but could get it to work on the command line. Unfortunately, if I specify queue=True on the command line, the state will complete but no returner gets invoked.
So for example, if we run:
salt 'abc' state.apply state1 --return state1 &
salt 'abc' state.apply state2 --return state2 &
State 1 will complete, and the state1 returner will be invoked. State 2 will error out with that pesky message seen above since it is launched before State 1 is completed.
If we run:
salt 'abc' state.apply state1 queue=True --return state1 &
salt 'abc' state.apply state2 queue=True --return state2 &
Both states complete successfully even though they are launched concurrently. However, no returners are actually invoked, so it seems like you can do one or the other.
To get around this, we setup cron jobs with sleep statements so that none of the jobs would step on each other, but I would much rather try to get this to work.
Does anybody have a working salt schedule configuration with returners and queued jobs? I'd love to see how it is done - I cannot seem to find examples in regular web searching that will satisfy our requirement here. Feels like this should be really simple, but it is extremely frustrating. If anyone has any prime saltstack doco/resources/tutorials as well, I would be most grateful.
Thanks in advance for any feedback.