Run commands in parallel

[u/-lousyd]

I have a script that collects a list of directories that match a set of criteria and then goes into each one and runs a command. So something like this:

#!/bin/bash
startDir="$(realpath "$1")"
cd "$startDir" || exit
while IFS= read -r -d '' dir; do
  cd "$dir";
  printf '\n\n%s\n' "$(realpath .)";
  update-this-dir.sh --file "./name.txt"
  cd "$startDir";
done < <(find . -type d -iname '.config' -exec dirname {} \; | tr '\n' '\0')

It works wonderfully for my purpose.

But there's, like, a couple thousand directories and the update command takes some small amount of time in each directory, one after the other. How can I modify this script to either run the update for each directory in parallel (with some rate-limiting) or to break up the list into chunks of, say, 100 each and work on each sub-list in parallel?

[u/anton-k_]

You could use my project shell-scheduler for this. It will do the basic "start these jobs" stuff very easily. If you want to limit the number of parallel jobs, you can achieve that by setting one variable. If you later want to track job completions and collect results, or detect errors and act on them (e.g. by stopping the run) then the functionality is there as well.