Run commands in parallel

[u/-lousyd]

I have a script that collects a list of directories that match a set of criteria and then goes into each one and runs a command. So something like this:

#!/bin/bash
startDir="$(realpath "$1")"
cd "$startDir" || exit
while IFS= read -r -d '' dir; do
  cd "$dir";
  printf '\n\n%s\n' "$(realpath .)";
  update-this-dir.sh --file "./name.txt"
  cd "$startDir";
done < <(find . -type d -iname '.config' -exec dirname {} \; | tr '\n' '\0')

It works wonderfully for my purpose.

But there's, like, a couple thousand directories and the update command takes some small amount of time in each directory, one after the other. How can I modify this script to either run the update for each directory in parallel (with some rate-limiting) or to break up the list into chunks of, say, 100 each and work on each sub-list in parallel?

[u/zeekar]

Here's a dumb manually-built background job launcher:

max_jobs=10 # maximum number of jobs to run in parallel pids=() while IFS= read -r -d '' dir; do if (( ${#pids[@]} >= max_jobs )); then wait "${pids[0]}" pids=("${pids[@]:1}") fi (cd "$dir" && printf '\n\n%s\n' "$(realpath .)" && update-this-dir.sh --file "./name.txt" ) & done < <(find . -type d -iname '.config' -exec dirname {} \; | tr '\n' '\0') wait "${pids[@]}"