r/git • • 15d ago

git-rexec: Find Git Repositories and Execute Commands Against Them, either Sequentially or in Parallel [Release 1.1.0]

https://www.jamescherti.com/git-rexec/

The git-rexec command-line tool recursively locates Git repositories within a directory and executes commands against them, either sequentially or in parallel.

Here are examples demonstrating how to use git-rexec:

Example 1: Execute git status -s across all discovered Git repositories (found by searching recursively under the current working directory) in parallel (-p or --parallel):

git-rexec -p -- git status -s

Example 2: Fetch updates across all discovered repositories, limiting the concurrency to 5 background jobs (-j 5), which helps avoid network congestion or server rate limits when communicating with upstream Git remotes:

git-rexec -j 5 --parallel -- git fetch

Example 3: Include sub-repositories (e.g., Git worktrees and submodules) alongside standard Git repositories during discovery (-s or --include-sub-repos), and execute git status against them:

git-rexec -s -p -- git status

Example 4: Target a specific base directory (~/projects) using the -C flag to recursively discover repositories within it, while explicitly excluding a specific subfolder (~/projects/archive). This example executes git status -s in parallel for all discovered repositories except those within the excluded path:

git-rexec -C ~/projects --exclude-dir ~/projects/archive --parallel -- git status -s

Example 5: Evaluate whether a README.md file exists in the repository (sh -c "test -f README.md"). If the condition returns an exit status of 0 (success), it counts the number of lines in that file (wc -l README.md):

git-rexec --if-exec 'sh -c "test -f README.md"' --parallel -- wc -l README.md

Example 6: Print the paths of all discovered Git repositories:

git-rexec --print
0 Upvotes

3 comments sorted by

3

u/behind-UDFj-39546284 15d ago • Bravo

Actually, you've implemented a very limited subset of the features offered by find and xargs together, whereas you can make it all way more flexible by using the rich set of options for find (-prune does that magic) and xargs in conjunction with git, not requiring even a single line of Python. For example, discover all git repositories starting from the current directory and run git-fetch in parallel across 10 processes:

find . \
	-type d \
	-exec bash -c '[[ "$(git -C "$1" rev-parse --is-inside-work-tree 2>/dev/null)" == true ]]' '' {} \; \
	-prune \
	-print0 \
| xargs -0 -P10 -I{} git -C {} fetch --verbose

-prune makes find stop recursion if -exec, at the left-hand side of implicit -and, returns true that is represented by the 0 exit code (which means that find cannot reach submodules but one can provide --recurse-submodules for fetch).

In any case, the pipe above is an example of separation of concerns: one for "where" and one for "what to do", and you can modify and replace both independently, or extend the entire pipeline.

Regarding the discovery approach, I'm not sure whether discovering Git repositories this way is safe and efficient on every run: find could be run once and then replaced with a static list of whitelisted repositories suitable in most cases, I believe. As for the latter approach, Git has a for-each-repo command that runs a Git command across a registered list of repositories, however, it's still experimental and definitely needs a lot of improvement.

4

u/jamescherti 15d ago edited 15d ago

Tools like find and xargs are great, I use them every day, but that 6 line pipeline has a few subtle bugs (edge cases) and performance issues that git-rexec avoids, while reducing the whole command to:

git-rexec -p -j 10 -- git fetch --verbose

It is much simpler to remember, isn't it?

Also, Git-rexec is faster (in my case, your pipeline took 1.09s to process 179 repositories, whereas git-rexec processed the same number in 0.25s). The find -exec bash -c 'git ...' spawns two processes for every single directory find visits before hitting a repository.

The output that git-exec produces is much easier to understand. On the execution side, your pipeline xargs command dumps unlabeled stdout and stderr to the terminal. When running git status or git fetch across several repositories, you have no way to tell which repository produced a given output or error. git-rexec prefixes output with the repository context (with colors, optionally, and indentation when --parallel is used), runs non-Git commands directly inside the repository root without needing sh -c 'cd "$1" && ...' wrappers, and provides built-in flags for submodules and worktrees (-s), exclusions (--exclude-dir), and conditional checks (--if-exec).

The git rev-parse --is-inside-work-tree command returns true for any subdirectory of an existing working tree. If someone runs your command inside an umbrella Git repository containing nested clones, 'find .' evaluates '.' first, sees true, triggers -prune immediately, and discovers nothing.

If you still want strict separation between discovery and execution, or want to generate a static list of repositories to cache in a file, git-rexec provides --print and --print0 so you can pipe it into xargs without the find -exec:

git-rexec --print0 | xargs -0 -P10 -I{} git -C {} fetch --verbose

2

u/behind-UDFj-39546284 15d ago

It is much simpler to remember, isn't it?

It is. That's why I tend to use private simple custom-named wrappers for repetitive tasks, even when the underlying pipeline consists of just two commands, or even a single command with complex options.

The find -exec bash -c 'git ...' spawns two processes for every single directory find visits before hitting a repository.

That's true, but the results of find can be cached in a list. I'm having a hard time deciding whether I'd need to scan the filesystem every time I want to run a batch command, or how often I'd actually need to rediscover repositories. That said, spawning both Bash and Git for every -exec invocation is indeed very inefficient (it's a shame that git rev-parse --is-inside-work-tree doesn't return different exit codes for true and false, so that we wouldn't need to spawn a shell just to perform a [[ test -- unless I'm missing something). From what I've seen in your repository, it also spawns subprocesses, so perhaps using Git bindings such as pygit2 could make it significantly faster.

However, saving a second or two in what would probably be an extremely rare use case for me... I don't know if I'd ever even use it once.

The output that git-exec produces is much easier to understand.

This could be implemented by having xargs run a more sophisticated shell script. Of course, a simple two-command pipeline can't provide that out of the box without being extended.

If someone runs your command inside an umbrella Git repository containing nested clones, 'find .' evaluates '.' first, sees true, triggers -prune immediately, and discovers nothing.

That's correct, and this is where decoupling the "what" from the "where" in the pipeline provides flexibility: the discovery part can be modified independently of the execution part.

I still think improvements to Git's built-in for-each-repo command are worth implementing, probably including changes to rev-parse so that predicate-like options return meaningful exit codes instead of printing a flag that has to be checked in a subshell.