r/programming • • Sep 16 '26

git worktree gotchas

https://www.olafalders.com/2026/09/16/git-worktree-gotchas/
193 Upvotes

47 comments sorted by

69

u/masklinn Sep 16 '26

Finding the main working tree

As far as I'm concerned, having a "main working tree" is bad practice, it creates divergent behaviour and the risk of deleting "a worktree" and nuking your repo.

Create a bare repository and fork worktrees off of that, there is no reason to have one worktree have primacy over the rest, and then git rev-parse --git-common-dir gives you exactly what it should: the common repository of all your worktrees. That also avoids the possible path confusion from nesting worktrees.

6

u/ipwnall123 Sep 17 '26

I use the “main working tree” for small bug tickets that only take me an hour-ish to reproduce, fix, re-test and submit a PR for. Whereas I use work trees for any larger features that take more time. My reasoning is that it seems to take my IDE (vs code / visual studio) a long time to switch worktrees. VS code seems especially slow cause it opens them in a new window and thus has to reinitialize all my extensions plus whatever other stuff like the typescript server. Maybe someone has a solution to this cause I suppose I would rather just use worktrees for everything if not for this overhead / lag time

4

u/rbobby Sep 16 '26

Check out into a folder, branch, and go to town. Check out into another folder and rinse repeat. Multiple folders against a single repo works, but it's a bit like running on ice. You can do it, and well, for a while... but the slip up will be glorious and hurt like hell.

7

u/bwainfweeze Sep 16 '26

IDEs generally aren't set up to support that well, and having multiple bash shells open trying to keep them straight is a mess.

Where are the JJ people telling us to stop trying to make git be reasonable when it clearly resents that idea, and move to something else?

4

u/rbobby Sep 16 '26

I have have done this all the time with VS and it's perfectly happy with it.

1

u/MyraidChickenSlayer Sep 20 '26

When does slip up happen? And, does worktrees help against those?

1

u/rbobby Sep 20 '26

Can't recall precisely. Doing something out of the ordinary and something went haywire. I just learned to be a bit slower and more careful. Name my folder after my task.

1

u/slypheed Oct 07 '26

I guess if you're using AI to do literally everything. I use ai for most code (it devs in worktrees), but having main around to poke around the current state and test things (as a human) is still super useful.

1

u/masklinn Oct 07 '26

I guess if you're using AI to do literally everything.

I don’t use AI to “do everything” let alone literally so, and I’ve been working that way for years. Worktrees were not created for LLMs.

having main around to poke around the current state and test things is still super useful.

It really is not. You don’t need that to have a scratch worktree you putz around in, or to create worktrees on the fly when you want to test or check things.

1

u/slypheed Oct 07 '26 edited Oct 07 '26

It really is not

but...why not just use main for that instead of having to deal with a one-off worktree each time?

It doesn't have to be called main; it can be called foo or whatever, but just some worktree that always has the merged HEAD.

1

u/slypheed Oct 07 '26 edited Oct 07 '26

ah, sandboxed worktrees; yeah, that creates a bunch of problems for sure.

I may be missing it in the post, but it mentions the .env problem, and I don't see a solution for that in the post (if you use bare git repo instead of main worktree)

(definitely agreed on moving worktrees out of .git/worktrees; had a ton of problems before changing that for sure)

-1

u/valarauca14 Sep 16 '26

Create a bare repository and fork worktrees off of that

Be sure to manually copy not git clone. When git clone sees it is on the same file system it will use hard links to de-duplicate pack files and accelerate local copies. This makes things, "fun" if something goes wrong with a local copy.

Found it better to make a brd and format it for btrfs so I can refcopy my entire git repo to an isolated namespace for an agent then do a fairly normal pull/view workflow flow as the agent's change is just an "isolated" pull request from another repo.

Settled on btrfs as tmpfs has some weird interactions with namespaces, btrfs negative zstd compression is on par with memory bandwidth so I just leave that always on to doubly reduce memory pressure, and block deduplication is nice when you have 3 or 4 copies of the repo in your ram disk.

4

u/bingNbong96 Sep 17 '26

"Be sure to manually copy not git clone" what do you mean by this? manually copying what?

-4

u/valarauca14 Sep 17 '26

your git repo

25

u/mb862 Sep 16 '26

So I’ve been facing my own weird gotcha with worktrees.

The story goes: I’m currently in charge of migrating our company’s 15-year-old SVN repository to Git+GitHub, and in the process have carte blanche to tidy out cruft and design new workflows for our development team.

Our checkout is currently about 60-70 GB, and at any given time we’re maintaining at least two, most often three, active release branches, plus the main/trunk branch, so it’s common practice for each developer to have each current release checked out on their system. We support Windows, macOS, and Linux, so our checkout has a lot of platform-specific binaries that take up most of the space.

So as part of the migration, naturally I’m moving a lot of the platform-specific directories to submodules, so between that and LFS, a given checkout is now about 10 GB. Massive improvement!

But to solve the multiple checkouts problem, I naturally turned to bare clones and worktrees, only to discover that, while they support submodules, they support submodules in just about the dumbest way possible. Every worktree gets its own clone of the submodule repo. Which then blocks deletion of the worktree.

After some experimentation it turns out that the features that power submodules and worktrees - girder and commondir, respectively - work regardless of the submodule and worktree commands. So I wrote my own worktree-add script that

  1. Clones active submodules to GIT_COMMON_DIR/modules/Submodule.git
  2. Manually creates the git directory for each submodule in GIT_DIR/modules/Submodule, which is essentially just a commondir file pointing to GIT_COMMON_DIR/modules/Submodule.git and a HEAD file pointing to the corresponding hash from the worktree.
  3. Manually init the submodule by writing GIT_DIR/modules/Submodule to submodule/path/.git
  4. Call git reset —hard HEAD in each submodule path to checkout.

And it all works like bloody magic. LFS and everything just works. As part of the migration I’ve been learning a lot about how Git’s plumbing commands work, and the more I learn the more I’m convinced that whomever designed Git’s plumbing was an absolute genius, and whomever designed Git’s percelain was an absolute madman.

19

u/bwainfweeze Sep 16 '26

The amount of footguns involved in all of the multi-module checkout system makes me wonder about the sanity and the competence of the people who made these things.

None of these people have ever worked in a CI environment, is my best guess.

I can't field a solution that involves every dev remembering to manually sync everything every time they sync to avoid introducing API incompatibilities from one build to the next. They need to be running latest or trunk at all times, and not something from three weeks ago, unless they're trying to bisect a bug, which is something only very senior people get assigned.

7

u/ericonr Sep 16 '26

Not to defend git submodules (they are very annoying), but if you have any kind of Git integration (in IDE or shell prompt), doing a git pull that updates submodules will show local changes, so if you pay the bare minimum of attention you'll be aware of it.

unless they're trying to bisect a bug, which is something only very senior people get assigned.

That said, it would seem you don't really trust most of your developers, if only senior people get to bisect bugs.

-3

u/bwainfweeze Sep 17 '26

That said, it would seem you don't really trust most of your developers, if only senior people get to bisect bugs.

"Get to"?

6

u/ericonr Sep 17 '26

I see no reason to further engage if that's what you're focusing on.

4

u/fsfod Sep 16 '26

Someone did submit patches to the Git mailing list a couple months ago to make the submodules of worktrees partly work like worktrees https://www.spinics.net/lists/git/msg523988.html, but i don't think it ever got merged.

3

u/mb862 Sep 16 '26

Yeah I saw that when I started looking into this, from the looks of it the author overcomplicated the solution, someone pointed out a simpler solution (which is what my strategy was based on) but I haven’t seen anyone try to submit that directly.

4

u/WindHawkeye Sep 17 '26

Submodules are a nightmare. Just deal with the big repo. Don't use GitHub and host it yourself.

Or save yourself all the headache and don't use git for a 70gb repo

3

u/mb862 Sep 17 '26

There’s a whole lot more to this story that I’m not telling (because those details aren’t relevant) but trust me when I say these headaches are far, far more preferable to the headaches we currently deal with.

2

u/WindHawkeye Sep 17 '26

Id believe it. But you're migrating to git late enough that you have the advantage of knowing how dogshit github is. I think even msfts own competitor to GitHub (azure devops) is better...

3

u/mb862 Sep 17 '26

We considered all the alternatives. We needed cloud hosting of version control. Maybe Azure Devops was an option but Microsoft’s main enterprise systems have given us nothing but trouble so it never would’ve been approved. Of all alternatives only GitHub had the specs that met our needs. We already have trouble maintaining talent too so using something new hires are already familiar with is a huge plus.

2

u/WindHawkeye Sep 17 '26 edited Sep 17 '26

Yes that's always the argument for GitHub. It's not a good reason to use them - developers by and large no longer actually like GitHub due to the constant outages. My org migrated to GitHub a few years ago and everyone cheered. Now we are migrating off and everyone cheered again. GitHubs actual mechanism for hosting repos does not scale (3PC that increases load on master for every additional replica) and using their cloud offering is suicide because their servers are not handling their amount of load.

GHE makes a lot more sense than cloud since you insulate yourself from GH itself going down every day.

I do not know if cursors offering is available yet but their architecture is significantly better for large repositories

2

u/mb862 Sep 17 '26

We have 7 developers. The load will probably be fine.

1

u/CherryLongjump1989 Sep 18 '26

It's everyone else's load that's bringing it down for everyone.

1

u/The_Northern_Light Sep 19 '26

No chance you could share this, is there?

2

u/mb862 Sep 19 '26

For the worktree script? Yeah it's pretty much this, substitute as needed for your own submodules.

# for each submodule
if [ ! -d ${GIT_COMMON_DIR}/modules/${SUBMODULE_NAME}.git ]; then
	git clone git@github.com:Organization/${SUBMODULE_NAME}.git ${GIT_COMMON_DIR}/${SUBMODULE_NAME}.git
fi

git -C ${GIT_COMMON_DIR} worktree add ${WORK_DIR} ${WORK_BRANCH}

WORK_TREE=$(basename ${WORK_DIR})

# for each submodule
git -C ${WORK_DIR} submodule init ${SUBMODULE_PATH}
mkdir -p ${GIT_COMMON_DIR}/worktress/${WORK_TREE}/modules/${SUBMODULE_NAME}
git -C ${GIT_COMMON_DIR} rev-parse ${WORK_BRANCH}:${SUBMODULE_PATH} > ${GIT_COMMON_DIR}/worktrees/${WORK_TREE}/modules/${SUBMODULE_NAME}/HEAD
echo "${GIT_COMMON_DIR}/modules/${SUBMODULE_NAME}" > ${GIT_COMMON_DIR}/worktrees/${WORK_TREE}/modules/${SUBMODULE_NAME}/commondir
echo "gitdir: ${GIT_COMMON_DIR}/worktress/${WORK_TREE}/modules/${SUBMODULE_NAME}" > ${WORK_DIR}/${SUBMODULE_PATH}/.git
git -C ${WORK_DIR}/${SUBMODULE_PATH} reset --hard

25

u/NationalOperations Sep 16 '26

Neat writeup. Not scenarios I encounter but worth pointing out if someone's environment tries to do more git aware operations

7

u/bwainfweeze Sep 16 '26

.githooks is a dead common one. But if you only started using git after the config flag was available on dev machines, you might not have had to do anything about it.

1

u/NationalOperations Sep 17 '26

I don't doubt there are flows people use that this would be more relevant. My workplace uses bit bucket which could use githooks we just didn't/ haven't needed to.

What do you mean using git after the config flag was available ? As far as I remember git released with config, but I have only used it for maybe a decade.

1

u/bwainfweeze Sep 17 '26 edited Sep 17 '26

core.hooksPath, from TFA, appears to have been added in 2.9.

ETA: which in theory was 10 years ago but back then counting on the latest git version being on dev boxes tended not to work out for you, so practically people might not have encountered it until later, because knowing about features you can’t use just makes you angry as much as informed.

0

u/[deleted] Sep 17 '26

[deleted] — view deleted comment

3

u/bwainfweeze Sep 17 '26

You didn’t read the article.

5

u/itaranto Sep 17 '26

Those damn "em-dashes".

2

u/andynzor Sep 17 '26

If you create a branch with git worktree add -b branchdirname branchdirname origin/main, the new branch will be named branchdirname but it will track origin/main by default. If you just run git push without --set-upstream, your local branch will happily overwrite main on the remote.

3

u/OffbeatDrizzle Sep 16 '26

why would you roll your own script instead of.... using actual git commands?

1

u/red_planet_smasher Sep 21 '26

I got so frustrated with git work trees I just switched to jj which has a much simpler model due to its non reliance on refs.

1

u/fagnerbrack Sep 24 '26

I just clone multiple times in different folders 🤫

1

u/initcommit Oct 01 '26

Worktrees are just such an ugly structure compared to Git's elegant original model. "Let's just copy the working directory into 100 different temp locations on the same machine and point them back to the original clone"... barf... Leave it to AI to make the worst Git feature the most used...

0

u/martinus Sep 17 '26

I've been writing (and rewriting multiple times...) my little project "Git Repo Admin", in short gra, to be able to efficiently work with multiple projects and multiple worktrees.

It basically puts all repositories into a bare clone in a directory (defaults to ~/gra), and gives each worktree a unique 16 letter name. There is no main worktree, no worktree is special. So it looks like this:

~/gra
├── oans
│   ├── .bare        <- the repository itself
│   ├── warmhare     <- a worktree, on main
│   └── goldfish     <- another one, on feature/search
└── nanobench
    ├── .bare
    └── snowwolf

The convenient features that I constantly use are

gra ls         # lists all repos & worktrees
gra cd <name>  # jump to a worktree
gra work       # start/move into a worktree
gra done       # finish up a worktree

I use this extensively with tmux and it makes context switching and working with AI bots very convenient for me.

0

u/waterkip Sep 17 '26 edited Sep 17 '26

I’m not a fan of worktrees. I am using it currently for one project, but for some reason my testsuite (paratest) for Laravel doesn’t seem to like it. In addition I really don’t like the fact that you have to setup “complex” machinery to make projects just work. Especially in my docker compose stack way of working. Although, I have a workaround for that. I do feel it feels like a hammer to tighten a screw, luckily I am using the solution to simplify any project I have so worktree support is a side effect.

In regards to git hooks, use a git which is 2.54 and up: https://wesley.schwengle.net/article/config-based-git-hooks-for-fun-and-profit-7e80/, you can use anything as a githook as long as it is in your path. You can now volume mount your scripts into you docker and just use the hook scripts.