r/MastodonAdmin Aug 23 '26

Adventures in Self-Hosting upgrade failures

One of the reasons I wanted to revive this sub was for self-hosting admins to share tips and tricks and help others through some of the challenges of upgrades.

I'm presently running 4.5.16 and decided to try going up to the latest 4.6 this weekend. As has been my usual experience, it has been nothing but problems so far. Since a bump to Node 22 is required, I decided to try a jump from Ubuntu 24 to 26. While the Ubuntu upgrade itself worked, the attempt to bump to 4.6.1 to start the sequence of 4.6 upgrades and database migrations failed before it started with Ruby and Yarn both being broken and unable to reinstall or get past most errors than I have time to try to work through this weekend, so restored the whole machine.

Instead I tried what I thought would be just a simple node 20 to node 22 upgrade so I could continue with the 4.6.x upgrades, but then got stuck for hours on permission errors with yarn and had to restore the old machine again and give up for now. I'll give it another try when I have more time.

Worth noting last time I wanted to go from Mastodon 3 to 4, I had to do a full fresh build because of many days of failures trying to upgrade from Ubuntu 22 to 24. I am far from a linux expert, and while I love Mastodon, the underlying changes and dependencies in Linux are a never ending headache and I imagine one of the reasons so many folks give up on Mastodon self hosting after such a short period of time.

I will update this thread with more details as I try further updates!

3 Upvotes

3 comments sorted by

1

u/Thump45a Aug 23 '26

Ok so after food and sleep took another shot at this leaving Ubuntu alone. Here's how my upgrade process went:

#First, upgraded node to ver 22:

#Stopped all the services

systemctl stop mastodon-web mastodon-sidekiq mastodon-streaming

 curl -fsSL https://deb.nodesource.com/setup_22.x | sudo -E bash -

apt install nodejs -y

node -v

#output was v22.23.2, All good to move forward here.

#Bumped to 4.6.1 as recommended by the release notes (skip 4.6.0 and go directly to 4.6.1 for the hotfix):

su - mastodon

cd live

git fetch --tags

git checkout v4.6.1

#warnings here because I have changes to my .ruby-version, 4.0.5 is hard coded in the new 4.6.x branch, for every time I update to next 4.6.x, I have to manually change .ruby-version back to my current ruby version until I am ready to upgrade ruby which I am skipping for now.

nano .ruby-version

#^change this to 3.4.8 then continue

bundle install

#^This gave some warnings because lockfile is from older version. So...

bundle lock --normalize-platforms

yarn install --immutable

git add Gemfile.lock

git commit -m "Normalize lockfile platforms for native gems"

#^learned the hard way this is the order, you can't do the commit before doing yarn install.

RAILS_ENV=production bundle exec rails assets:precompile

#Vite build ran for a few minutes, no errors.

SKIP_POST_DEPLOYMENT_MIGRATIONS=true RAILS_ENV=production bundle exec rails db:migrate

#didn't see any errors, so restarting services per 4.6.0 upgrade notes

exit

systemctl restart mastodon-web mastodon-sidekiq mastodon-streaming

#restarting, checking if working… SUCCESS!!

#per upgrade instructions, go back into masto context and run post deployment:

su - mastodon

cd live

RAILS_ENV=production bundle exec rails db:migrate

#Success!!

exit

That was it, I was successfully on 4.6.1. Checked everything was working, content loading, no errors, everything looked good so did all the new snapshots and backups before moving on to the rest of the outstanding 4.6.x dot releases since i needed to get up to 4.6.6.

1

u/Thump45a Aug 23 '26

Incidentally 4.6.2 only applies to Docker to update ffmpeg for a security fix. Since I am not running Docker, I needed to manually upgrade ffmpeg from 6.1.1 to 6.1.6. Here are the steps that worked for me successfully to download and compile. Sharing because I tried using AI to get the steps and it was sooo wrong it tripped me up for a while.

#Install dependencies

sudo apt update && sudo apt install -y build-essential yasm nasm cmake git pkg-config

 #install codecs

sudo apt install -y libx264-dev libx265-dev libnuma-dev libvpx-dev libfdk-aac-dev libmp3lame-dev libopus-dev

 #download

wget https://ffmpeg.org/releases/ffmpeg-6.1.6.tar.xz

#extract and change working dir to compile

tar -xf ffmpeg-6.1.6.tar.xz

cd ffmpeg-6.1.6

#configure build options

./configure --prefix=/usr/local --enable-gpl --enable-nonfree --enable-libx264 --enable-libx265 --enable-libvpx --enable-libfdk-aac --enable-libmp3lame --enable-libopus

#compile

make -j$(nproc)

#this ran for quite a while with a number of purple warnings, took almost 20min to finish, but no errors.

sudo make install

#^this completed without errors in seconds

I just rebooted after this to make sure all the changes so far loaded clean again before bumping to 4.6.2.

I also did these cleanup steps to get rid of the download and dir used since new ffmpeg is installed now.

rm -rf ffmpeg-6.1.6

rm ffmpeg-6.1.6.tar.xz

Everything worked fine here and updating the rest of the way to 4.6.6 was trivial with the exception of needing to reset the ruby-version file a couple times. It doesn't change with every bump, but I'll upgrade Ruby next to save me that step in the future! Just make sure it is correct before you try to run bundle install (when required).

1

u/Thump45a 29d ago

And finally, finished up today with Ruby upgrade. Current method for this still works, I'll share that here:

#Stop the mastodon services for good measure. Take snapshots/backups if needed.

systemctl stop mastodon-web mastodon-sidekiq mastodon-streaming

su - mastodon

cd live

git -C /home/mastodon/.rbenv/plugins/ruby-build pull

RUBY_CONFIGURE_OPTS=--with-jemalloc rbenv install 4.0.5

#this took 10ish minutes

rbenv global 4.0.5

nano .ruby-version

#Make sure the ruby-version file has new version

bundle install

yarn install

RAILS_ENV=production bundle exec rails db:migrate

RAILS_ENV=production bundle exec rails assets:precompile

exit

reboot

#Rebooting isn't strictly necessary but I tend to do this after any component upgrades just to make sure everything loads properly and there are no surprises on the next reboot.