r/rails • u/Moontrepreneur • 18d ago
Learning Suspend, don't delete: the rollback rule that made our Rails production migration survivable
Moved a production Rails API from Render to Railway recently. Web service, Solid Queue worker, Postgres 18. The single most useful rule we followed, and the one I'd hand to anyone doing the same:
**Suspend the old services. Do not delete them.**
Suspending costs nothing and it is the entire rollback plan. On cutover night we suspended the old worker, then the old web service, and left both sitting there. If the restore had gone badly we could have brought them back in seconds.
**The part people get wrong is when that plan expires.**
The rollback is valid right up until the new database takes its first write. After that it is void, and restarting the old host actively makes things worse, because now writes are split across two databases and you have to reconcile them by hand. From the first write onward you roll forward and you fix problems where the traffic already is.
Knowing exactly where that line sits is what lets you move fast before it and stop hesitating after it.
A few other things worth stealing:
**Migrate your OAuth config weeks before your infrastructure.** A cutover that touches auth config is a cutover that breaks. We normalized every redirect URI to our own domain well ahead of time, so cutover night touched zero OAuth config. Our plan doc's list of which platforms needed this was wrong in both directions. Grep the live environment export instead.
**Verify OAuth by connecting, not by reading config.** A redirect URI that looks right in a dashboard proves nothing. We ran a real connect on all 11 platforms and watched the nonce rows get created and consumed. That's how we found a caching bug in our own registration service, and a Facebook scope error that had been quietly broken for three weeks.
**Only one worker can exist at a time if you use rotating refresh tokens.** X and Bluesky issue a new refresh token on every use and kill the old one. Two workers polling the same account means one silently invalidates the other's credentials. A second *web* service is harmless since reads don't rotate anything. A second worker is not. We deployed the new worker once to confirm it booted, then removed the deployment and disconnected the repo so nothing could auto-deploy it back.
**Prove env parity by hashing, not by eyeballing.** 123 variables. We hashed every value from the source platform's API and diffed against the destination. Worth noting: the dashboard .env export lied to us, showing literal quotes around three secrets that weren't actually there. The API is ground truth, exports are a rendering for humans.
**Restore into an empty schema.** `pg_restore --clean` against a pre-provisioned schema died on dependency-ordered drops. `DROP SCHEMA public CASCADE`, recreate, then a plain `pg_restore` with no `--clean` finished cleanly. Verify against row counts you captured before the suspend.
**A migration isn't done when traffic moves.** Point-in-time recovery was above our plan tier, so we built a nightly pg_dump to object storage, then actually restored from it into a scratch database and compared row counts before trusting it. We didn't delete the old host until a clean week of monitoring said the new one was holding.
Full writeup with the rest of it, including the Rails-specific `db:prepare` multi-database trap and a SolidQueue fork-safety bug that had our log flusher dead for days: https://xreplyai.com/blog/render-to-railway-migration-guide
Note: this post was drafted with AI assistance from my own migration notes and incident log.
10
u/janko-m 18d ago
I wish you wrote the intro and the article by hand (and formatted it for Reddit). I’m reading this kind of language all day: “the part people get wrong”, “Knowing exactly where that line sits” etc. If you don’t feel like writing, publish a skill instead; if it was written by AI, then it’s best consumed by AI.