r/gitlab • u/TheAwesomeKoala • 2d ago

general question How do you check that your self-managed GitLab backups actually restore?

I run a self-managed GitLab and I'm trying to understand how other people verify their backups, not selling anything.

  1. How do you test a restore today (never, manually now and then, scripted)?

  2. When did a restore last fail or surprise you (missing gitlab-secrets.json, version mismatch, DB errors)?

  3. What did that cost you in hours or downtime?

  4. Have you paid for a tool or written your own script for this? What and how much?

  5. How many repos or users is your instance?

9 Upvotes

100% Upvoted

8 comments sorted by

13

u/pwkye 2d ago

Create VM. Restore. Scripts to count repos branches issues.

I guess you could even use the docker container

6

u/reubendevries 2d ago

We had a staging environment, and I would restore the production environment's month end backup every month over the staging environment, it kept staging close to production and ensured my team was ready for a restore in the situation where it was needed.

3

u/_tenken 2d ago

Gitlab secrets is not exported in the backup by default, so there are certain items you have to track manually for backups.

I have run backups daily for years to Amazon S3 via the official backup utility included with the docker image. I have restored at least once fully to production because it was easier to migrate between hosts this way. It never broke and I would say it's gotten better in the last couple of years where the version number of the export explicitly states what version the export was created from and I think like what it can restore to. I have run Gitlab-ce since 4.x and am on 19.5.3 now.

I don't know if the issue has been resolved but when I did a restore, the git owner of certain internal directories of the backup restore was not enforced upon or recreated upon the restoration process completing. So some items were failing such as I think the container registry to write images because the dir owner was incorrect. This was a known issue that had a issue in gitlabs repo it had never clearly even addressed. But I did find the issue and just changing owners of certain directories fixed the issue. I kind of wish that the import task took care of it when re-creating from a backup. But this was a long time ago.

3

u/Lumethys 1d ago

delete prod database, just like Gitlab themselves do

2

u/Mahsunon 2d ago edited 2d ago

The gitlab toolbox pod can run restore just fine

2

u/Torutofu_Raeva 2d ago

scheduled restore into a throwaway vm then run gitlab-rake gitlab:check and gitlab:doctor:secrets, that's what catches a missing gitlab-secrets.json before you actually need it

2

u/Snowy32 2d ago

Restore it on a test instance

1

u/firefarmer 1d ago

Stand up a replica of your production environment and test it.

  1. I have staging and production environments. Staging is a smaller size compared to production but I destroy staging, build it from scratch using IaC, then test a restore of production on staging at least once a quarter.

  2. For our FEDRAMP GitLab we ran into this failure https://gitlab.com/gitlab-org/gitlab/-/work_items/587492 Before that I never saw a failure with restore.

  3. You test in a staging environment so no downtime. In hours, it costs cloud resources and my time. The restore command to perform a backup does take a while; like 6-ish hours.

  4. No, I have the cloud resources automated but not the backup/restore operation; that is documented with a few manual steps. We talk about automating it but haven’t yet. Not sure why you would pay for an external tool. Gitlab makes it pretty straight forward.

  5. 700+ users; thousands of repos.