Bring a demoted primary site back online (PREMIUM SELF)
After a failover, it is possible to fail back to the demoted primary site to restore your original configuration. This process consists of two steps:
- Making the old primary site a secondary site.
- Promoting a secondary site to a primary site.
WARNING: If you have any doubts about the consistency of the data on this site, we recommend setting it up from scratch.
Configure the former primary site to be a secondary site
Since the former primary site is out of sync with the current primary site, the first step is to bring the former primary site up to date. Note, deletion of data stored on disk like repositories and uploads is not replayed when bringing the former primary site back into sync, which may result in increased disk usage. Alternatively, you can set up a new secondary GitLab instance to avoid this.
To bring the former primary site up to date:
SSH into the former primary site that has fallen behind.
/etc/gitlab/gitlab-cluster.jsonif it exists.
If the site to be re-added as a secondary site was promoted with the
gitlab-ctl geo promotecommand, then it may contain a
/etc/gitlab/gitlab-cluster.jsonfile. For example during
gitlab-ctl reconfigure, you may notice output like:
The 'geo_primary_role' is defined in /etc/gitlab/gitlab-cluster.json as 'true' and overrides the setting in the /etc/gitlab/gitlab.rb
If so, then
/etc/gitlab/gitlab-cluster.jsonmust be deleted from every Sidekiq, PostgreSQL, Gitaly, and Rails node in the site, to make
/etc/gitlab/gitlab.rbthe single source of truth again.
Make sure all the services are up:
sudo gitlab-ctl start
NOTE: If you disabled the primary site permanently, you need to undo those steps now. For distributions with systemd, such as Debian/Ubuntu/CentOS7+, you must run
sudo systemctl enable gitlab-runsvdir. For distributions without systemd, such as CentOS 6, you need to install the GitLab instance from scratch and set it up as a secondary site by following Setup instructions. In this case, you don't need to follow the next step.
NOTE: If you changed the DNS records for this site during disaster recovery procedure you may need to block all the writes to this site during this procedure.
Set up database replication. In this case, the secondary site refers to the former primary site.
- If PgBouncer was enabled on the current secondary site
(when it was a primary site) disable it by editing
sudo gitlab-ctl reconfigure.
- You can then set up database replication on the secondary site.
- If PgBouncer was enabled on the current secondary site (when it was a primary site) disable it by editing
If you have lost your original primary site, follow the setup instructions to set up a new secondary site.
Promote the secondary site to primary site
When the initial replication is complete and the primary site and secondary site are closely in sync, you can do a planned failover.
Restore the secondary site
If your objective is to have two sites again, you need to bring your secondary site back online as well by repeating the first step (configure the former primary site to be a secondary site) for the secondary site.