Multiple Databases (FREE SELF)

Introduced in GitLab 15.7.

WARNING: This feature is not ready for production use

By default, GitLab uses a single application database, referred to as the main database.

To scale GitLab, you can configure GitLab to use multiple application databases.

Due to known issues, configuring GitLab with multiple databases is an Experiment.

After you have set up multiple databases, GitLab uses a second application database for CI/CD features, referred to as the ci database. We do not exclude hosting both databases on a single PostgreSQL instance.

All tables have exactly the same structure in both the main, and ci databases. Some examples:

  • When multiple databases are configured, the ci_pipelines table exists in both the main and ci databases, but GitLab reads and writes only to the ci_pipelines table in the ci database.
  • Similarly, the projects table exists in both the main and ci databases, but GitLab reads and writes only to the projects table in the main database.
  • For some tables (such as loose_foreign_keys_deleted_records) GitLab reads and writes to both the main and ci databases. See the development documentation

Known issues

  • Once data is migrated to the ci database, you cannot migrate it back.

Migrate existing installations

To migrate existing data from the main database to the ci database, you can copy the database across.

Existing self-compiled installation

  1. Stop GitLab, except for PostgreSQL:

    sudo service gitlab stop
    sudo service postgresql start
  2. Dump the main database:

    sudo -u git pg_dump -f gitlabhq_production.sql gitlabhq_production
  3. Create the ci database, and copy the data from the previous dump:

    sudo -u postgres psql -d template1 -c "CREATE DATABASE gitlabhq_production_ci OWNER git;"
    sudo -u git psql -f gitlabhq_production.sql gitlabhq_production_ci
  4. Configure GitLab to use multiple databases.

Existing Linux package installations

  1. Stop GitLab, except for PostgreSQL:

    sudo gitlab-ctl stop
    sudo gitlab-ctl start postgresql
  2. Dump the main database:

    sudo -u gitlab-psql /opt/gitlab/embedded/bin/pg_dump -h /var/opt/gitlab/postgresql -f gitlabhq_production.sql gitlabhq_production
  3. Create the ci database, and copy the data from the previous dump:

    sudo -u gitlab-psql /opt/gitlab/embedded/bin/psql -h /var/opt/gitlab/postgresql -d template1 -c "CREATE DATABASE gitlabhq_production_ci OWNER gitlab;"
    sudo -u gitlab-psql  /opt/gitlab/embedded/bin/psql -h /var/opt/gitlab/postgresql -f gitlabhq_production.sql gitlabhq_production_ci
  4. Configure GitLab to use multiple databases.

Set up multiple databases

To configure GitLab to use multiple application databases, follow the instructions below for your installation type.

WARNING: You must stop GitLab before setting up multiple databases. This prevents split-brain situations, where main data is written to the ci database, and the other way around.

Self-compiled installations

  1. For existing installations, migrate the data first.

  2. Back up GitLab in case of unforeseen issues.

  3. Stop GitLab:

    sudo service gitlab stop
  4. Open config/database.yml, and add a ci: section under production:. See config/database.yml.decomposed-postgresql for possible values for this new ci: section. Once modified, the config/database.yml should look like:

    production:
      main:
        # ...
      ci:
        adapter: postgresql
        encoding: unicode
        database: gitlabhq_production_ci
        # ...
  5. Save the config/database.yml file.

  6. Update the service files to set the GITLAB_ALLOW_SEPARATE_CI_DATABASE environment variable to true.

  7. For new installations only. Create the gitlabhq_production_ci database:

    sudo -u postgres psql -d template1 -c "CREATE DATABASE gitlabhq_production OWNER git;"
    sudo -u git -H bundle exec rake db:schema:load:ci
  8. Lock writes for ci tables in main database, and the other way around:

    sudo -u git -H bundle exec rake gitlab:db:lock_writes
  9. Restart GitLab:

    sudo service gitlab restart

Linux package installations

  1. For existing installations, migrate the data first.

  2. Back up GitLab in case of unforeseen issues.

  3. Stop GitLab:

    sudo gitlab-ctl stop
  4. Edit /etc/gitlab/gitlab.rb and add the following lines:

    gitlab_rails['env'] = { 'GITLAB_ALLOW_SEPARATE_CI_DATABASE' => 'true' }
    gitlab_rails['databases']['ci']['enable'] = true
    gitlab_rails['databases']['ci']['db_database'] = 'gitlabhq_production_ci'
  5. Save the /etc/gitlab/gitlab.rb file.

  6. Reconfigure GitLab:

    sudo gitlab-ctl reconfigure
  7. Optional, for new installations only. Reconfiguring GitLab should create the gitlabhq_production_ci database if it does not exist. If the database is not created automatically, create it manually:

    sudo gitlab-ctl start postgresql
    sudo -u gitlab-psql /opt/gitlab/embedded/bin/psql -h /var/opt/gitlab/postgresql -d template1 -c "CREATE DATABASE gitlabhq_production_ci OWNER gitlab;"
    sudo gitlab-rake db:schema:load:ci
  8. Lock writes for ci tables in main database, and the other way around:

    sudo gitlab-ctl start postgresql
    sudo gitlab-rake gitlab:db:lock_writes
  9. Restart GitLab:

    sudo gitlab-ctl restart

Further information

For more information on multiple databases, see issue 6168.

For more information on how multiple databases work in GitLab, see the development guide for multiple databases.

Since 2022-07-02, GitLab.com has been running with two separate databases. For more information, see this blog post.