Repository checks

DETAILS: Tier: Free, Premium, Ultimate Offering: Self-managed

You can use git fsck to verify the integrity of all data committed to a repository. GitLab administrators can:

Manually trigger this check for a project.
Schedule this check to run automatically for all projects.
Run this check from the command line.
Run a Rake task for checking Git repositories, which can be used to run git fsck against all repositories and generate repository checksums, as a way to compare repositories on different servers.

Checks that aren't manually run on the command line are executed through a Gitaly node. For information on Gitaly repository consistency checks, some disabled checks, and how to configure consistency checks, see Repository consistency checks.

Check a project's repository using GitLab UI

To check a project's repository using GitLab UI:

On the left sidebar, at the bottom, select Admin area.
Select Overview > Projects.
Select the project to check.
In the Repository check section, select Trigger repository check.

The checks run asynchronously so it may take a few minutes before the check result is visible on the project page in the Admin area. If the checks fail, see what to do.

Enable repository checks for all projects

Instead of checking repositories manually, GitLab can be configured to run the checks periodically:

On the left sidebar, at the bottom, select Admin area.
Select Settings > Repository.
Expand Repository maintenance.
Enable Enable repository checks.

When enabled, GitLab periodically runs a repository check on all project repositories and wiki repositories to detect possible data corruption. A project is checked no more than once per month. Administrators can configure the frequency of repository checks. To edit the frequency:

For Linux package installations, edit gitlab_rails['repository_check_worker_cron'] in /etc/gitlab/gitlab.rb.
For source-based installations, edit [gitlab.cron_jobs.repository_check_worker] in /home/git/gitlab/config/gitlab.yml.

If any projects fail their repository checks, all GitLab administrators receive an email notification of the situation. By default, this notification is sent out once a week at midnight at the start of Sunday.

Repositories with known check failures can be found at /admin/projects?last_repository_check_failed=1.

Run a check using the command line

You can run git fsck using the command line on repositories on Gitaly servers. To locate the repositories:

Go to the storage location for repositories:
- For Linux package installations, repositories are stored in the /var/opt/gitlab/git-data/repositories directory by default.
- For GitLab Helm chart installations, repositories are stored in the /home/git/repositories directory inside the Gitaly pod by default.
Identify the subdirectory that contains the repository that you need to check.
Run the check. For example:
```
sudo -u git /opt/gitlab/embedded/bin/git \
   -C /var/opt/gitlab/git-data/repositories/@hashed/0b/91/0b91...f9.git fsck --no-dangling
```
The error fatal: detected dubious ownership in repository means you're running the command using the wrong account. For example, root.

What to do if a check failed

If a repository check fails, locate the error in the repocheck.log file on disk at:

/var/log/gitlab/gitlab-rails for Linux package installations.
/home/git/gitlab/log for self-compiled installations.
/var/log/gitlab in the Sidekiq pod for GitLab Helm chart installations.

If periodic repository checks cause false alarms, you can clear all repository check states:

On the left sidebar, at the bottom, select Admin area.
Select Settings > Repository.
Expand Repository maintenance.
Select Clear all repository checks.

Troubleshooting

When working with repository checks, you might encounter the following issues.

Error: `failed to parse commit <commit SHA> from object database for commit-graph`

You can see a failed to parse commit <commit SHA> from object database for commit-graph error in repository check logs. This error occurs if your commit-graph cache is out of date. The commit-graph cache is an auxiliary cache and is not required for regular Git operations.

While the message can be safely ignored, see the issue error: Could not read from object database for commit-graph for more details.

Dangling commit, tag, or blob messages

Repository check output often includes tags, blobs, and commits that must be pruned:

dangling tag 5c6886c774b713a43158aae35c4effdb03a3ceca
dangling blob 3e268c23fcd736db92e89b31d9f267dd4a50ac4b
dangling commit 919ff61d8d78c2e3ea9a32701dff70ecbefdd1d7

This is common in Git repositories. They're generated by operations like force pushing to branches, because this generates a commit in the repository that is not longer referred to by a ref or by another commit.

If a repository check fails, the output is likely to include these warnings.

Ignore these messages, and identify the root cause of the repository check failure from the other output.

GitLab 15.8 and later no longer includes these messages in the check output. Use the --no-dangling option to suppress then when run from the command line.