gitlab 502 错误排查

动态查看日志

tail -f  /var/log/gitlab/gitlab-rails/production.log

迁移或恢复备份后部分操作服务端响应500

进入容器, 运行

gitlab-rails dbconsole

依次清除 token, 注意! 这样会清除全部的 token, 不过解决问题简单粗暴

-- clear project tokens
UPDATE projects SET runners_token = null, runners_token_encrypted = null;
-- clear group tokens
UPDATE namespaces SET runners_token = null, runners_token_encrypted = null;
-- clear instance tokens
UPDATE application_settings SET runners_registration_token_encrypted = null;

造成这个问题的原因是因为没有备份这个文件

gitlab-secrets.json

以后操作的时候还是要多看看官方文档, 文档说的很清楚…

https://docs.gitlab.com/ee/raketasks/backup_restore.html

When the secrets file is lost
If you didn’t back up the secrets file, you must complete several steps to get GitLab working properly again.

The secrets file is responsible for storing the encryption key for the columns that contain required, sensitive information. If the key is lost, GitLab can’t decrypt those columns, preventing access to the following items:

CI/CD variables
Kubernetes / GCP integration
Custom Pages domains
Project error tracking
Runner authentication
Project mirroring
Web hooks
In cases like CI/CD variables and runner authentication, you can experience unexpected behaviors, such as:

Stuck jobs.
500 errors.

Whoops, GitLab is currently unavailable.

可以正常查看项目列表, 但是无法进入项目, 错误提示

Whoops, GitLab is currently unavailable.

项目基于docker容器部署, 首先查看日志, 发现一条错误, 权限拒绝.

{"error":"open /var/opt/gitlab/gitaly/gitaly.pid: permission denied","level":"fatal","msg":"find gitaly","time":"2021-05-05T18:05:26Z","wrapper":3514}

google 发现, 这个文件在 docker 容器里的权限应该是 git:git

但是我进入容器后发现该文件的权限为 root:root

-rw-r--r-- 1 root root   63 May  5 16:51 RUBY_VERSION
-rw-r--r-- 1 root root   23 May  5 17:56 VERSION
-rw-r----- 1 root git   819 May  5 17:56 config.toml
-rw------- 1 root root    3 Mar  8 18:57 gitaly.pid
srwxr-xr-x 1 root root    0 Mar  8 18:57 gitaly.socket
drwx------ 2 git  root 4096 Mar  8 18:57 internal_sockets

修改权限

chown git:git gitaly.pid

Errno::EEXIST: File exists @ dir_s_mkdir - /var/opt/gitlab/backups/repositories

备份的时候出现这个问题, 十有八九是权限问题.

直接

chmod -R 777 bakcups

参考连接

https://forum.gitlab.com/t/local-gitlab-is-not-longer-working/29319/19
https://docs.gitlab.com/ee/raketasks/backup_restore.html