Join Log in
Score:1
avatar Server

GitLab Runner failing to register after migration to new cluster

ec flag
cclloyd

I have GitLab installed in Kubernetes with their Helm chart.

I migrated my old Gitlab deployment from one cluster to another with the following steps:

  • Scale down all pods in old cluster
  • Apply values.yml with helm to new cluster (to create PVCs)
  • Scale down all pods in new cluster
  • Change DNS records, HAProxy, etc
  • Manually rsync data from old PVCs to new PVCs (minio, gitaly, redis, postgres, prometheus)
  • Run helm upgrade to bring deployments back online in new cluster

After all that the deployment for the most part works fine. Able to login and use git.

But the runner is failing to register, so I can't run any CI. Looking at the gitlab-gitlab-runner pod, I see the message below repeated over and over:

Registration attempt 30 of 30
Runtime platform                                    arch=amd64 os=linux pid=691 revision=3b6f852e version=14.0.0
WARNING: Running in user-mode.
WARNING: The user-mode requires you to manually start builds processing:
WARNING: $ gitlab-runner run
WARNING: Use sudo for system-mode:
WARNING: $ sudo gitlab-runner...
 
ERROR: Registering runner... failed                 runner=y6ixJoR1 status=500 Internal Server Error
PANIC: Failed to register the runner. You may be having network problems.

As you can see, it's failing to register the runner. Trying to go to /admin/runners gives me a 500 error.

Where can I see more information as to why I am getting this 500 error?

1003
1 + 0
Score:1
Mikołaj Głodziak avatar Server
id flag
Mikołaj Głodziak

It looks like you have restored data and configuration from the previous cluster, however you are missing the secrets on the new cluster. Here you can find a procedure, what should you do, when the secrets are lost. You can also find there:

In cases like CI/CD variables and runner authentication, you can experience unexpected behaviors, such as:

  • Stuck jobs.
  • 500 errors. #you got this error

See also similar problems:

0 + 1
ec flag
cclloyd
This worked for me. Specifically, resetting the CI vars, runner tokens, project integrations. Though after I still had to manually go to the runners admin page, generate a new registration token, and paste that into the runner secret. After that I deleted the runner pod and it registered just fine. Thanks!
1
Reply
Score:0
Dr. Alexander avatar Server
cn flag
Dr. Alexander

Q: Trying to go to /admin/runners gives me a 500 error.

A: Assure you've restored /etc/gitlab/gitlab-secrets.json file.

Q: Where can I see more information as to why I am getting this 500 error?

A: It's in /var/log/gitlab/gitlab-rails/production.log

0 + 0
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.