Join Log in
Score:0
avatar Server

Sidekiq crash loop, no jobs run

ec flag
cclloyd

After upgrading GitLab to 14.7, sidekiq fails to stay running, and no jobs are running.

All CI jobs are stuck saying This job depends on upstream jobs that need to succeed in order for this job to be triggered. I also cannot push in this state.

Kubernetes version: 1.21.6 Helm chart version: 5.7.0 (gitlab 14.7.0)

Output of kubectl get pods

NAME                                         READY   STATUS      RESTARTS   AGE
db-postgresql-0                              1/1     Running     0          19h
gitlab-gitaly-0                              1/1     Running     0          6h8m
gitlab-gitlab-exporter-74669c96b7-d7wpz      1/1     Running     0          6h9m
gitlab-gitlab-pages-657d7f6ffc-d9vlt         1/1     Running     0          6h9m
gitlab-gitlab-runner-75b77988c8-g2q88        1/1     Running     5          6h9m
gitlab-gitlab-shell-5d64558c69-gzjsz         1/1     Running     0          6h9m
gitlab-gitlab-shell-5d64558c69-j45qg         1/1     Running     0          6h3m
gitlab-gitlab-upgrade-check-gfjql            0/1     Completed   0          3h53m
gitlab-migrations-14-x8g8m                   0/1     Completed   4          4h15m
gitlab-minio-864888b9fb-2rmmn                1/1     Running     7          6h9m
gitlab-minio-create-buckets-14-fqkbc         0/1     Completed   0          4h15m
gitlab-prometheus-server-5b47c879b4-p7k4h    2/2     Running     0          19h
gitlab-registry-b6f64686b-jrn8v              1/1     Running     6          6h9m
gitlab-shared-secrets-13-jl6-l8cbj           0/1     Completed   0          5h7m
gitlab-shared-secrets-15-awv-mtqzf           0/1     Completed   0          3h53m
gitlab-sidekiq-all-in-1-v2-ff654dcd9-n88fh   0/1     Running     12         55m
gitlab-toolbox-7bd697cb59-nrpcv              1/1     Running     0          6h9m
gitlab-webservice-default-6f67f58788-dmvnw   2/2     Running     8          6h9m
gitlab-webservice-default-6f67f58788-tfwh8   2/2     Running     16         5h42m
redis-master-0                               1/1     Running     1          19h

Last Bunch of lines from the sidekiq container

Creating scope :order_name_asc. Overwriting existing method Label.order_name_asc.
Creating scope :order_name_desc. Overwriting existing method Label.order_name_desc.
2022-01-28T21:35:42.947Z pid=22 tid=9k6 WARN: Creating scope :open. Overwriting existing method ExternalPullRequest.open.
Creating scope :open. Overwriting existing method ExternalPullRequest.open.
2022-01-28T21:35:43.563Z pid=22 tid=9k6 WARN: Creating scope :order_name_asc. Overwriting existing method Member.order_name_asc.
2022-01-28T21:35:43.564Z pid=22 tid=9k6 WARN: Creating scope :order_name_desc. Overwriting existing method Member.order_name_desc.
2022-01-28T21:35:43.914Z pid=22 tid=9k6 WARN: Creating scope :with_logging_info. Overwriting existing method PagesDomain.with_logging_info.
Creating scope :order_name_asc. Overwriting existing method Member.order_name_asc.
Creating scope :order_name_desc. Overwriting existing method Member.order_name_desc.
Creating scope :with_logging_info. Overwriting existing method PagesDomain.with_logging_info.
2022-01-28T21:35:44.564Z pid=22 tid=9k6 WARN: Creating scope :system. Overwriting existing method PrometheusMetric.system.
Creating scope :system. Overwriting existing method PrometheusMetric.system.
2022-01-28T21:35:45.568Z pid=22 tid=9k6 WARN: Creating scope :with_user. Overwriting existing method GroupMember.with_user.
2022-01-28T21:35:45.579Z pid=22 tid=9k6 WARN: Creating scope :reporters. Overwriting existing method GroupMember.reporters.
2022-01-28T21:35:45.580Z pid=22 tid=9k6 WARN: Creating scope :guests. Overwriting existing method GroupMember.guests.
Creating scope :with_user. Overwriting existing method GroupMember.with_user.
Creating scope :reporters. Overwriting existing method GroupMember.reporters.
Creating scope :guests. Overwriting existing method GroupMember.guests.
2022-01-28T21:35:47.612Z pid=22 tid=9k6 WARN: Creating scope :extended. Overwriting existing method GitlabSubscription.extended.
Creating scope :extended. Overwriting existing method GitlabSubscription.extended.
2022-01-28T21:35:51.276Z pid=22 tid=9k6 WARN: Creating scope :without_statuses. Overwriting existing method Ci::Stage.without_statuses.
2022-01-28T21:35:51.279Z pid=22 tid=9k6 WARN: Creating scope :created. Overwriting existing method Ci::Stage.created.
2022-01-28T21:35:51.280Z pid=22 tid=9k6 WARN: Creating scope :pending. Overwriting existing method Ci::Stage.pending.
2022-01-28T21:35:51.281Z pid=22 tid=9k6 WARN: Creating scope :running. Overwriting existing method Ci::Stage.running.
2022-01-28T21:35:51.282Z pid=22 tid=9k6 WARN: Creating scope :success. Overwriting existing method Ci::Stage.success.
2022-01-28T21:35:51.283Z pid=22 tid=9k6 WARN: Creating scope :failed. Overwriting existing method Ci::Stage.failed.
2022-01-28T21:35:51.284Z pid=22 tid=9k6 WARN: Creating scope :canceled. Overwriting existing method Ci::Stage.canceled.
2022-01-28T21:35:51.284Z pid=22 tid=9k6 WARN: Creating scope :skipped. Overwriting existing method Ci::Stage.skipped.
2022-01-28T21:35:51.285Z pid=22 tid=9k6 WARN: Creating scope :manual. Overwriting existing method Ci::Stage.manual.
2022-01-28T21:35:51.293Z pid=22 tid=9k6 WARN: Creating scope :scheduled. Overwriting existing method Ci::Stage.scheduled.
2022-01-28T21:35:51.294Z pid=22 tid=9k6 WARN: Creating scope :preparing. Overwriting existing method Ci::Stage.preparing.
2022-01-28T21:35:51.296Z pid=22 tid=9k6 WARN: Creating scope :waiting_for_resource. Overwriting existing method Ci::Stage.waiting_for_resource.
2022-01-28T21:35:51.680Z pid=22 tid=9k6 WARN: Creating scope :database. Overwriting existing method Ci::BuildTraceChunk.database.
Creating scope :without_statuses. Overwriting existing method Ci::Stage.without_statuses.
Creating scope :created. Overwriting existing method Ci::Stage.created.
Creating scope :pending. Overwriting existing method Ci::Stage.pending.
Creating scope :running. Overwriting existing method Ci::Stage.running.
Creating scope :success. Overwriting existing method Ci::Stage.success.
Creating scope :failed. Overwriting existing method Ci::Stage.failed.
Creating scope :canceled. Overwriting existing method Ci::Stage.canceled.
Creating scope :skipped. Overwriting existing method Ci::Stage.skipped.
Creating scope :manual. Overwriting existing method Ci::Stage.manual.
Creating scope :scheduled. Overwriting existing method Ci::Stage.scheduled.
Creating scope :preparing. Overwriting existing method Ci::Stage.preparing.
Creating scope :waiting_for_resource. Overwriting existing method Ci::Stage.waiting_for_resource.
Creating scope :database. Overwriting existing method Ci::BuildTraceChunk.database.
2022-01-28T21:35:59.333Z pid=22 tid=9k6 WARN: Creating scope :approved_by_committer. Overwriting existing method MergeRequests::ComplianceViolation.approved_by_committer.
2022-01-28T21:35:59.508Z pid=22 tid=9k6 WARN: Creating scope :runnable_schedules. Overwriting existing method Security::OrchestrationPolicyRuleSchedule.runnable_schedules.
Creating scope :approved_by_committer. Overwriting existing method MergeRequests::ComplianceViolation.approved_by_committer.
Creating scope :runnable_schedules. Overwriting existing method Security::OrchestrationPolicyRuleSchedule.runnable_schedules.
2022-01-28T21:36:06.992Z pid=22 tid=9k6 WARN: Creating scope :regular. Overwriting existing method EE::Gitlab::BackgroundMigration::MigrateApproverToApprovalRules::ApprovalMergeRequestRule.regular.
2022-01-28T21:36:06.994Z pid=22 tid=9k6 WARN: Creating scope :code_owner. Overwriting existing method EE::Gitlab::BackgroundMigration::MigrateApproverToApprovalRules::ApprovalMergeRequestRule.code_owner.
Creating scope :regular. Overwriting existing method EE::Gitlab::BackgroundMigration::MigrateApproverToApprovalRules::ApprovalMergeRequestRule.regular.
Creating scope :code_owner. Overwriting existing method EE::Gitlab::BackgroundMigration::MigrateApproverToApprovalRules::ApprovalMergeRequestRule.code_owner.
2022-01-28T21:36:21.933Z pid=22 tid=9k6 WARN: Creating scope :system. Overwriting existing method Gitlab::DatabaseImporters::CommonMetrics::PrometheusMetric.system.
Creating scope :system. Overwriting existing method Gitlab::DatabaseImporters::CommonMetrics::PrometheusMetric.system.
{"severity":"INFO","time":"2022-01-28T21:36:58.434Z","message":"A worker terminated, shutting down the cluster"}
126
0 + 0
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.