Score:0

GitLab with local IP install hangs at logrotate

in flag

I am trying and failing to install GitLab (EE) onto an on-premises bare-metal server PC running Ubuntu Server 20.04. The server is intended to later have internet-facing access, but for now it needs to be internal LAN only, accessed by IP or hostname if possible. The PC is currently set up with a static internal IP via netplan. We do not have a domain name reserved for this server or an SSL certificate at the moment.

I have a working Git server set up on the machine with successful access via SSH and via HTTP (Apache2, then switched to Nginx). My goal is to set up GitLab on the machine running in parallel with the existing Git server.

After getting Nginx running correctly yesterday, I attempted to install GitLab through Omnibus, using the procedure described at the official GitLab install page for Ubuntu, first the curl adding the repository, then running EXTERNAL_URL="http://10.7.2.xxx" apt install gitlab-ee (internal IP, but censored just in case). The process finished without errors, but when attempting to connect to the server with that URL, I only got a 403 from Nginx. I suspect this was due to Nginx already running previously for plain Git on /srv/www/git. I attempted to remove it with apt remove gitlab-ee, followed by deleting the the remaining directories at /etc/gitlab, /var/opt/gitlab, /opt/gitlab, and /var/log/gitlab. At the time I did not know that was the wrong way to uninstall GitLab.

This morning I attempted a reinstall. First I stopped and disabling Nginx with systemctl, backed up my config by moving it out of /etc/nginx/enabled-sites into my home directory, then uninstalling it with apt remove nginx. After a reboot and confirming neither it nor Apache2 were running, I attempted to reinstall GitLab with apt install gitlab-ee again. It finished unexpectedly quickly, as it appears its resources were still present on the system. Connecting from a browser failed completely, no connection at all.

I attempted a reinstall again, this time uninstalling GitLab fully with the following commands:

# gitlab-ctl uninstall
# gitlab-ctl cleanse
# gitlab-ctl remove-accounts
# dpkg -P gitlab-ee

Removing accounts hung for 10min with no output so I had to ^C it. After removing it with dpkg I then removed the same four directories as before, though only /opt/gitlab and /var/log/gitlab remained.

After a supposedly clean uninstall, I rebooted and attempted what I thought to be a clean reinstall. It hung on * ruby_block[wait for logrotate service socket] action run endlessly, for over an hour. Research pointed to opening up another terminal and running the following commands while omnibus was hung:

# nohup /opt/gitlab/embedded/bin/runsvdir-start &
# gitlab-ctl reconfigure

This didn't work, gitlab-ctl reconfigure hung for 10 mins before I gave up, and Omnibus on the other terminal didn't continue. It wouldn't even respond to ^C. I had to force a reboot from another session, and attempted a "clean" uninstall, reboot, and reinstall again.

Uninstall this time did not have any problems with remove-accounts (Pastebin log, username, hostname, IP censored). However, once again it hung at logrotate, for over an hour again, and again not responding to ^C. Pastebin output, Pastebin log from /var/log/gitlab/reconfigure/1636492507.log.

What has not solved my problem:

  • systemctl start logrotate.service (immediately stops/finishes, awaiting the timer), logrotate.timer (was already running). No response from Omnibus when starting either.
  • Running runsvdir-start and gitlab-ctl reconfigure (see above)
  • Running gitlab-ctl tail (throws tail: cannot follow '-' by name)
  • multi-user.target is loaded and active, according to systemctl -t target.

All the other results I can find are related to specific aspects of GitLab failing, rather than the initial install. I'm stumped and I can't find anything else relevant. What do I do?

Score:0
in flag

Last night I forced another sudo reboot, and this morning upon running an apt upgrade I was prompted by dpkg to run dpkg --configure -a, which resumed the installation and finished it, successfully. I am even able to connect to the IP through a browser and get a GitLab login page. I don't know why this worked now.

While this did solve my particular instance of this problem, for posterity I'd like to keep this question open for now to try to understand why this problem occurred (hanging at logrotate twice), and why this solved it. If anyone else comes across this problem, knowing why it happened would be much more helpful than a magic command that might or might not work and doesn't explain why. If anyone could help explain why this problem happened or why this command worked for me, I'd appreciate it.

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.