Score:2

Server

How to run DNF without going OOM on GCP e2-micro

jrefior

1/16/24, 4:16 PM

I'm trying to run Fedora Cloud Base 36 on an e2-micro in the US central region of Google Cloud Platform (GCP). I haven't upgraded packages in a couple months so I tried to do that today. When I run:

$ sudo dnf upgrade

the OS runs out of memory, kills my command, and terminates my SSH session. From journalctl logs:

sudo[917]:  jrefior : TTY=pts/0 ; PWD=/home/jrefior ; USER=root ; COMMAND=/usr/bin/dnf upgrade
sudo[917]: pam_unix(sudo:session): session opened for user root(uid=0) by jrefior(uid=1000)
systemd-oomd[613]: Killed /user.slice/user-1000.slice/session-3.scope due to memory used (955539456) / total (1007927296) and swap used (945311744) / total (1007>
systemd-oomd[613]: Killed /user.slice/user-1000.slice/session-3.scope due to memory used (956289024) / total (1007927296) and swap used (981708800) / total (1007>
systemd[1]: session-3.scope: systemd-oomd killed 6 process(es) in this unit.
systemd-logind[798]: Session 3 logged out. Waiting for processes to exit.
systemd[1]: session-3.scope: Deactivated successfully.
systemd[1]: session-3.scope: Consumed 10.737s CPU time.
systemd-logind[798]: Removed session 3.

An e2-micro has 1 GB memory, and the instance was created with 1 GB swap as well:

$ free -m
            total       used        free    shared  buff/cache   available
Mem:            961         146     558         0       256         680
Swap:           960         27      933

Can Fedora Cloud Base be managed on 2 GB of memory (including swap)? GCP e2-micro specs are here. Is there a way to upgrade kernel and packages without using so much memory?

Would I run into a problem upgrading Fedora Cloud Base as well when the time comes?

493

2 + 4

fedora

oom

google-cloud-platform

dnf

Jeffrey D.

1/16/24, 10:18 PM

To decrease memory usage, I preferred to [manually upgrade](https://docs.fedoraproject.org/en-US/fedora/latest/system-administrators-guide/kernel-module-driver-configuration/Manually_Upgrading_the_Kernel/) the kernel. If this works, try this.

jrefior

1/17/24, 3:11 AM

Thanks @JeffreyD., I hadn't seen that doc before

Jeffrey D.

1/17/24, 5:11 PM

Can you try upgrading your VM instance's specifications first, and if that succeeds, the kernel and packages? Try to restore your VM's original specifications. Make a security snapshot first before executing that.

jrefior

1/18/24, 10:47 PM

Thanks @JeffreyD., that's a good idea. After reading John Mahowald's answer and other information, I'm considering switching to Arch for my e2-micro. Seems like a better fit to the resources available.

Score:2

Server

John Mahowald

1/18/24, 8:37 AM

No, 1 GB of RAM is not enough for package management of a Fedora based system. Which in my opinion includes RHEL and rebuilds, but the extent of the problem on the size of the repos involved. Yes, a distro major version upgrade transaction is among the biggest, and could be problematic as well.

Paging out during a package transaction is a miserable experience. Slow. And even with the swap possibly oom killed in a dangerous place. I am being unfair to swap space by only calling this 1 GB, but the memory system behaves differently than if you had 2 GB of DRAM.

For recent reports of this happening, see mailing list thread dnf makecache memory usage increase, and Bugzilla "dnf update" runs out of memory on swapless machines with 1G or less of RAM. Note that the entire dnf metadata is being converted into some libsolv database in memory, on top of what it does to parse those dependencies.

If you moved your instances to image based updates, that could be done once on a bigger box. Upgrade a primary copy, make a disk image out of that, and boot new instances from the updated one. Likely a significant change of how you do updates. See also: rpm-ostree.

Commenters on the bug have noted file lists blowing up the size of metadata. Especially with the number of packages in Fedora, how often they update, and how many have still have legacy file dependencies. Consider maintaining your own custom repos of just a subset of the packages you use.

microdnf is the package manger without Python. While there might be some memory savings to switching, its authors have not laid out in detail its limitations. Possibly wait on this one, Fedora is inevitably going to change the package manager again anyway.

+ 1

jrefior

1/18/24, 10:44 PM

Thanks. I appreciate the inclusion of context I wasn't aware of.

Score:0

Server

Jamie Fargen

4/21/24, 12:35 PM

You can create a swapfile and utilize it while running dnf updates.

$ sudo dd if=/dev/zero of=/swapfile count=1024 bs=1MiB
$ sudo chmod 600 /swapfile
$ sudo mkswap /swapfile
$ sudo swapon /swapfile
$ sudo dnf update

Once the updates are finished you can turn swap off by running:

$ sudo swapoff

When using a swapfile over the long-term there were spurious instance stability issues, but these were mainly over come by using zram. Unfortunately even with 384MB of zram enabled updates would be killed by oom, otherwise the system was stable and has run a LAMP stack for over a year.

+ 0

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: How to run DNF without going OOM on GCP e2-micro

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.