Ubuntu/Apache: /tmp files unexpectedly removed

EML

6/1/24, 12:39 PM

I'm losing files in /tmp on current Ubuntu (22.04.2), possibly related to virtualisation. Any insights appreciated. This is not a normal cleanup; the file is lost immediately.

I have two systems: a remote VPS, and a local staging system, both running the same software. On both systems, /tmp is not a tmpfs; it's part of the LVM volume group mounted at /.

Apache runs program1 and program2 immediately after each other:

program1 creates a new subdirectory under /tmp, with 2 files: file1 (a Python source file) and file2 (the output of file1). If debug is enabled, file1 is retained; otherwise, it it deleted after creating file2 (with a C library remove)
program2 returns file2 to the client

This all works as expected on the staging system. If debug is enabled, the new /tmp/xxx directory is still on the filesystem, and contains both file1 and file2. If debug isn't enabled, it contains only file2.

On the VPS:

if debug is not enabled, everything works, but there's nothing left under /tmp
if debug is enabled, program2 fails, reporting that file2 doesn't exist

I have 'fixed' the code by removing the C lib remove, so that file1 is always retained, but I don't understand what's going on - it appears that the file remove has somewhere triggered a remove of the entire directory.

0 + 6

linux

filesystems

DubStep

6/1/24, 1:30 PM

Do you have functionality in the application that removes these files after certain times? I'd suspect permissions and on one filesystem you application doesn't have delete privileges and on the other filesystem it does. If you are auditing the filesystem, OS logs should show what process is deleting the file and when.

EML

6/1/24, 1:37 PM

@DubStep - no, nothing in the app deletes the files, apart from this one remove op. The app just relies on the OS to eventually clean up everything else. The app runs as `www-data` on both systems, and both `/tmp` are `drwxrwxrwt`, as expected. No auditing enabled.

HBruijn

6/1/24, 1:38 PM

`/tmp` traditionally has been a shared space for all local services and users and also been a big source of security problems for all kinds of services. Symlink attacks and DoS vulnerabilities due to guessable /tmp temporary files are common. That's why systemd introduced the [`PrivateTmp=yes`](https://www.freedesktop.org/software/systemd/man/systemd.exec.html#PrivateTmp=) setting. If `true` that sets up a new file system namespace for the executed processes and mounts private `/tmp/` and `/var/tmp/` directories inside it that are not shared by processes outside of the namespace.

HBruijn

6/1/24, 1:38 PM

Afaik `PrivateTmp=yes` is the default for Apache2 in the recent Ubuntu versions, which may explain some of your problems.

EML

6/1/24, 2:39 PM

@HBruijn - interesting - I think you must be right, but both machines have `PrivateTmp` true. So I presume that one Apache managed to run both `program1` and `program2` in the same process, and the other has them in different processes, so maybe the `remove` is irrelevant. Both apaches have `mod_cgid`. I think the answer must be to set `PrivateTmp` off. But why is it on anyway? This isn't a system service - why would `systemd` care if I want to have two Apache processes talking to each other?

HBruijn

6/1/24, 2:52 PM

It might be interesting to see what happens when you don't use `/tmp` to store your files and/or (temporarily) disable the `PrivateTmp=true` setting by changing the apache2.service systemd unit file (see https://serverfault.com/a/840999/37681)

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: Ubuntu/Apache: /tmp files unexpectedly removed

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.