The location of the core file depends on /proc/sys/kernel/core_pattern
E.g. on Ubuntu 22.04:
cat /proc/sys/kernel/core_pattern
gives:
|/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E
The magic leading pipe syntax means that the Linux kernel will call the program /usr/share/apport/apport with a bunch of informational arguments and then the apport executable will then handle and store the dump somewhere.
apport in particular is part of Ubuntu's ultra automated error statistics setup, I think Canonical keeps a database of all crashes by users who agree to provide them to help prioritize development https://wiki.ubuntu.com/Apport
If you want to get a simple raw core file in the current directory as Linux intended, you have to instead set it to something like core with:
echo 'core' | sudo tee /proc/sys/kernel/core_pattern
How to change it persistently across reboot: How to permanently edit the core_pattern file?
If you do that and then:
ulimit -c unlimited
the current kernel 5.15.0 dumps files with form:
core.<pid>
e.g.:
core.494536
under the current working directory.
It is also apparently possible to turn off apport with: How do I enable or disable Apport?
sudo systemctl disable apport.service
You can test this e.g. with a minimal C program:
segfault.c
#include <stdlib.h>
int main(void) {
*(int *)0 = 1;
return EXIT_SUCCESS;
}
compile and run with:
gcc -ggdb3 -O0 -pedantic-errors -std=c89 -Wall -Wextra -o segfault.out segfault.c
./segfault.out
How to obtain apport core dumps?
Explained at: https://stackoverflow.com/questions/2065912/core-dumped-but-core-file-is-not-in-the-current-directory/47481884#47481884
Let's learn to use it if you decide to keep Ubuntu's mega opinionated apport system.
Maybe it is the for the best that non programs won't get random core.1234 files randomly created in their directories from time to time.
apport stores core dumps under .crash files under /var/crash/. .crash files are wrappers that contain the core dumps and some more system logs to help Ubuntu devs debug.
If you just run:
./segfault.out
you don't get anything. "Great" default behavior!
But Where do I find the core dump in ubuntu 16.04LTS? explains that we can look for apport logs under:
cat /var/log/apport.log
which now contains entries of type:
ERROR: apport (pid 503174) Sat Nov 26 21:51:47 2022: called for pid 503173, signal 11, core limit 18446744073709551615, dump mode 1
ERROR: apport (pid 503174) Sat Nov 26 21:51:47 2022: ignoring implausibly big core limit, treating as unlimited
ERROR: apport (pid 503174) Sat Nov 26 21:51:47 2022: executable: /home/ciro/segfault.out (command line "./segfault.out")
ERROR: apport (pid 503174) Sat Nov 26 21:51:47 2022: executable does not belong to a package, ignoring
ERROR: apport (pid 503174) Sat Nov 26 21:51:47 2022: writing core dump to core._home_ciro_segfault_out.1000.57a7653e-d57e-4871-ad0d-f7b8d9f5b3b9.503173.5035472 (limit: -1)
and no file is generated as per the ignore.
https://stackoverflow.com/questions/14204961/how-to-change-apport-default-behaviour-for-non-packaged-application-crashes asks how to enable the cores so we try:
mkdir -p ~/.config/apport
printf '[main]
unpackaged=true
' >> ~/.config/apport/settings
and now it works, we have a .crash file under /var/crash:
-rw-r--r-- 1 ciro whoopsie 0 Nov 26 22:09 _home_ciro_segfault.out.1000.upload
-rw-r----- 1 ciro whoopsie 137K Nov 26 22:09 _home_ciro_segfault.out.1000.crash
-rw------- 1 whoopsie whoopsie 37 Nov 26 22:09 _home_ciro_segfault.out.1000.uploaded
We can then extract the core dump with:
apport-unpack /var/crash/_home_ciro_segfault.out.1000.crash segfault
which creates a directory named segfault/ containing a bunch of files split out of the .crash file, including our core dump segfault/CoreDump.
The program does crash however due to a bug, it is pitiful:
Traceback (most recent call last):
File "/usr/bin/apport-unpack", line 77, in <module>
pr.extract_keys(f, bin_keys, dir)
File "/usr/lib/python3/dist-packages/problem_report.py", line 269, in extract_keys
raise ValueError('%s has no binary content' %
ValueError: ['separator'] has no binary content
Report at: https://bugs.launchpad.net/ubuntu/+source/apport/+bug/1889443 But it crashes after generating our CoreDumpso we can successfully run:
gdb segfault.out segfault/CoreDump
There is also a potentially more automated way with:
apport-retrace /var/crash/_home_ciro_segfault.out.1000.crash
but it fails for non packaged programs with:
ERROR: report file does not contain one of the required fields: Package
Tested on Ubuntu 22.04.
How to prevent "this executable already crashed 2 times, ignoring"?
If you keep testing with our minimal ./segfault.out, you will notice that at some point .crash files stop being generated.
cat /var/log/apport.log
then clarifies that:
this executable already crashed 2 times, ignoring
OMG this system was not made with devs in mind!
Looking at at source code from apt source apport does not seem configurable at first sight:
crash_counter = 0
# Create crash report file descriptor for writing the report into
# report_dir
try:
report = '%s/%s.%i.crash' % (apport.fileutils.report_dir, info['ExecutablePath'].replace('/', '_'), pidstat.st_uid)
if os.path.exists(report):
if apport.fileutils.seen_report(report):
# do not flood the logs and the user with repeated crashes
# and make sure the file isn't a FIFO or symlink
fd = os.open(report, os.O_NOFOLLOW | os.O_RDONLY | os.O_NONBLOCK)
st = os.fstat(fd)
if stat.S_ISREG(st.st_mode):
with os.fdopen(fd, 'rb') as f:
crash_counter = apport.fileutils.get_recent_crashes(f)
crash_counter += 1
if crash_counter > 1:
write_user_coredump(
pid, process_start, core_ulimit, coredump_fd
)
error_log('this executable already crashed %i times, ignoring' % crash_counter)
sys.exit(0)
# remove the old file, so that we can create the new one with
# os.O_CREAT|os.O_EXCL
os.unlink(report)
How to prevent apport from uploading/requesting to upload reports for non packaged binaries?
On Ubuntu 22.04 under:
- Settings
- Privacy
- Diagnostics
- Send error reports to Canonical
there are three possible options:
- Never
- Automatic, which seems to mean more precisely: "Always without asking"
- Manual, which in plain English means "Ask before sending" with a popup
I don't see how to prevent upload with the "Automatic" method. And "Manual" will drive you mad with popups. So the only reasonable options are "Never" or disabling apport:
How to analyze core dumps?
This is very Ubuntu independent, so just go for:
Mozilla rr reverse debugging as the ultimate "core file"
Core files allow you to inspect the stack at break.
But in general what you really need to do is to go back in time to further decide the root failure cause.
The amazing Mozilla rr allows you to do that, at the cost of a larger trace file, and a slight performance hit.
Example at: https://stackoverflow.com/questions/1470434/how-does-reverse-debugging-work/53063242#53063242
