Score:0

qemu-kvm crash "Resource deadlock avoided"

om flag

I meet qemu-kvm crash with this:

terminate called after throwing an instance of 'std::system_error'
  what():  Resource deadlock avoided
2022-10-10 13:40:53.238+0000: shutting down, reason=crashed

HOST OS: ubuntu 20.04

/usr/bin/qemu-system-x86_64 --version

QEMU emulator version 4.2.1 (Debian 1:4.2-3ubuntu6.23)
Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers

Is there any bug with:

qemu-system-x86/focal-updates,focal-security,now 1:4.2-3ubuntu6.23 amd64 [installed,automatic]

librbd:

librbd1/stable,now 16.2.10-1focal amd64 [installed,automatic]
libvirt-daemon-driver-storage-rbd/focal-updates,focal-security,now 6.0.0-0ubuntu8.16 amd64 [installed,automatic]
python3-rbd/stable,now 16.2.10-1focal amd64 [installed,automatic]

qemu-kvm boot from ceph rdb, will crash.

coredump:

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007f2c2aec3859 in __GI_abort () at abort.c:79
#2  0x00007f2c286f7911 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#3  0x00007f2c2870338c in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00007f2c28702369 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007f2c28702d21 in __gxx_personality_v0 () from /lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007f2c2b0c6bef in ?? () from /lib/x86_64-linux-gnu/libgcc_s.so.1
#7  0x00007f2c2b0c7281 in _Unwind_RaiseException () from /lib/x86_64-linux-gnu/libgcc_s.so.1
#8  0x00007f2c2870369c in __cxa_throw () from /lib/x86_64-linux-gnu/libstdc++.so.6
#9  0x00007f2c286fa73f in std::__throw_system_error(int) () from /lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00007f2c28730060 in std::thread::join() () from /lib/x86_64-linux-gnu/libstdc++.so.6
#11 0x00007f2c28965e14 in ?? () from /lib/librados.so.2
#12 0x00007f2c1b932cb1 in ceph::common::ConfigProxy::set_mon_vals(ceph::common::CephContext*, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<void>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>) ()
   from /usr/lib/ceph/libceph-common.so.2
#13 0x00007f2c1b91c3d5 in ?? () from /usr/lib/ceph/libceph-common.so.2
#14 0x00007f2c1b931c85 in boost::asio::detail::strand_service::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) ()
   from /usr/lib/ceph/libceph-common.so.2
#15 0x00007f2c28973ca2 in ?? () from /lib/librados.so.2
#16 0x00007f2c2897adea in ?? () from /lib/librados.so.2
#17 0x00007f2c2872fde4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#18 0x00007f2c2b09b609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#19 0x00007f2c2afc0133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

CEPH.CONF

[global]

#admin socket = /var/run/ceph/guests/-....asok
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
#public network = 10.3.0.0/24
mon_pg_warn_max_per_osd = 128
mon_max_pg_per_osd = 128
osd_recovery_max_active = 32
osd_max_backfills = 32
osd_op_num_threads_per_shard_hdd = 2
rbd_mirror_concurrent_image_deletions = 2
rbd_op_threads = 4
bluestore_compression_mode = aggressive
osd_pool_default_size = 4
osd_pool_default_min_size = 0
cluster_network = 10.3.0.0/24
#cluster_network_interface = ens255f1
#public_network_interface = ens255f1
[client]
rgw frontends = civetweb port=80
rbd cache = true
rbd cache writethrough until flush = true
#admin socket = /var/run/ceph/guests/-....asok
#log file = /var/log/qemu/qemu-guest-.log
rbd concurrent management ops = 20
osd_op_num_threads_per_shard_hdd = 2
rbd_mirror_concurrent_image_deletions = 2
rbd_op_threads = 4
[osd]
osd_recovery_max_active = 32
osd_max_backfills = 32
osd_op_num_threads_per_shard_hdd = 2
rbd_mirror_concurrent_image_deletions = 2
rbd_op_threads = 4
bluestore_compression_mode = aggressive
[mon]
mon allow pool delete = true
David avatar
cn flag
The copy write info you show here tells me this project may have not been updated since 2019. Are you sure it works with this version of Ubuntu?
notpeter avatar
uz flag
Can you provide additional configuration details? (e.g. what are you trying to run with this happens, resources for your VM, etc)? @David This version is the latest package for Ubuntu20.
David avatar
cn flag
Ubuntu Core 20?
Nan B avatar
om flag
vm boot to netboot.xys image as vda and do nothing. @notpeter. Is there any way to generate a coredump when qemu-kvm crash?
Nan B avatar
om flag
qemu-kvm/focal-updates,focal-security,now 1:4.2-3ubuntu6.23 amd64 [installed,automatic] @David
Score:0
mk flag

The only occasion I've seen this exact message from QEMU was with weird configuration. Essentially librbd is crashing and with it taking down the whole QEMU process.

Could be a configuration option like rbd_op_threads (don't play with this option in production, leave it at the defualt value of 1).

If it's not that, can you please paste your ceph.conf?

Nan B avatar
om flag
Hi, ceph.conf added. Thanks a lot.
stathis avatar
mk flag
Hey, great. Please comment out `rbd_op_threads = 4` completely. This might be your culprit.
Nan B avatar
om flag
Hi, Thanks, is it need to restart osd.x daemon or ?
stathis avatar
mk flag
You just need to stop/start (NOT restart) the virtual machines using librbd, so they read the new config.
Nan B avatar
om flag
Thanks a lot. Is there any way to know how many rbd op thread is the vm using? So that I don't need to wait long time for crash observing.
stathis avatar
mk flag
Hi, well, when it's more than `1`, your VMs can crash even when you restart ceph-mon, ceph-osd etc. So if you make your OSDs or monitors restart and the VM doesn't crash, you should be OK.
Nan B avatar
om flag
Hi, Thanks for your tips!
stathis avatar
mk flag
No problem has it been solved then?
Nan B avatar
om flag
Yes! Thanks a lot!
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.