I just setup three storage nodes for ceph in a qemu/kvm environment to learn about it. OS is Ubuntu Server 22.04 LTS. The ceph and ceph-common packages are from standard repos. I followed the guidelines on the ceph website. I set up one vm and cloned it before installing ceph. The hosts and the dashboards are up and running.
Each host has a empty block device /dev/vdb. According to the guide I tried to add them by
ceph orch apply osd --all-available-devices
It worked for two nodes but failed for the third. The daemon tries to integrate the block device with a failure. It retries every 10s. Unfortuntely, the error message is almost unreadable:
5/31/23 2:29:05 PM[WRN]Health check update: Failed to apply 2 service(s): osd.all-available-devices,osd.cost_capacity (CEPHADM_APPLY_SPEC_FAIL)
5/31/23 2:29:05 PM[ERR]Failed to apply osd.cost_capacity spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd service_id: cost_capacity service_name: osd.cost_capacity placement: host_pattern: '*' spec: data_devices: rotational: 1 filter_logic: AND objectstore: bluestore ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/cd1df3a0-ff28-11ed-b01a-525400e83d82/mon.san2/config Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:b8df01a568f4dec7bac6d5040f9391dcca14e00ec7f4de8a3dcf3f2a6502d3a9 -e NODE_NAME=san2 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=cost_capacity -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cd1df3a0-ff28-11ed-b01a-525400e83d82:/var/run/ceph:z -v /var/log/ceph/cd1df3a0-ff28-11ed-b01a-525400e83d82:/var/log/ceph:z -v /var/lib/ceph/cd1df3a0-ff28-11ed-b01a-525400e83d82/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpu0fdkklo:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpbb0nzrrm:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:b8df01a568f4dec7bac6d5040f9391dcca14e00ec7f4de8a3dcf3f2a6502d3a9 lvm batch --no-auto /dev/vdb --yes --no-systemd /usr/bin/docker: stderr --> passed data devices: 1 physical, 0 LVM /usr/bin/docker: stderr --> relative data size: 1.0 /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new fec3748c-f6d3-4388-9402-1bde5664df85 /usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgcreate --force --yes ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86 /dev/vdb /usr/bin/docker: stderr stdout: Physical volume "/dev/vdb" successfully created. /usr/bin/docker: stderr stdout: Volume group "ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86" successfully created /usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/lvcreate --yes -l 2559 -n osd-block-fec3748c-f6d3-4388-9402-1bde5664df85 ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86 /usr/bin/docker: stderr stdout: Logical volume "osd-block-fec3748c-f6d3-4388-9402-1bde5664df85" created. /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key /usr/bin/docker: stderr Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-1 /usr/bin/docker: stderr Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86/osd-block-fec3748c-f6d3-4388-9402-1bde5664df85 /usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 /usr/bin/docker: stderr Running command: /usr/bin/ln -s /dev/ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86/osd-block-fec3748c-f6d3-4388-9402-1bde5664df85 /var/lib/ceph/osd/ceph-1/block /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-1/activate.monmap /usr/bin/docker: stderr stderr: got monmap epoch 3 /usr/bin/docker: stderr --> Creating keyring file for osd.1 /usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1/keyring /usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1/ /usr/bin/docker: stderr Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 1 --monmap /var/lib/ceph/osd/ceph-1/activate.monmap --keyfile - --osdspec-affinity cost_capacity --osd-data /var/lib/ceph/osd/ceph-1/ --osd-uuid fec3748c-f6d3-4388-9402-1bde5664df85 --setuser ceph --setgroup ceph /usr/bin/docker: stderr stderr: 2023-05-31T12:29:03.012+0000 7f4ccaaed540 -1 bluestore(/var/lib/ceph/osd/ceph-1/) _read_fsid unparsable uuid /usr/bin/docker: stderr stderr: 2023-05-31T12:29:03.212+0000 7f4ccaaed540 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2023-05-31T12:29:03.212+0000 7f4ccaaed540 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2023-05-31T12:29:03.212+0000 7f4ccaaed540 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2023-05-31T12:29:03.212+0000 7f4ccaaed540 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2023-05-31T12:29:03.216+0000 7f4ccaaed540 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2023-05-31T12:29:03.216+0000 7f4ccaaed540 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2023-05-31T12:29:03.536+0000 7f4ccaaed540 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2023-05-31T12:29:03.536+0000 7f4ccaaed540 -1 bdev(0x55746fa91400 /var/lib/ceph/osd/ceph-1//block) open open got: (13) Permission denied /usr/bin/docker: stderr stderr: 2023-05-31T12:29:03.536+0000 7f4ccaaed540 -1 OSD::mkfs: ObjectStore::mkfs failed with error (13) Permission denied /usr/bin/docker: stderr stderr: 2023-05-31T12:29:03.536+0000 7f4ccaaed540 -1 [0;31m ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-1/: (13) Permission denied[0m /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.1 --yes-i-really-mean-it /usr/bin/docker: stderr stderr: purged osd.1 /usr/bin/docker: stderr --> Zapping: /dev/ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86/osd-block-fec3748c-f6d3-4388-9402-1bde5664df85 /usr/bin/docker: stderr --> Unmounting /var/lib/ceph/osd/ceph-1 /usr/bin/docker: stderr Running command: /usr/bin/umount -v /var/lib/ceph/osd/ceph-1 /usr/bin/docker: stderr stderr: umount: /var/lib/ceph/osd/ceph-1 unmounted /usr/bin/docker: stderr Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86/osd-block-fec3748c-f6d3-4388-9402-1bde5664df85 bs=1M count=10 conv=fsync /usr/bin/docker: stderr stderr: 10+0 records in /usr/bin/docker: stderr 10+0 records out /usr/bin/docker: stderr stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0415005 s, 253 MB/s /usr/bin/docker: stderr --> Only 1 LV left in VG, will proceed to destroy volume group ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86 /usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgremove -v -f ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86 /usr/bin/docker: stderr stderr: Removing ceph--7408d699--5fa5--4ad7--94c6--364338c2ef86-osd--block--fec3748c--f6d3--4388--9402--1bde5664df85 (253:0) /usr/bin/docker: stderr stderr: Archiving volume group "ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86" metadata (seqno 5). /usr/bin/docker: stderr stderr: Releasing logical volume "osd-block-fec3748c-f6d3-4388-9402-1bde5664df85" /usr/bin/docker: stderr stderr: Creating volume group backup "/etc/lvm/backup/ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86" (seqno 6). /usr/bin/docker: stderr stdout: Logical volume "osd-block-fec3748c-f6d3-4388-9402-1bde5664df85" successfully removed /usr/bin/docker: stderr stderr: Removing physical volume "/dev/vdb" from volume group "ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86" /usr/bin/docker: stderr stdout: Volume group "ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86" successfully removed /usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/pvremove -v -f -f /dev/vdb /usr/bin/docker: stderr stdout: Labels on physical volume "/dev/vdb" successfully wiped. /usr/bin/docker: stderr --> Zapping successful for OSD: 1 /usr/bin/docker: stderr Traceback (most recent call last): /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ /usr/bin/docker: stderr self.main(self.argv) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc /usr/bin/docker: stderr return f(*a, **kw) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch /usr/bin/docker: stderr instance.main() /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch /usr/bin/docker: stderr instance.main() /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root /usr/bin/docker: stderr return func(*a, **kw) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 441, in main /usr/bin/docker: stderr self._execute(plan) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 460, in _execute /usr/bin/docker: stderr c.create(argparse.Namespace(**args)) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root /usr/bin/docker: stderr return func(*a, **kw) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create /usr/bin/docker: stderr prepare_step.safe_prepare(args) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 252, in safe_prepare /usr/bin/docker: stderr self.prepare() /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root /usr/bin/docker: stderr return func(*a, **kw) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 394, in prepare /usr/bin/docker: stderr osd_fsid, /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 119, in prepare_bluestore /usr/bin/docker: stderr db=db /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 484, in osd_mkfs_bluestore /usr/bin/docker: stderr raise RuntimeError('Command failed with exit code %s: %s' % (returncode, ' '.join(command))) /usr/bin/docker: stderr RuntimeError: Command failed with exit code 250: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 1 --monmap /var/lib/ceph/osd/ceph-1/activate.monmap --keyfile - --osdspec-affinity cost_capacity --osd-data /var/lib/ceph/osd/ceph-1/ --osd-uuid fec3748c-f6d3-4388-9402-1bde5664df85 --setuser ceph --setgroup ceph Traceback (most recent call last): File "/var/lib/ceph/cd1df3a0-ff28-11ed-b01a-525400e83d82/cephadm.7ab03136237675497d535fb1b85d1d0f95bbe5b95f32cd4e6f3ca71a9f97bf3c", line 9653, in main() File "/var/lib/ceph/cd1df3a0-ff28-11ed-b01a-525400e83d82/cephadm.7ab03136237675497d535fb1b85d1d0f95bbe5b95f32cd4e6f3ca71a9f97bf3c", line 9641, in main r = ctx.func(ctx) File "/var/lib/ceph/cd1df3a0-ff28-11ed-b01a-525400e83d82/cephadm.7ab03136237675497d535fb1b85d1d0f95bbe5b95f32cd4e6f3ca71a9f97bf3c", line 2153, in _infer_config return func(ctx) File "/var/lib/ceph/cd1df3a0-ff28-11ed-b01a-525400e83d82/cephadm.7ab03136237675497d535fb1b85d1d0f95bbe5b95f32cd4e6f3ca71a9f97bf3c", line 2069, in _infer_fsid return func(ctx) File "/var/lib/ceph/cd1df3a0-ff28-11ed-b01a-525400e83d82/cephadm.7ab03136237675497d535fb1b85d1d0f95bbe5b95f32cd4e6f3ca71a9f97bf3c", line 2181, in _infer_image return func(ctx) File "/var/lib/ceph/cd1df3a0-ff28-11ed-b01a-525400e83d82/cephadm.7ab03136237675497d535fb1b85d1d0f95bbe5b95f32cd4e6f3ca71a9f97bf3c", line 2056, in _validate_fsid return func(ctx) File "/var/lib/ceph/cd1df3a0-ff28-11ed-b01a-525400e83d82/cephadm.7ab03136237675497d535fb1b85d1d0f95bbe5b95f32cd4e6f3ca71a9f97bf3c", line 6254, in command_ceph_volume out, err, code = call_throws(ctx, c.run_cmd(), verbosity=CallVerbosity.QUIET_UNLESS_ERROR) File "/var/lib/ceph/cd1df3a0-ff28-11ed-b01a-525400e83d82/cephadm.7ab03136237675497d535fb1b85d1d0f95bbe5b95f32cd4e6f3ca71a9f97bf3c", line 1853, in call_throws raise RuntimeError('Failed command: %s' % ' '.join(command)) RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:b8df01a568f4dec7bac6d5040f9391dcca14e00ec7f4de8a3dcf3f2a6502d3a9 -e NODE_NAME=san2 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=cost_capacity -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cd1df3a0-ff28-11ed-b01a-525400e83d82:/var/run/ceph:z -v /var/log/ceph/cd1df3a0-ff28-11ed-b01a-525400e83d82:/var/log/ceph:z -v /var/lib/ceph/cd1df3a0-ff28-11ed-b01a-525400e83d82/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpu0fdkklo:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpbb0nzrrm:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:b8df01a568f4dec7bac6d5040f9391dcca14e00ec7f4de8a3dcf3f2a6502d3a9 lvm batch --no-auto /dev/vdb --yes --no-systemd Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 509, in _apply_all_services if self._apply_service(spec): File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 77, in create_from_spec ret = self.mgr.wait_async(all_hosts()) File "/usr/share/ceph/mgr/cephadm/module.py", line 615, in wait_async return self.event_loop.get_result(coro) File "/usr/share/ceph/mgr/cephadm/ssh.py", line 56, in get_result return asyncio.run_coroutine_threadsafe(coro, self._loop).result() File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result return self.__get_result() File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result raise self._exception File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 75, in all_hosts return await gather(*futures) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 64, in create_from_spec_one replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 95, in create_single_host code, '\n'.join(err))) RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/cd1df3a0-ff28-11ed-b01a-525400e83d82/mon.san2/config Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:b8df01a568f4dec7bac6d5040f9391dcca14e00ec7f4de8a3dcf3f2a6502d3a9 -e NODE_NAME=san2 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=cost_capacity -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/cd1df3a0-ff28-11ed-b01a-525400e83d82:/var/run/ceph:z -v /var/log/ceph/cd1df3a0-ff28-11ed-b01a-525400e83d82:/var/log/ceph:z -v /var/lib/ceph/cd1df3a0-ff28-11ed-b01a-525400e83d82/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpu0fdkklo:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpbb0nzrrm:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:b8df01a568f4dec7bac6d5040f9391dcca14e00ec7f4de8a3dcf3f2a6502d3a9 lvm batch --no-auto /dev/vdb --yes --no-systemd /usr/bin/docker: stderr --> passed data devices: 1 physical, 0 LVM /usr/bin/docker: stderr --> relative data size: 1.0 /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new fec3748c-f6d3-4388-9402-1bde5664df85 /usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgcreate --force --yes ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86 /dev/vdb /usr/bin/docker: stderr stdout: Physical volume "/dev/vdb" successfully created. /usr/bin/docker: stderr stdout: Volume group "ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86" successfully created /usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/lvcreate --yes -l 2559 -n osd-block-fec3748c-f6d3-4388-9402-1bde5664df85 ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86 /usr/bin/docker: stderr stdout: Logical volume "osd-block-fec3748c-f6d3-4388-9402-1bde5664df85" created. /usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key /usr/bin/docker: stderr Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-1 /usr/bin/docker: stderr Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86/osd-block-fec3748c-f6d3-4388-9402-1bde5664df85 /usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 /usr/bin/docker: stderr Running command: /usr/bin/ln -s /dev/ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86/osd-block-fec3748c-f6d3-4388-9402-1bde5664df85 /var/lib/ceph/osd/ceph-1/block /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-1/activate.monmap /usr/bin/docker: stderr stderr: got monmap epoch 3 /usr/bin/docker: stderr --> Creating keyring file for osd.1 /usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1/keyring /usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1/ /usr/bin/docker: stderr Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 1 --monmap /var/lib/ceph/osd/ceph-1/activate.monmap --keyfile - --osdspec-affinity cost_capacity --osd-data /var/lib/ceph/osd/ceph-1/ --osd-uuid fec3748c-f6d3-4388-9402-1bde5664df85 --setuser ceph --setgroup ceph /usr/bin/docker: stderr stderr: 2023-05-31T12:29:03.012+0000 7f4ccaaed540 -1 bluestore(/var/lib/ceph/osd/ceph-1/) _read_fsid unparsable uuid /usr/bin/docker: stderr stderr: 2023-05-31T12:29:03.212+0000 7f4ccaaed540 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2023-05-31T12:29:03.212+0000 7f4ccaaed540 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2023-05-31T12:29:03.212+0000 7f4ccaaed540 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2023-05-31T12:29:03.212+0000 7f4ccaaed540 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2023-05-31T12:29:03.216+0000 7f4ccaaed540 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2023-05-31T12:29:03.216+0000 7f4ccaaed540 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2023-05-31T12:29:03.536+0000 7f4ccaaed540 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1//block: (13) Permission denied /usr/bin/docker: stderr stderr: 2023-05-31T12:29:03.536+0000 7f4ccaaed540 -1 bdev(0x55746fa91400 /var/lib/ceph/osd/ceph-1//block) open open got: (13) Permission denied /usr/bin/docker: stderr stderr: 2023-05-31T12:29:03.536+0000 7f4ccaaed540 -1 OSD::mkfs: ObjectStore::mkfs failed with error (13) Permission denied /usr/bin/docker: stderr stderr: 2023-05-31T12:29:03.536+0000 7f4ccaaed540 -1 [0;31m ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-1/: (13) Permission denied[0m /usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.1 --yes-i-really-mean-it /usr/bin/docker: stderr stderr: purged osd.1 /usr/bin/docker: stderr --> Zapping: /dev/ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86/osd-block-fec3748c-f6d3-4388-9402-1bde5664df85 /usr/bin/docker: stderr --> Unmounting /var/lib/ceph/osd/ceph-1 /usr/bin/docker: stderr Running command: /usr/bin/umount -v /var/lib/ceph/osd/ceph-1 /usr/bin/docker: stderr stderr: umount: /var/lib/ceph/osd/ceph-1 unmounted /usr/bin/docker: stderr Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86/osd-block-fec3748c-f6d3-4388-9402-1bde5664df85 bs=1M count=10 conv=fsync /usr/bin/docker: stderr stderr: 10+0 records in /usr/bin/docker: stderr 10+0 records out /usr/bin/docker: stderr stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0415005 s, 253 MB/s /usr/bin/docker: stderr --> Only 1 LV left in VG, will proceed to destroy volume group ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86 /usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgremove -v -f ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86 /usr/bin/docker: stderr stderr: Removing ceph--7408d699--5fa5--4ad7--94c6--364338c2ef86-osd--block--fec3748c--f6d3--4388--9402--1bde5664df85 (253:0) /usr/bin/docker: stderr stderr: Archiving volume group "ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86" metadata (seqno 5). /usr/bin/docker: stderr stderr: Releasing logical volume "osd-block-fec3748c-f6d3-4388-9402-1bde5664df85" /usr/bin/docker: stderr stderr: Creating volume group backup "/etc/lvm/backup/ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86" (seqno 6). /usr/bin/docker: stderr stdout: Logical volume "osd-block-fec3748c-f6d3-4388-9402-1bde5664df85" successfully removed /usr/bin/docker: stderr stderr: Removing physical volume "/dev/vdb" from volume group "ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86" /usr/bin/docker: stderr stdout: Volume group "ceph-7408d699-5fa5-4ad7-94c6-364338c2ef86" successfully removed /usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/pvremove -v -f -f /dev/vdb /usr/bin/docker: stderr stdout: Labels on physical volume "/dev/vdb" successfully wiped. /usr/bin/docker: stderr --> Zapping successful for OSD: 1 /usr/bin/docker: stderr Traceback (most recent call last): /usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in /usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ /usr/bin/docker: stderr self.main(self.argv) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc /usr/bin/docker: stderr return f(*a, **kw) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main /usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch /usr/bin/docker: stderr instance.main() /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main /usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv) /usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
[...]
The important part seems to be (reformatted):
/usr/bin/docker: stderr stderr: 2023-05-31T12:31:39.132+0000 7fa204745540 -1 bluestore(/var/lib/ceph/osd/ceph-1/) _read_fsid unparsable uuid /usr/bin/docker: stderr stderr: 2023-05-31T12:31:39.308+0000 7fa204745540 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1//block: (13) Permission denied
/usr/bin/docker: stderr stderr: 2023-05-31T12:31:39.312+0000 7fa204745540 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1//block: (13) Permission denied
/usr/bin/docker: stderr stderr: 2023-05-31T12:31:39.312+0000 7fa204745540 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1//block: (13) Permission denied
/usr/bin/docker: stderr stderr: 2023-05-31T12:31:39.312+0000 7fa204745540 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1//block: (13) Permission denied
/usr/bin/docker: stderr stderr: 2023-05-31T12:31:39.312+0000 7fa204745540 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1//block: (13) Permission denied
/usr/bin/docker: stderr stderr: 2023-05-31T12:31:39.312+0000 7fa204745540 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1//block: (13) Permission denied
/usr/bin/docker: stderr stderr: 2023-05-31T12:31:39.668+0000 7fa204745540 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1//block: (13) Permission denied
/usr/bin/docker: stderr stderr: 2023-05-31T12:31:39.668+0000 7fa204745540 -1 bdev(0x55897a203400 /var/lib/ceph/osd/ceph-1//block) open open got: (13) Permission denied /usr/bin/docker: stderr stderr: 2023-05-31T12:31:39.668+0000 7fa204745540 -1 OSD::mkfs: ObjectStore::mkfs failed with error (13) Permission denied
/usr/bin/docker: stderr stderr: 2023-05-31T12:31:39.668+0000 7fa204745540 -1 [0;31m ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-1/: (13) Permission denied[0m
/usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes /usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.1 --yes-i-really-mean-it /usr/bin/docker: stderr stderr: purged osd.1
/usr/bin/docker: stderr --> Zapping: /dev/ceph-d36a727e-b4e7-43c9-aa8b-bd8d82c0f84f/osd-block-42b7003b-a98d-4020-8513-07299f23c63a /usr/bin/docker: stderr --> Unmounting /var/lib/ceph/osd/ceph-1
/usr/bin/docker: stderr Running command: /usr/bin/umount -v /var/lib/ceph/osd/ceph-1
/usr/bin/docker: stderr stderr: umount: /var/lib/ceph/osd/ceph-1 unmounted /usr/bin/docker: stderr Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-d36a727e-b4e7-43c9-aa8b-bd8d82c0f84f/osd-block-42b7003b-a98d-4020-8513-07299f23c63a bs=1M count=10 conv=fsync
I tried to give write permission to the whole tree /var/lib/ceph for anybody. But it did not help.
It is already the second time to start from scratch. The other time I had the same error on another node.
