I have an XCP-ng cluster running XCP-ng 8.2.1 and following a patch and reboot the local storage SR is showing as unplugged. This is an XCP create LVM on a RAID drive, the same RAID drives contain the OS which is booting and I can see evidence of any actual disk issues.
Asking the stack to plug the SR in (either in XO or with the command line) results in the same error:
[15:39 hal02 log]# xe pbd-plug uuid=99d59bb4-284e-605c-76c1-547daac0294a
Error code: SR_BACKEND_FAILURE_1200
Error parameters: , Invalid argument,
And the full recorded XCP log entry is as follows:
pbd.connect
{
"id": "99d59bb4-284e-605c-76c1-547daac0294a"
}
{
"code": "SR_BACKEND_FAILURE_1200",
"params": [
"",
"Invalid argument",
""
],
"task": {
"uuid": "608314d8-ae78-90f1-0659-8ffad0aaffb6",
"name_label": "Async.PBD.plug",
"name_description": "",
"allowed_operations": [],
"current_operations": {},
"created": "20230609T14:49:04Z",
"finished": "20230609T14:49:07Z",
"status": "failure",
"resident_on": "OpaqueRef:1557b1ab-f767-4943-b44a-51137351dc7c",
"progress": 1,
"type": "<none/>",
"result": "",
"error_info": [
"SR_BACKEND_FAILURE_1200",
"",
"Invalid argument",
""
],
"other_config": {},
"subtask_of": "OpaqueRef:NULL",
"subtasks": [],
"backtrace": "(((process xapi)(filename lib/backtrace.ml)(line 210))((process xapi)(filename ocaml/xapi/storage_access.ml)(line 32))((process xapi)(filename ocaml/xapi/xapi_pbd.ml)(line 194))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 131))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/xapi/rbac.ml)(line 205))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 95)))"
},
"message": "SR_BACKEND_FAILURE_1200(, Invalid argument, )",
"name": "XapiError",
"stack": "XapiError: SR_BACKEND_FAILURE_1200(, Invalid argument, )
at Function.wrap (/etc/xen-orchestra/packages/xen-api/src/_XapiError.js:16:12)
at _default (/etc/xen-orchestra/packages/xen-api/src/_getTaskResult.js:11:29)
at Xapi._addRecordToCache (/etc/xen-orchestra/packages/xen-api/src/index.js:954:24)
at forEach (/etc/xen-orchestra/packages/xen-api/src/index.js:988:14)
at Array.forEach (<anonymous>)
at Xapi._processEvents (/etc/xen-orchestra/packages/xen-api/src/index.js:978:12)
at Xapi._watchEvents (/etc/xen-orchestra/packages/xen-api/src/index.js:1144:14)"
}
Here is the output from lsblk showing the setup - you can see the volume group and also the local volumes created:
[15:39 hal02 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdb 8:16 1 972M 0 disk
├─sdb4 8:20 1 4M 0 part
├─sdb7 8:23 1 110M 0 part
├─sdb5 8:21 1 250M 0 part
├─sdb1 8:17 1 1K 0 part
├─sdb8 8:24 1 286M 0 part
└─sdb6 8:22 1 250M 0 part
sr0 11:0 1 1024M 0 rom
sda 8:0 0 5.5T 0 disk
├─sda4 8:4 0 512M 0 part
├─sda2 8:2 0 18G 0 part
├─sda5 8:5 0 4G 0 part /var/log
├─sda3 8:3 0 5.4T 0 part
│ ├─VG_XenStorage--e0b493c8--041d--345e--84d6--1cdac0fe17c9-VHD--908b187e--dd22--4bb2--9e63--af874ee98f72 253:1 0 200.4G 0 lvm
│ ├─VG_XenStorage--e0b493c8--041d--345e--84d6--1cdac0fe17c9-VHD--01a2a59f--5f5c--45c3--baa5--08d520c94d96 253:10 0 8M 0 lvm
│ ├─VG_XenStorage--e0b493c8--041d--345e--84d6--1cdac0fe17c9-VHD--b784f499--3084--42db--afbe--7048993c7cd4 253:8 0 8M 0 lvm
│ ├─VG_XenStorage--e0b493c8--041d--345e--84d6--1cdac0fe17c9-VHD--0efcf947--faf7--40a4--97c2--3cf12a842cf3 253:6 0 8M 0 lvm
│ ├─VG_XenStorage--e0b493c8--041d--345e--84d6--1cdac0fe17c9-VHD--71a27718--4a03--437f--b0ad--3b3dbfd53600 253:4 0 24.4G 1 lvm
│ ├─VG_XenStorage--e0b493c8--041d--345e--84d6--1cdac0fe17c9-VHD--0e9f8505--c05f--4856--9bde--15edf45038c0 253:2 0 64.1G 0 lvm
│ ├─VG_XenStorage--e0b493c8--041d--345e--84d6--1cdac0fe17c9-VHD--c4554de9--d378--473c--ab26--71f7b64d5b65 253:11 0 100M 0 lvm
│ ├─VG_XenStorage--e0b493c8--041d--345e--84d6--1cdac0fe17c9-MGT 253:0 0 4M 0 lvm
│ ├─VG_XenStorage--e0b493c8--041d--345e--84d6--1cdac0fe17c9-VHD--ad7c8561--1044--4eb5--9acd--00c548280410 253:9 0 236M 0 lvm
│ ├─VG_XenStorage--e0b493c8--041d--345e--84d6--1cdac0fe17c9-VHD--7b5feb84--a722--4e63--a260--b8fcd7dabac8 253:7 0 812M 1 lvm
│ ├─VG_XenStorage--e0b493c8--041d--345e--84d6--1cdac0fe17c9-VHD--e764c429--fb6f--4b38--9a49--41717f92ccbe 253:5 0 540M 1 lvm
│ └─VG_XenStorage--e0b493c8--041d--345e--84d6--1cdac0fe17c9-VHD--87343508--90df--498a--a504--11440e091d38 253:3 0 200.4G 0 lvm
├─sda1 8:1 0 18G 0 part /
└─sda6 8:6 0 1G 0 part [SWAP]
And the output from lvscan:
[15:50 hal02 ~]# lvscan
ACTIVE '/dev/VG_XenStorage-e0b493c8-041d-345e-84d6-1cdac0fe17c9/MGT' [4.00 MiB] inherit
ACTIVE '/dev/VG_XenStorage-e0b493c8-041d-345e-84d6-1cdac0fe17c9/VHD-908b187e-dd22-4bb2-9e63-af874ee98f72' [<200.40 GiB] inherit
ACTIVE '/dev/VG_XenStorage-e0b493c8-041d-345e-84d6-1cdac0fe17c9/VHD-0e9f8505-c05f-4856-9bde-15edf45038c0' [64.13 GiB] inherit
ACTIVE '/dev/VG_XenStorage-e0b493c8-041d-345e-84d6-1cdac0fe17c9/VHD-87343508-90df-498a-a504-11440e091d38' [<200.40 GiB] inherit
ACTIVE '/dev/VG_XenStorage-e0b493c8-041d-345e-84d6-1cdac0fe17c9/VHD-71a27718-4a03-437f-b0ad-3b3dbfd53600' [24.39 GiB] inherit
ACTIVE '/dev/VG_XenStorage-e0b493c8-041d-345e-84d6-1cdac0fe17c9/VHD-e764c429-fb6f-4b38-9a49-41717f92ccbe' [540.00 MiB] inherit
ACTIVE '/dev/VG_XenStorage-e0b493c8-041d-345e-84d6-1cdac0fe17c9/VHD-0efcf947-faf7-40a4-97c2-3cf12a842cf3' [8.00 MiB] inherit
ACTIVE '/dev/VG_XenStorage-e0b493c8-041d-345e-84d6-1cdac0fe17c9/VHD-7b5feb84-a722-4e63-a260-b8fcd7dabac8' [812.00 MiB] inherit
ACTIVE '/dev/VG_XenStorage-e0b493c8-041d-345e-84d6-1cdac0fe17c9/VHD-b784f499-3084-42db-afbe-7048993c7cd4' [8.00 MiB] inherit
ACTIVE '/dev/VG_XenStorage-e0b493c8-041d-345e-84d6-1cdac0fe17c9/VHD-ad7c8561-1044-4eb5-9acd-00c548280410' [236.00 MiB] inherit
ACTIVE '/dev/VG_XenStorage-e0b493c8-041d-345e-84d6-1cdac0fe17c9/VHD-01a2a59f-5f5c-45c3-baa5-08d520c94d96' [8.00 MiB] inherit
ACTIVE '/dev/VG_XenStorage-e0b493c8-041d-345e-84d6-1cdac0fe17c9/VHD-c4554de9-d378-473c-ab26-71f7b64d5b65' [100.00 MiB] inherit
inactive '/dev/VG_XenStorage-e0b493c8-041d-345e-84d6-1cdac0fe17c9/leaf_c4554de9-d378-473c-ab26-71f7b64d5b65_1' [4.00 MiB] inherit
(Note the inactive volume is I think normal, I see these on working hosts and also I can activate it, it switches back to inactive after a short period).
I assume there is some issue with the LVM which is stopping it being plugged into XCP but cannot find any obvious errors or repairs. Looking through the storage logs there is one error reported for one of the volumes as follows (from /var/log/SMlog, first lines included for context and any info they provide - error "failed to get physical size: -22" on line 33 onwards):
Jun 9 15:49:06 hal02 SM: [28507] ['/usr/bin/vhd-util', 'scan', '-f', '-m', 'VHD-*', '-l', 'VG_XenStorage-e0b493c8-041d-345e-84d6-1cdac0fe17c9']
Jun 9 15:49:06 hal02 SM: [28507] pread SUCCESS
Jun 9 15:49:06 hal02 SMGC: [28507] SR e0b4 ('Local storage') (11 VDIs in 4 VHD trees):
Jun 9 15:49:06 hal02 SMGC: [28507] 87343508[VHD](200.000G//200.398G|a)
Jun 9 15:49:06 hal02 SMGC: [28507] 908b187e[VHD](200.000G//200.398G|a)
Jun 9 15:49:06 hal02 SMGC: [28507] 0e9f8505[VHD](64.000G//64.133G|a)
Jun 9 15:49:06 hal02 SMGC: [28507] *71a27718[VHD](250.000G//24.391G|a)
Jun 9 15:49:06 hal02 SMGC: [28507] *e764c429[VHD](250.000G//540.000M|a)
Jun 9 15:49:06 hal02 SMGC: [28507] *7b5feb84[VHD](250.000G//812.000M|a)
Jun 9 15:49:06 hal02 SMGC: [28507] 01a2a59f[VHD](250.000G//8.000M|a)
Jun 9 15:49:06 hal02 SMGC: [28507] ad7c8561[VHD](250.000G//236.000M|a)
Jun 9 15:49:06 hal02 SMGC: [28507] c4554de9[VHD](250.000G//100.000M|a)
Jun 9 15:49:06 hal02 SMGC: [28507] b784f499[VHD](250.000G//8.000M|a)
Jun 9 15:49:06 hal02 SMGC: [28507] 0efcf947[VHD](250.000G//8.000M|a)
Jun 9 15:49:06 hal02 SMGC: [28507]
Jun 9 15:49:06 hal02 SM: [28507] lock: acquired /var/lock/sm/.nil/lvm
Jun 9 15:49:06 hal02 SM: [28507] ['/sbin/lvchange', '-ay', '/dev/VG_XenStorage-e0b493c8-041d-345e-84d6-1cdac0fe17c9/leaf_c4554de9-d378-473c-ab26-71f7b64d5b65_1']
Jun 9 15:49:06 hal02 SM: [28507] pread SUCCESS
Jun 9 15:49:06 hal02 SM: [28507] lock: released /var/lock/sm/.nil/lvm
Jun 9 15:49:06 hal02 SM: [28507] lock: acquired /var/lock/sm/.nil/lvm
Jun 9 15:49:06 hal02 SM: [28507] ['/sbin/lvchange', '-an', '/dev/VG_XenStorage-e0b493c8-041d-345e-84d6-1cdac0fe17c9/leaf_c4554de9-d378-473c-ab26-71f7b64d5b65_1']
Jun 9 15:49:07 hal02 SM: [28507] pread SUCCESS
Jun 9 15:49:07 hal02 SM: [28507] lock: released /var/lock/sm/.nil/lvm
Jun 9 15:49:07 hal02 SM: [28507] ['/sbin/dmsetup', 'status', 'VG_XenStorage--e0b493c8--041d--345e--84d6--1cdac0fe17c9-leaf_c4554de9--d378--473c--ab26--71f7b64d5b65_1']
Jun 9 15:49:07 hal02 SM: [28507] pread SUCCESS
Jun 9 15:49:07 hal02 SMGC: [28507] *** UNDO LEAF-COALESCE
Jun 9 15:49:07 hal02 SM: [28507] lock: opening lock file /var/lock/sm/lvm-e0b493c8-041d-345e-84d6-1cdac0fe17c9/ad7c8561-1044-4eb5-9acd-00c548280410
Jun 9 15:49:07 hal02 SM: [28507] lock: acquired /var/lock/sm/lvm-e0b493c8-041d-345e-84d6-1cdac0fe17c9/ad7c8561-1044-4eb5-9acd-00c548280410
Jun 9 15:49:07 hal02 SM: [28507] Refcount for lvm-e0b493c8-041d-345e-84d6-1cdac0fe17c9:ad7c8561-1044-4eb5-9acd-00c548280410 (9, 0) + (1, 0) => (10, 0)
Jun 9 15:49:07 hal02 SM: [28507] Refcount for lvm-e0b493c8-041d-345e-84d6-1cdac0fe17c9:ad7c8561-1044-4eb5-9acd-00c548280410 set => (10, 0b)
Jun 9 15:49:07 hal02 SM: [28507] lock: released /var/lock/sm/lvm-e0b493c8-041d-345e-84d6-1cdac0fe17c9/ad7c8561-1044-4eb5-9acd-00c548280410
Jun 9 15:49:07 hal02 SM: [28507] ['/usr/bin/vhd-util', 'query', '--debug', '-s', '-n', '/dev/VG_XenStorage-e0b493c8-041d-345e-84d6-1cdac0fe17c9/VHD-ad7c8561-1044-4eb5-9acd-00c548280410']
Jun 9 15:49:07 hal02 SM: [28507] FAILED in util.pread: (rc 22) stdout: 'failed to get physical size: -22
Jun 9 15:49:07 hal02 SM: [28507] ', stderr: ''
Jun 9 15:49:07 hal02 SM: [28507] lock: released /var/lock/sm/e0b493c8-041d-345e-84d6-1cdac0fe17c9/sr
Jun 9 15:49:07 hal02 SM: [28507] ***** Local VHD on LVM: EXCEPTION <class 'util.CommandException'>, Invalid argument
Jun 9 15:49:07 hal02 SM: [28507] File "/opt/xensource/sm/SRCommand.py", line 376, in run
Jun 9 15:49:07 hal02 SM: [28507] sr = driver(cmd, cmd.sr_uuid)
Jun 9 15:49:07 hal02 SM: [28507] File "/opt/xensource/sm/SR.py", line 156, in __init__
Jun 9 15:49:07 hal02 SM: [28507] self.load(sr_uuid)
Jun 9 15:49:07 hal02 SM: [28507] File "/opt/xensource/sm/LVMSR", line 197, in load
Jun 9 15:49:07 hal02 SM: [28507] self._undoAllJournals()
Jun 9 15:49:07 hal02 SM: [28507] File "/opt/xensource/sm/LVMSR", line 1158, in _undoAllJournals
Jun 9 15:49:07 hal02 SM: [28507] self._handleInterruptedCoalesceLeaf()
Jun 9 15:49:07 hal02 SM: [28507] File "/opt/xensource/sm/LVMSR", line 914, in _handleInterruptedCoalesceLeaf
Jun 9 15:49:07 hal02 SM: [28507] cleanup.gc_force(self.session, self.uuid)
Jun 9 15:49:07 hal02 SM: [28507] File "/opt/xensource/sm/cleanup.py", line 3412, in gc_force
Jun 9 15:49:07 hal02 SM: [28507] sr.scanLocked(force)
Jun 9 15:49:07 hal02 SM: [28507] File "/opt/xensource/sm/cleanup.py", line 1606, in scanLocked
Jun 9 15:49:07 hal02 SM: [28507] self.scan(force)
Jun 9 15:49:07 hal02 SM: [28507] File "/opt/xensource/sm/cleanup.py", line 2627, in scan
Jun 9 15:49:07 hal02 SM: [28507] self._handleInterruptedCoalesceLeaf()
Jun 9 15:49:07 hal02 SM: [28507] File "/opt/xensource/sm/cleanup.py", line 2705, in _handleInterruptedCoalesceLeaf
Jun 9 15:49:07 hal02 SM: [28507] self._undoInterruptedCoalesceLeaf(uuid, parentUuid)
Jun 9 15:49:07 hal02 SM: [28507] File "/opt/xensource/sm/cleanup.py", line 2748, in _undoInterruptedCoalesceLeaf
Jun 9 15:49:07 hal02 SM: [28507] parent.deflate()
Jun 9 15:49:07 hal02 SM: [28507] File "/opt/xensource/sm/cleanup.py", line 1144, in deflate
Jun 9 15:49:07 hal02 SM: [28507] lvhdutil.deflate(self.sr.lvmCache, self.fileName, self.getSizeVHD())
Jun 9 15:49:07 hal02 SM: [28507] File "/opt/xensource/sm/cleanup.py", line 1201, in getSizeVHD
Jun 9 15:49:07 hal02 SM: [28507] self._loadInfoSizeVHD()
Jun 9 15:49:07 hal02 SM: [28507] File "/opt/xensource/sm/cleanup.py", line 1212, in _loadInfoSizeVHD
Jun 9 15:49:07 hal02 SM: [28507] self._sizeVHD = vhdutil.getSizePhys(self.path)
Jun 9 15:49:07 hal02 SM: [28507] File "/opt/xensource/sm/vhdutil.py", line 253, in getSizePhys
Jun 9 15:49:07 hal02 SM: [28507] ret = ioretry(cmd)
Jun 9 15:49:07 hal02 SM: [28507] File "/opt/xensource/sm/vhdutil.py", line 102, in ioretry
Jun 9 15:49:07 hal02 SM: [28507] errlist = [errno.EIO, errno.EAGAIN])
Jun 9 15:49:07 hal02 SM: [28507] File "/opt/xensource/sm/util.py", line 330, in ioretry
Jun 9 15:49:07 hal02 SM: [28507] return f()
Jun 9 15:49:07 hal02 SM: [28507] File "/opt/xensource/sm/vhdutil.py", line 101, in <lambda>
Jun 9 15:49:07 hal02 SM: [28507] return util.ioretry(lambda: util.pread2(cmd),
Jun 9 15:49:07 hal02 SM: [28507] File "/opt/xensource/sm/util.py", line 227, in pread2
Jun 9 15:49:07 hal02 SM: [28507] return pread(cmdlist, quiet = quiet)
Jun 9 15:49:07 hal02 SM: [28507] File "/opt/xensource/sm/util.py", line 190, in pread
Jun 9 15:49:07 hal02 SM: [28507] raise CommandException(rc, str(cmdlist), stderr.strip())
Jun 9 15:49:07 hal02 SM: [28507]
