Join Log in
Score:1
小太郎 avatar Server

pvmove: "Insufficient free space" when trying to move mirrored LV

ng flag
小太郎

Originally I had two PVs /dev/nvme0n1p2 and /dev/nvme1n1p1 containing various mirrored LVs.
I recently added two new larger capacity PVs /dev/nvme2n1p1 and /dev/nvme3n1p1, and expanded a LV (home/windows-persistent) to make use of the new space.

However, I noticed the LV is now split across all 4 PVs, whilst I prefer it to be on just 2 PVs, so I attempted to move the LV to just the new PVs with pvmove:

lvm> pvmove /dev/nvme0n1p2 -n home/windows-persistent /dev/nvme2n1p1
  Insufficient free space: 57793 extents needed, but only 0 available
  Unable to allocate mirror extents for home/pvmove0.
  Failed to convert pvmove LV to mirrored.
lvm> pvmove /dev/nvme1n1p1 -n home/windows-persistent /dev/nvme3n1p1
  Insufficient free space: 57793 extents needed, but only 0 available
  Unable to allocate mirror extents for home/pvmove0.
  Failed to convert pvmove LV to mirrored.

How come LVM is complaining of insufficient free space? How can I move the LVs correctly?

pvdisplay/vgdisplay/lvdisplay output:

lvm> pvdisplay -m
  --- Physical volume ---
  PV Name               /dev/nvme0n1p2
  VG Name               home
  PV Size               476.74 GiB / not usable 0   
  Allocatable           yes 
  PE Size               4.00 MiB
  Total PE              122046
  Free PE               2812
  Allocated PE          119234
  PV UUID               n6vD1c-pF3t-mqlg-g4Ge-HPT7-FwOC-p7t6wK
   
  --- Physical Segments ---
  Physical extent 0 to 51199:
    Logical volume  /dev/home/dom0
    Logical extents 0 to 51199
  Physical extent 51200 to 51200:
    Logical volume  /dev/home/windows-persistent_rmeta_0
    Logical extents 0 to 0
  Physical extent 51201 to 89600:
    Logical volume  /dev/home/windows-persistent_rimage_0
    Logical extents 0 to 38399
  Physical extent 89601 to 89601:
    Logical volume  /dev/home/windows_rmeta_0
    Logical extents 0 to 0
  Physical extent 89602 to 99841:
    Logical volume  /dev/home/windows_rimage_0
    Logical extents 0 to 10239
  Physical extent 99842 to 119233:
    Logical volume  /dev/home/windows-persistent_rimage_0
    Logical extents 38400 to 57791
  Physical extent 119234 to 122045:
    FREE
   
  --- Physical volume ---
  PV Name               /dev/nvme1n1p1
  VG Name               home
  PV Size               465.76 GiB / not usable 3.00 MiB
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              119234
  Free PE               0
  Allocated PE          119234
  PV UUID               1dmaeN-j0D3-EU60-kQg2-a5t7-CNdP-qpBYyp
   
  --- Physical Segments ---
  Physical extent 0 to 51199:
    Logical volume  /dev/home/dom0-mirror
    Logical extents 0 to 51199
  Physical extent 51200 to 51200:
    Logical volume  /dev/home/windows-persistent_rmeta_1
    Logical extents 0 to 0
  Physical extent 51201 to 81920:
    Logical volume  /dev/home/windows-persistent_rimage_1
    Logical extents 0 to 30719
  Physical extent 81921 to 92161:
    Logical volume  /dev/home/windows-persistent_rimage_1
    Logical extents 38400 to 48640
  Physical extent 92162 to 99841:
    Logical volume  /dev/home/windows-persistent_rimage_1
    Logical extents 30720 to 38399
  Physical extent 99842 to 99842:
    Logical volume  /dev/home/windows_rmeta_1
    Logical extents 0 to 0
  Physical extent 99843 to 110082:
    Logical volume  /dev/home/windows_rimage_1
    Logical extents 0 to 10239
  Physical extent 110083 to 119233:
    Logical volume  /dev/home/windows-persistent_rimage_1
    Logical extents 48641 to 57791
   
  --- Physical volume ---
  PV Name               /dev/nvme2n1p1
  VG Name               home
  PV Size               <1.82 TiB / not usable 4.00 MiB
  Allocatable           yes 
  PE Size               4.00 MiB
  Total PE              476931
  Free PE               457923
  Allocated PE          19008
  PV UUID               UzD83f-tnya-9wbR-TRrf-qkxe-2lf7-ewF3M7
   
  --- Physical Segments ---
  Physical extent 0 to 19007:
    Logical volume  /dev/home/windows-persistent_rimage_0
    Logical extents 57792 to 76799
  Physical extent 19008 to 476930:
    FREE
   
  --- Physical volume ---
  PV Name               /dev/nvme3n1p1
  VG Name               home
  PV Size               <1.82 TiB / not usable 4.00 MiB
  Allocatable           yes 
  PE Size               4.00 MiB
  Total PE              476931
  Free PE               457923
  Allocated PE          19008
  PV UUID               Rf7CoD-NxTs-ohO3-pqJx-dMZ8-6SCh-SXXhAf
   
  --- Physical Segments ---
  Physical extent 0 to 19007:
    Logical volume  /dev/home/windows-persistent_rimage_1
    Logical extents 57792 to 76799
  Physical extent 19008 to 476930:
    FREE

lvm> vgdisplay
  --- Volume group ---
  VG Name               home
  System ID             
  Format                lvm2
  Metadata Areas        4
  Metadata Sequence No  209
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                4
  Open LV               2
  Max PV                0
  Cur PV                4
  Act PV                4
  VG Size               <4.56 TiB
  PE Size               4.00 MiB
  Total PE              1195142
  Alloc PE / Size       276484 / 1.05 TiB
  Free  PE / Size       918658 / 3.50 TiB
  VG UUID               1QhLGQ-ljr2-p0or-yAiD-530e-17n4-y8VN0d
   
lvm> lvdisplay
  --- Logical volume ---
  LV Path                /dev/home/dom0
  LV Name                dom0
  VG Name                home
  LV UUID                XKpxWe-g8Rh-faKQ-s8Ht-KPC0-oLHc-tLneqN
  LV Write Access        read/write
  LV Creation host, time home, 2016-07-02 18:55:56 +1000
  LV Status              available
  # open                 1
  LV Size                200.00 GiB
  Current LE             51200
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:0
   
  --- Logical volume ---
  LV Path                /dev/home/windows-persistent
  LV Name                windows-persistent
  VG Name                home
  LV UUID                4g6nXt-h6LL-YaHc-wfcU-Iote-rFJd-rpzgeF
  LV Write Access        read/write
  LV Creation host, time home, 2018-01-08 22:18:42 +1100
  LV Status              available
  # open                 0
  LV Size                300.00 GiB
  Current LE             76800
  Mirrored volumes       2
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:5
   
  --- Logical volume ---
  LV Path                /dev/home/dom0-mirror
  LV Name                dom0-mirror
  VG Name                home
  LV UUID                WtfyuY-LRxR-nQBk-ft8C-T1gn-2hvE-TIQLgm
  LV Write Access        read/write
  LV Creation host, time home, 2018-12-06 19:02:41 +1100
  LV Status              available
  # open                 1
  LV Size                200.00 GiB
  Current LE             51200
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:6
   
  --- Logical volume ---
  LV Path                /dev/home/windows
  LV Name                windows
  VG Name                home
  LV UUID                JKQjjn-4vUO-bwTD-eCl3-eZiM-mato-1R09It
  LV Write Access        read/write
  LV Creation host, time home, 2023-07-06 17:10:07 +1000
  LV Status              available
  # open                 0
  LV Size                40.00 GiB
  Current LE             10240
  Mirrored volumes       2
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:11

pvs/vgs/lvs output:

lvm> pvs
  PV             VG   Fmt  Attr PSize    PFree 
  /dev/nvme0n1p2 home lvm2 a--   476.74g 10.98g
  /dev/nvme1n1p1 home lvm2 a--  <465.76g     0 
  /dev/nvme2n1p1 home lvm2 a--    <1.82t <1.75t
  /dev/nvme3n1p1 home lvm2 a--    <1.82t <1.75t
lvm> vgs
  VG   #PV #LV #SN Attr   VSize  VFree
  home   4   4   0 wz--n- <4.56t 3.50t
lvm> lvs
  LV                 VG   Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  dom0               home -wi-ao---- 200.00g                                                    
  dom0-mirror        home -wi-ao---- 200.00g                                                    
  windows            home rwi-aor---  40.00g                                    100.00          
  windows-persistent home rwi-aor--- 300.00g                                    100.00
39
1 + 4
lvm
Romeo Ninov avatar
in flag
Romeo Ninov
Have you try to change the order in `pvmove` command: `pvmove -n home/windows-persistent /dev/nvme0n1p2 /dev/nvme2n1p1`
0
Reply
小太郎 avatar
ng flag
小太郎
Yes, it produces the same error
0
Reply
Romeo Ninov avatar
in flag
Romeo Ninov
What is the **current** result of command: `pvs`?
0
Reply
小太郎 avatar
ng flag
小太郎
Added to the post
1
Reply
Score:0
小太郎 avatar Server
ng flag
小太郎

Due to how pvmove tries to prevent reducing redundancy from moving two mirrors to the same PV (e.g., one PV should not contain both _rimage_0 and _rimage_1 sub LVs), the only* way to combine/colocate different parts of the same mirror is to move each sub LV one-by-one, and only by name.
Since the current error is quite cryptic, I've raised a PR to LVM to improve the messaging: https://github.com/lvmteam/lvm2/pull/126

* Maybe this can be bypassed with --alloc anywhere, but I have not tried

In addition to that, pvmove ended up having a bug in the very "allow parts of the same mirror to colocate" code that prevented it from working correctly. In the same PR above, I've fixed the bug

Ultimately with the above fixes, I ended up running the following commands to successfully move my LV:

pvmove -n home/windows-persistent_rmeta_0 /dev/nvme0n1p2 /dev/nvme2n1p1
pvmove -n home/windows-persistent_rimage_0 /dev/nvme0n1p2 /dev/nvme2n1p1
pvmove -n home/windows-persistent_rmeta_1 /dev/nvme1n1p1 /dev/nvme3n1p1
pvmove -n home/windows-persistent_rimage_1 /dev/nvme1n1p1 /dev/nvme3n1p1
+ 0
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.