Join Log in
Score:3
Pierre avatar Server

How to repair zsysctl after manually deleting snapshots with `zfs destroy`?

pm flag
Pierre

I am running Linux Mint 20.3 with ZFS (including for /). Up until very recently, whenever I installed or removed packages, it would also create a snapshot so that I could select it when booting to revert a change that might have broken the system.

But it seems there is a limitation of this system, where it can't cope with a large (1200+) number of snapshots. I can't find the issue anymore, but it suggested running for dataset in $(zfs list -t snapshot -o name | grep @autozsys); do sudo zfs destroy "$dataset"; done to delete most snapshots.

Since I didn't need these older snapshots, I gave it a go. I couldn't delete some because they were clones of others and not knowing whether this would make me lose live data, I left it at that.

I later learned about the zsysctl command to manage snapshots... And it seems I broke my system by not using that command but zfs destroy directly.

Now, whenever I apt install or apt remove, I'm getting this error (after a 30s pause): ERROR Service took too long to respond. Disconnecting client.

I think zsysctl is partially broken because running zsysctl show results in the last entry in the lists for bpool and rpool having a date of 0001-01-01 00:00:00 which is obviously wrong:

$ zsysctl show
Name:           rpool/ROOT/ubuntu_uyv8gh
ZSys:           true
Last Used:      current
History:        
  - Name:       rpool/ROOT/ubuntu_uyv8gh@autozsys_nma9lm
    Created on: 2022-11-17 17:43:38
  - Name:       rpool/ROOT/ubuntu_uyv8gh@autozsys_1ayzvs
    Created on: 2022-11-14 13:22:37
  - Name:       rpool/ROOT/ubuntu_uyv8gh@autozsys_yv8nrd
    Created on: 2022-11-11 17:13:10
  - Name:       rpool/ROOT/ubuntu_uyv8gh@autozsys_yu0btn
    Created on: 2022-02-15 15:58:44
  - Name:       rpool/ROOT/ubuntu_6xf31g
    Created on: 2021-10-22 17:13:50
Users:
  - Name:    pierre
    History: 
     - rpool/USERDATA/pierre_gg46xm@autozsys_nma9lm (2022-11-17 17:43:40)
     - rpool/USERDATA/pierre_gg46xm@autozsys_m6wubi (2022-11-17 14:29:58)
     - rpool/USERDATA/pierre_gg46xm@autozsys_17okng (2022-11-17 11:29:23)
     - rpool/USERDATA/pierre_gg46xm@autozsys_dgvgfz (2022-11-17 10:29:19)
     - rpool/USERDATA/pierre_gg46xm@autozsys_5mk4hd (2022-11-16 15:55:03)
     - rpool/USERDATA/pierre_gg46xm@autozsys_z90d2e (2022-11-16 14:54:03)
     - rpool/USERDATA/pierre_gg46xm@autozsys_pfutl5 (2022-11-16 13:54:03)
     - rpool/USERDATA/pierre_gg46xm@autozsys_5dfwde (2022-11-16 12:42:58)
     - rpool/USERDATA/pierre_gg46xm@autozsys_iuusno (2022-11-16 11:42:54)
     - rpool/USERDATA/pierre_gg46xm@autozsys_zzrd7w (2022-11-16 10:41:57)
     - rpool/USERDATA/pierre_gg46xm@autozsys_d4kkrw (2022-11-16 09:41:55)
     - rpool/USERDATA/pierre_gg46xm@autozsys_2ws9fj (2022-11-15 16:06:23)
     - rpool/USERDATA/pierre_gg46xm@autozsys_cfb7iu (2022-11-15 15:06:23)
     - rpool/USERDATA/pierre_gg46xm@autozsys_3rcqfi (2022-11-15 13:35:17)
     - rpool/USERDATA/pierre_gg46xm@autozsys_56mb7s (2022-11-15 12:07:32)
     - rpool/USERDATA/pierre_gg46xm@autozsys_9b068o (2022-11-15 11:06:38)
     - rpool/USERDATA/pierre_gg46xm@autozsys_o610qz (2022-11-15 10:06:35)
     - rpool/USERDATA/pierre_gg46xm@autozsys_fw4jbj (2022-11-15 08:38:21)
     - rpool/USERDATA/pierre_gg46xm@autozsys_j9ow55 (2022-11-14 17:00:43)
     - rpool/USERDATA/pierre_gg46xm@autozsys_mfc8hk (2022-11-14 15:59:43)
     - rpool/USERDATA/pierre_gg46xm@autozsys_1ayzvs (2022-11-14 13:22:39)
     - rpool/USERDATA/pierre_gg46xm@autozsys_y6xypr (2022-11-14 12:57:47)
     - rpool/USERDATA/pierre_gg46xm@autozsys_o0aj49 (2022-11-14 11:57:46)
     - rpool/USERDATA/pierre_gg46xm@autozsys_iljt6z (2022-11-14 10:57:43)
     - rpool/USERDATA/pierre_gg46xm@autozsys_2fmmg5 (2022-11-14 09:56:43)
     - rpool/USERDATA/pierre_gg46xm@autozsys_lxkohx (2022-11-14 08:55:51)
     - rpool/USERDATA/pierre_gg46xm@autozsys_czv8zx (2022-11-13 16:58:50)
     - rpool/USERDATA/pierre_gg46xm@autozsys_8kl96y (2022-11-13 15:57:50)
     - rpool/USERDATA/pierre_gg46xm@autozsys_5ks5h8 (2022-11-13 14:56:50)
     - rpool/USERDATA/pierre_gg46xm@autozsys_5m7wwk (2022-11-13 13:56:50)
     - rpool/USERDATA/pierre_gg46xm@autozsys_40udn4 (2022-11-13 12:28:40)
     - rpool/USERDATA/pierre_gg46xm@autozsys_uagprp (2022-11-13 11:28:39)
     - rpool/USERDATA/pierre_gg46xm@autozsys_4vbncy (2022-11-13 10:28:39)
     - rpool/USERDATA/pierre_gg46xm@autozsys_9buhaa (2022-11-13 09:26:59)
     - rpool/USERDATA/pierre_gg46xm@autozsys_92de9m (2022-11-12 14:48:45)
     - rpool/USERDATA/pierre_gg46xm@autozsys_x5vgcm (2022-11-12 12:00:12)
     - rpool/USERDATA/pierre_gg46xm@autozsys_pm05qo (2022-11-12 10:43:35)
     - rpool/USERDATA/pierre_gg46xm@autozsys_kxd0z1 (2022-11-12 09:43:32)
     - rpool/USERDATA/pierre_gg46xm@autozsys_welapl (2022-11-11 17:18:27)
     - rpool/USERDATA/pierre_gg46xm@autozsys_yv8nrd (2022-11-11 17:13:11)
     - rpool/USERDATA/pierre_gg46xm-rpool.ROOT.ubuntu-6xf31g (2022-11-09 07:17:16)
     - rpool/USERDATA/pierre_gg46xm@autozsys_yu0btn (2022-02-15 15:58:45)
     - rpool/USERDATA/pierre_hntkn5 (0001-01-01 00:00:00)
  - Name:    root
    History: 
     - rpool/USERDATA/root_gg46xm@autozsys_nma9lm (2022-11-17 17:43:40)
     - rpool/USERDATA/root_gg46xm@autozsys_1ayzvs (2022-11-14 13:22:39)
     - rpool/USERDATA/root_gg46xm@autozsys_yv8nrd (2022-11-11 17:13:11)
     - rpool/USERDATA/root_gg46xm-rpool.ROOT.ubuntu-6xf31g (2022-11-09 07:17:16)
     - rpool/USERDATA/root_gg46xm@autozsys_yu0btn (2022-02-15 15:58:45)
     - rpool/USERDATA/root_hntkn5 (0001-01-01 00:00:00)

Running zsysctl save -vv doesn't give much more information:

$ zsysctl save -vv
DEBUG /zsys.Zsys/SaveUserState() call logged as [e560ff96:3c3c6ace] 
DEBUG Check if grpc request peer is authorized     
DEBUG Polkit call result, authorized: true         
DEBUG Didn't receive any information from service in 30s 
ERROR Service took too long to respond. Disconnecting client.

How do I recover from this partially broken state?

359
1 + 0
zfs
Score:1
ewwhite avatar Server
ng flag
ewwhite

You can remove multiple snapshots by using %.

Example:

zfs destroy rpool/USERDATA/pierre_gg46xm@autozsys_nma9lm%autozsys_yv8nrd

This removes everything from autozsys_nma9lm through autozsys_yv8nrd.

If you have clones intermixed amongst the 1200+ snapshots, then it may be worth reevaluating the setup or separating your data into data-specific pools.

I see you've used the zsysctl. I'm not familiar with it, but ZFS is already streamlined with two main commands; zfs and zpool.

There doesn't seem to be much utility to using zsysctl here because it's definitely complicated your setup. It may be an unnecessary abstraction and doesn't have a future on the platform. Consider removing it.

zsys being removed from ubuntu #235

zsys cannot handle large amounts of snapshots #224

(I'd say the same for ZFS root and the high number of snapshots)

+ 3
Pierre avatar
pm flag
Pierre
Thanks. I have two issues with this: 1. I have clones, and I don't know how to make sure I'm not deleting live data by deleting the clones; 2. I don't know if I can just yank zsys like that and what to replace it with as having the snapshots in grub is nice. zfsbootmenu is mentioned, but I'm wary of relying on my making it work with the system rather than having an "out of the box" solution.
0
Reply
ewwhite avatar
ng flag
ewwhite
Is this a server or professional environment?
0
Reply
Pierre avatar
pm flag
Pierre
No, it’s my workstation.
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.