Join Log in
Score:1
aesnak avatar Server

PCS Cluster - Binding nfsserver resource to specific IP

it flag
aesnak

I'm trying to deploy a HA NFS Cluster Architecture with PCS on Rocky Linux 8.5 . Current kernel and nfs related package versions, pcs configurations are shown with detail below.

I'm not able to declare specific IP for NFSD instances (rpc.statd, rpc.mountd etc) to bind. Whatever i do, services keeps binding 0.0.0.0:$default-ports.

I would like to initiate different "ocf:Hearthbeat:nfsserver" with specific VirtualIP resource per each of my NFS(block) resource group. When i declare second NFS share resource group on the same node of cluster (where i plan to have more NFS than cluster size), "ocf:Hearthbeat:nfsserver" resources blocks each other, and one evantually wins and the other resource is gets in to the "blocked" state.

[root@node1 ~]# uname -a
Linux node1.local 4.18.0-348.12.2.el8_5.x86_64 #1 SMP Wed Jan 19 17:53:40 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
[root@node1 ~]# rpm -qa nfs* rpc*
nfs-utils-2.3.3-46.el8.x86_64
rpcbind-1.2.5-8.el8.x86_64
[root@node1 ~]#

PCS Cluster Status

[root@node1 ~]#  pcs status
Cluster name: cluster01
Cluster Summary:
  * Stack: corosync
  * Current DC: node5 (version 2.1.0-8.el8-7c3f660707) - partition with quorum
  * Last updated: Thu Mar 24 13:10:09 2022
  * Last change:  Thu Mar 24 13:03:48 2022 by root via crm_resource on node3
  * 5 nodes configured
  * 5 resource instances configured

Node List:
  * Online: [ node1 node2 node3 node4 node5 ]

Full List of Resources:
  * Resource Group: Group_SHARE:
    * ROOT-FS_SHARE (ocf::heartbeat:Filesystem):     Started node2
    * NFSD_SHARE    (ocf::heartbeat:nfsserver):  Started node2
    * NFS_SHARE (ocf::heartbeat:exportfs):   Started node2
    * NFS-IP_SHARE  (ocf::heartbeat:IPaddr2):    Started node2
    * NFS-NOTIFY_SHARE  (ocf::heartbeat:nfsnotify):  Started node2

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled
[root@node1 ~]#

PCS Resource Config output

[root@node2 ~]# pcs resource config
 Group: Group_SHARE
  Resource: ROOT-FS_SHARE (class=ocf provider=heartbeat type=Filesystem)
   Attributes: device=/dev/disk/by-id/wwn-0x6001405ce6b7033688d497a91aa23547 directory=/srv/block/SHARE fstype=xfs
   Operations: monitor interval=20s timeout=40s (ROOT-FS_SHARE-monitor-interval-20s)
               start interval=0s timeout=60s (ROOT-FS_SHARE-start-interval-0s)
               stop interval=0s timeout=60s (ROOT-FS_SHARE-stop-interval-0s)
  Resource: NFSD_SHARE (class=ocf provider=heartbeat type=nfsserver)
   Attributes: nfs_ip=10.1.31.100 nfs_no_notify=true nfs_shared_infodir=/srv/block/SHARE/nfsinfo/
   Operations: monitor interval=10s timeout=20s (NFSD_SHARE-monitor-interval-10s)
               start interval=0s timeout=40s (NFSD_SHARE-start-interval-0s)
               stop interval=0s timeout=20s (NFSD_SHARE-stop-interval-0s)
  Resource: NFS_SHARE (class=ocf provider=heartbeat type=exportfs)
   Attributes: clientspec=10.1.31.0/255.255.255.0 directory=/srv/block/SHARE/SHARE fsid=0 options=rw,sync,no_root_squash
   Operations: monitor interval=10s timeout=20s (NFS_SHARE-monitor-interval-10s)
               start interval=0s timeout=40s (NFS_SHARE-start-interval-0s)
               stop interval=0s timeout=120s (NFS_SHARE-stop-interval-0s)
  Resource: NFS-IP_SHARE (class=ocf provider=heartbeat type=IPaddr2)
   Attributes: cidr_netmask=24 ip=10.1.31.100 nic=team31
   Operations: monitor interval=30s (NFS-IP_SHARE-monitor-interval-30s)
               start interval=0s timeout=20s (NFS-IP_SHARE-start-interval-0s)
               stop interval=0s timeout=20s (NFS-IP_SHARE-stop-interval-0s)
  Resource: NFS-NOTIFY_SHARE (class=ocf provider=heartbeat type=nfsnotify)
   Attributes: source_host=SHARE.local
   Operations: monitor interval=30s timeout=90s (NFS-NOTIFY_SHARE-monitor-interval-30s)
               reload interval=0s timeout=90s (NFS-NOTIFY_SHARE-reload-interval-0s)
               start interval=0s timeout=90s (NFS-NOTIFY_SHARE-start-interval-0s)
               stop interval=0s timeout=90s (NFS-NOTIFY_SHARE-stop-interval-0s)
[root@node2 ~]#

Virtual IP binded successfully on node2

[root@node2 ~]# ip -4 addr show team31
6: team31: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    inet 10.1.31.2/24 brd 10.1.31.255 scope global noprefixroute team31
       valid_lft forever preferred_lft forever
    inet 10.1.31.100/24 brd 10.1.31.255 scope global secondary team31
       valid_lft forever preferred_lft forever
[root@node2 ~]#

TCP LISTEN Binds

[root@node2 ~]# netstat -punta | grep LISTEN
tcp        0      0 0.0.0.0:2049            0.0.0.0:*               LISTEN      -                   
tcp        0      0 127.0.0.1:44321         0.0.0.0:*               LISTEN      1803/pmcd           
tcp        0      0 0.0.0.0:34661           0.0.0.0:*               LISTEN      630273/rpc.statd    
tcp        0      0 127.0.0.1:199           0.0.0.0:*               LISTEN      1257/snmpd            
tcp        0      0 127.0.0.1:4330          0.0.0.0:*               LISTEN      2834/pmlogger       
tcp        0      0 10.20.101.136:2379      0.0.0.0:*               LISTEN      3285/etcd           
tcp        0      0 10.20.101.136:2380      0.0.0.0:*               LISTEN      3285/etcd           
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      1/systemd           
tcp        0      0 0.0.0.0:20048           0.0.0.0:*               LISTEN      630282/rpc.mountd   
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      317707/nginx: maste 
tcp        0      0 0.0.0.0:2224            0.0.0.0:*               LISTEN      3725/platform-pytho 
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1170/sshd           
tcp        0      0 0.0.0.0:41017           0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      317707/nginx: maste 
tcp        0      0 0.0.0.0:35261           0.0.0.0:*               LISTEN      -                   
tcp6       0      0 :::2049                 :::*                    LISTEN      -                   
tcp6       0      0 ::1:44321               :::*                    LISTEN      1803/pmcd           
tcp6       0      0 ::1:4330                :::*                    LISTEN      2834/pmlogger       
tcp6       0      0 :::111                  :::*                    LISTEN      1/systemd           
tcp6       0      0 :::20048                :::*                    LISTEN      630282/rpc.mountd   
tcp6       0      0 :::2224                 :::*                    LISTEN      3725/platform-pytho 
tcp6       0      0 :::37329                :::*                    LISTEN      -                   
tcp6       0      0 :::22                   :::*                    LISTEN      1170/sshd           
tcp6       0      0 :::41179                :::*                    LISTEN      630273/rpc.statd    
tcp6       0      0 :::43487                :::*                    LISTEN      -                   
[root@node2 ~]#

PCS Cluster Resources (cib.xml format,in case you need deep dive)

<resources>
  <group id="Group_SHARE">
    <primitive class="ocf" id="ROOT-FS_SHARE" provider="heartbeat" type="Filesystem">
      <instance_attributes id="ROOT-FS_SHARE-instance_attributes">
        <nvpair id="ROOT-FS_SHARE-instance_attributes-device" name="device" value="/dev/disk/by-id/wwn-0x6001405ce6b7033688d497a91aa23547"/>
        <nvpair id="ROOT-FS_SHARE-instance_attributes-directory" name="directory" value="/srv/block/SHARE"/>
        <nvpair id="ROOT-FS_SHARE-instance_attributes-fstype" name="fstype" value="xfs"/>
      </instance_attributes>
      <operations>
        <op id="ROOT-FS_SHARE-monitor-interval-20s" interval="20s" name="monitor" timeout="40s"/>
        <op id="ROOT-FS_SHARE-start-interval-0s" interval="0s" name="start" timeout="60s"/>
        <op id="ROOT-FS_SHARE-stop-interval-0s" interval="0s" name="stop" timeout="60s"/>
      </operations>
    </primitive>
    <primitive class="ocf" id="NFSD_SHARE" provider="heartbeat" type="nfsserver">
      <instance_attributes id="NFSD_SHARE-instance_attributes">
        <nvpair id="NFSD_SHARE-instance_attributes-nfs_ip" name="nfs_ip" value="10.1.31.100"/>
        <nvpair id="NFSD_SHARE-instance_attributes-nfs_no_notify" name="nfs_no_notify" value="true"/>
        <nvpair id="NFSD_SHARE-instance_attributes-nfs_shared_infodir" name="nfs_shared_infodir" value="/srv/block/SHARE/nfsinfo/"/>
      </instance_attributes>
      <operations>
        <op id="NFSD_SHARE-monitor-interval-10s" interval="10s" name="monitor" timeout="20s"/>
        <op id="NFSD_SHARE-start-interval-0s" interval="0s" name="start" timeout="40s"/>
        <op id="NFSD_SHARE-stop-interval-0s" interval="0s" name="stop" timeout="20s"/>
      </operations>
    </primitive>
    <primitive class="ocf" id="NFS_SHARE" provider="heartbeat" type="exportfs">
      <instance_attributes id="NFS_SHARE-instance_attributes">
        <nvpair id="NFS_SHARE-instance_attributes-clientspec" name="clientspec" value="10.1.31.0/255.255.255.0"/>
        <nvpair id="NFS_SHARE-instance_attributes-directory" name="directory" value="/srv/block/SHARE/SHARE"/>
        <nvpair id="NFS_SHARE-instance_attributes-fsid" name="fsid" value="0"/>
        <nvpair id="NFS_SHARE-instance_attributes-options" name="options" value="rw,sync,no_root_squash"/>
      </instance_attributes>
      <operations>
        <op id="NFS_SHARE-monitor-interval-10s" interval="10s" name="monitor" timeout="20s"/>
        <op id="NFS_SHARE-start-interval-0s" interval="0s" name="start" timeout="40s"/>
        <op id="NFS_SHARE-stop-interval-0s" interval="0s" name="stop" timeout="120s"/>
      </operations>
    </primitive>
    <primitive class="ocf" id="NFS-IP_SHARE" provider="heartbeat" type="IPaddr2">
      <instance_attributes id="NFS-IP_SHARE-instance_attributes">
        <nvpair id="NFS-IP_SHARE-instance_attributes-cidr_netmask" name="cidr_netmask" value="24"/>
        <nvpair id="NFS-IP_SHARE-instance_attributes-ip" name="ip" value="10.1.31.100"/>
        <nvpair id="NFS-IP_SHARE-instance_attributes-nic" name="nic" value="team31"/>
      </instance_attributes>
      <operations>
        <op id="NFS-IP_SHARE-monitor-interval-30s" interval="30s" name="monitor"/>
        <op id="NFS-IP_SHARE-start-interval-0s" interval="0s" name="start" timeout="20s"/>
        <op id="NFS-IP_SHARE-stop-interval-0s" interval="0s" name="stop" timeout="20s"/>
      </operations>
    </primitive>
    <primitive class="ocf" id="NFS-NOTIFY_SHARE" provider="heartbeat" type="nfsnotify">
      <instance_attributes id="NFS-NOTIFY_SHARE-instance_attributes">
        <nvpair id="NFS-NOTIFY_SHARE-instance_attributes-source_host" name="source_host" value="SHARE.local"/>
      </instance_attributes>
      <operations>
        <op id="NFS-NOTIFY_SHARE-monitor-interval-30s" interval="30s" name="monitor" timeout="90s"/>
        <op id="NFS-NOTIFY_SHARE-reload-interval-0s" interval="0s" name="reload" timeout="90s"/>
        <op id="NFS-NOTIFY_SHARE-start-interval-0s" interval="0s" name="start" timeout="90s"/>
        <op id="NFS-NOTIFY_SHARE-stop-interval-0s" interval="0s" name="stop" timeout="90s"/>
      </operations>
    </primitive>
  </group>
</resources>

EDIT-1

It seems OpenClusterFramework - nfsserver resource doesn't use nfs_ip field for "rpc.nfsd -H $nfs_ip" at all. On RockyLinux 8.5 this resource also doesn't allow us to overwrite the default behaviour of supporting each NFS versions. RockyLinux 8.5 uses following packages resource-agents-4.1.1-98.el8_5.2.x86_64.

I will try to solve my issue by specifying custom pcs systemd resources for [email protected]

48
0 + 0
U. Windl avatar
it flag
U. Windl
Maybe show the log entry when the NFS server is started. Here (for SLES 12) it works!
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.