Join Log in
Score:0
cape avatar Server

Clustered vsftpd service stops working with no error log

de flag
cape

We've got a pacemaker cluster of vsftpd server made of two nodes in a active/pasive scheme. We have experienced the following behaviour.

  1. vsftpd.log stops recording activity while xferlog keeps recording (transfers keeps ocurring). vsftpd.log shows no error or warn message.
  2. Between 10 to 12 hours later service is unavailable and no new connections are happening
  3. We try to move resource but it fails a lot, we need to perform multiple clean-ups before service properly starts on the pasive node.

corsync config:

totem {
    version: 2
    secauth: off
    cluster_name: myCluster
    transport: udpu
}

nodelist {
    node {
        ring0_addr: node1
        nodeid: 1
    }

    node {
        ring0_addr: node2
        nodeid: 2
    }
}

quorum {
    provider: corosync_votequorum
    two_node: 1
}

logging {
    to_logfile: yes
    logfile: /var/log/cluster/corosync.log
    to_syslog: yes
}

vsftpd.conf

anonymous_enable=NO
local_enable=YES
write_enable=YES
local_umask=002
allow_ftpd_full_access
dirmessage_enable=YES
xferlog_enable=YES
connect_from_port_20=YES
xferlog_file=/var/log/xferlog

chroot_local_user=YES
chroot_list_enable=YES
chroot_list_file=/etc/vsftpd/chroot_list

listen=NO
listen_ipv6=YES

pam_service_name=vsftpd
userlist_enable=YES
tcp_wrappers=YES
allow_writeable_chroot=YES
dual_log_enable=YES
vsftpd_log_file=/var/log/vsftpd.log

The only significant log I've found it that produced by journalctl, which shows timeout while trying to stop server and a huge number of failure, that to be honest I don't how they happen.

Jun 09 17:50:02 node2.company.corp pengine[4720]:   notice: Stop    FTP_SERVICE        (node2.company.corp)
Jun 09 17:50:02 node2.company.corp pengine[4720]:   notice: Calculated transition 5460, saving inputs in /var/lib/pacemaker/pengine/pe-input-1567.bz2
Jun 09 17:50:02 node2.company.corp crmd[4721]:   notice: Initiating stop operation FTP_SERVICE_stop_0 locally on node2.company.corp | action 8
Jun 09 17:50:02 node2.company.corp systemd[1]: Reloading.
Jun 09 17:50:02 node2.company.corp systemd[1]: Stopping Vsftpd ftp daemon...
-- Subject: Unit vsftpd.service has begun shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit vsftpd.service has begun shutting down.
Jun 09 17:50:22 node2.company.corp lrmd[4718]:   notice: Giving up on FTP_SERVICE stop (rc=196): timeout (elapsed=19988ms, remaining=12ms)
Jun 09 17:50:22 node2.company.corp crmd[4721]:    error: Result of stop operation for FTP_SERVICE on node2.company.corp: Timed Out | call=255 key=FTP_SERVICE_stop_0 timeout=20000ms
Jun 09 17:50:22 node2.company.corp crmd[4721]:  warning: Action 8 (FTP_SERVICE_stop_0) on node2.company.corp failed (target: 0 vs. rc: 198): Error
Jun 09 17:50:22 node2.company.corp crmd[4721]:   notice: Transition aborted by operation FTP_SERVICE_stop_0 'modify' on node2.company.corp: Event failed | magic=2:198;8:5460:0:bf959758-83cf-4e4b-bd33-a360ffa08557 cib=0.449.1 source=match_graph_event:310 complete=false
Jun 09 17:50:22 node2.company.corp crmd[4721]:  warning: Action 8 (FTP_SERVICE_stop_0) on node2.company.corp failed (target: 0 vs. rc: 198): Error
Jun 09 17:50:22 node2.company.corp crmd[4721]:   notice: Transition 5460 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=1, Source=/var/lib/pacemaker/pengine/pe-input-1567.bz2): Complete
Jun 09 17:50:22 node2.company.corp pengine[4720]:   notice: On loss of CCM Quorum: Ignore
Jun 09 17:50:22 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:50:22 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:50:22 node2.company.corp pengine[4720]:   notice: Calculated transition 5461, saving inputs in /var/lib/pacemaker/pengine/pe-input-1568.bz2
Jun 09 17:50:22 node2.company.corp pengine[4720]:   notice: On loss of CCM Quorum: Ignore
Jun 09 17:50:22 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:50:22 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:50:22 node2.company.corp pengine[4720]:  warning: Forcing FTP_SERVICE away from node2.company.corp after 1000000 failures (max=1000000)
Jun 09 17:50:22 node2.company.corp pengine[4720]:   notice: Calculated transition 5462, saving inputs in /var/lib/pacemaker/pengine/pe-input-1569.bz2
Jun 09 17:50:22 node2.company.corp crmd[4721]:   notice: Transition 5462 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-1569.bz2): Complete
Jun 09 17:50:22 node2.company.corp crmd[4721]:   notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd
Jun 09 17:50:26 node2.company.corp crmd[4721]:   notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph
Jun 09 17:50:26 node2.company.corp pengine[4720]:   notice: On loss of CCM Quorum: Ignore
Jun 09 17:50:26 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:50:26 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:50:26 node2.company.corp pengine[4720]:  warning: Forcing FTP_SERVICE away from node2.company.corp after 1000000 failures (max=1000000)
Jun 09 17:50:26 node2.company.corp pengine[4720]:   notice: Calculated transition 5463, saving inputs in /var/lib/pacemaker/pengine/pe-input-1570.bz2
Jun 09 17:50:26 node2.company.corp crmd[4721]:   notice: Transition 5463 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-1570.bz2): Complete
Jun 09 17:50:26 node2.company.corp crmd[4721]:   notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd
Jun 09 17:50:33 node2.company.corp crmd[4721]:  warning: No reason to expect node 2 to be down
Jun 09 17:50:33 node2.company.corp crmd[4721]:   notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph
Jun 09 17:50:33 node2.company.corp crmd[4721]:  warning: No reason to expect node 2 to be down
Jun 09 17:50:33 node2.company.corp pengine[4720]:   notice: On loss of CCM Quorum: Ignore
Jun 09 17:50:33 node2.company.corp pengine[4720]:   notice: Start   FTP_SERVICE        (node2.company.corp)
Jun 09 17:50:33 node2.company.corp pengine[4720]:   notice: Calculated transition 5464, saving inputs in /var/lib/pacemaker/pengine/pe-input-1571.bz2
Jun 09 17:50:34 node2.company.corp pengine[4720]:   notice: On loss of CCM Quorum: Ignore
Jun 09 17:50:34 node2.company.corp pengine[4720]:   notice: Start   FTP_SERVICE        (node2.company.corp)
Jun 09 17:50:34 node2.company.corp pengine[4720]:   notice: Calculated transition 5465, saving inputs in /var/lib/pacemaker/pengine/pe-input-1571.bz2
Jun 09 17:50:34 node2.company.corp crmd[4721]:   notice: Initiating monitor operation FTP_SERVICE_monitor_0 locally on node2.company.corp | action 5
Jun 09 17:50:34 node2.company.corp crmd[4721]:   notice: Result of probe operation for FTP_SERVICE on node2.company.corp: 196 (OCF_PENDING) | call=260 key=FTP_SERVICE_monitor_0 confirmed=true cib-update=5999
Jun 09 17:50:34 node2.company.corp crmd[4721]:  warning: Action 5 (FTP_SERVICE_monitor_0) on node2.company.corp failed (target: 7 vs. rc: 196): Error
Jun 09 17:50:34 node2.company.corp crmd[4721]:   notice: Transition aborted by operation FTP_SERVICE_monitor_0 'create' on node2.company.corp: Event failed | magic=0:196;5:5465:7:bf959758-83cf-4e4b-bd33-a360ffa08557 cib=0.451.1 source=match_graph_event:310 complete=false
Jun 09 17:50:34 node2.company.corp crmd[4721]:  warning: Action 5 (FTP_SERVICE_monitor_0) on node2.company.corp failed (target: 7 vs. rc: 196): Error
Jun 09 17:50:34 node2.company.corp crmd[4721]:   notice: Transition 5465 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=2, Source=/var/lib/pacemaker/pengine/pe-input-1571.bz2): Complete
Jun 09 17:50:34 node2.company.corp pengine[4720]:   notice: On loss of CCM Quorum: Ignore
Jun 09 17:50:34 node2.company.corp pengine[4720]:  warning: Processing failed op monitor for FTP_SERVICE on node2.company.corp: OCF_PENDING (196)
Jun 09 17:50:34 node2.company.corp pengine[4720]:  warning: Processing failed op monitor for FTP_SERVICE on node2.company.corp: OCF_PENDING (196)
Jun 09 17:50:34 node2.company.corp pengine[4720]:   notice: Recover FTP_SERVICE        (Started node2.company.corp)
Jun 09 17:50:34 node2.company.corp pengine[4720]:   notice: Calculated transition 5466, saving inputs in /var/lib/pacemaker/pengine/pe-input-1572.bz2
Jun 09 17:50:34 node2.company.corp crmd[4721]:   notice: Initiating stop operation FTP_SERVICE_stop_0 locally on node2.company.corp | action 4
Jun 09 17:50:34 node2.company.corp systemd[1]: Reloading.
Jun 09 17:50:54 node2.company.corp lrmd[4718]:   notice: Giving up on FTP_SERVICE stop (rc=196): timeout (elapsed=19988ms, remaining=12ms)
Jun 09 17:50:54 node2.company.corp crmd[4721]:    error: Result of stop operation for FTP_SERVICE on node2.company.corp: Timed Out | call=261 key=FTP_SERVICE_stop_0 timeout=20000ms
Jun 09 17:50:54 node2.company.corp crmd[4721]:  warning: Action 4 (FTP_SERVICE_stop_0) on node2.company.corp failed (target: 0 vs. rc: 198): Error
Jun 09 17:50:54 node2.company.corp crmd[4721]:   notice: Transition aborted by operation FTP_SERVICE_stop_0 'modify' on node2.company.corp: Event failed | magic=2:198;4:5466:0:bf959758-83cf-4e4b-bd33-a360ffa08557 cib=0.451.2 source=match_graph_event:310 complete=false
Jun 09 17:50:54 node2.company.corp crmd[4721]:  warning: Action 4 (FTP_SERVICE_stop_0) on node2.company.corp failed (target: 0 vs. rc: 198): Error
Jun 09 17:50:54 node2.company.corp crmd[4721]:   notice: Transition 5466 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=3, Source=/var/lib/pacemaker/pengine/pe-input-1572.bz2): Complete
Jun 09 17:50:54 node2.company.corp pengine[4720]:   notice: On loss of CCM Quorum: Ignore
Jun 09 17:50:54 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:50:54 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:50:54 node2.company.corp pengine[4720]:   notice: Calculated transition 5467, saving inputs in /var/lib/pacemaker/pengine/pe-input-1573.bz2
Jun 09 17:50:54 node2.company.corp pengine[4720]:   notice: On loss of CCM Quorum: Ignore
Jun 09 17:50:54 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:50:54 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:50:54 node2.company.corp pengine[4720]:  warning: Forcing FTP_SERVICE away from node2.company.corp after 1000000 failures (max=1000000)
Jun 09 17:50:54 node2.company.corp pengine[4720]:   notice: Calculated transition 5468, saving inputs in /var/lib/pacemaker/pengine/pe-input-1574.bz2
Jun 09 17:50:54 node2.company.corp crmd[4721]:   notice: Transition 5468 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-1574.bz2): Complete
Jun 09 17:50:54 node2.company.corp crmd[4721]:   notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd
Jun 09 17:51:03 node2.company.corp crmd[4721]:  warning: No reason to expect node 2 to be down
Jun 09 17:51:03 node2.company.corp crmd[4721]:   notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph
Jun 09 17:51:03 node2.company.corp pengine[4720]:   notice: On loss of CCM Quorum: Ignore
Jun 09 17:51:03 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:51:03 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:51:03 node2.company.corp pengine[4720]:   notice: Calculated transition 5469, saving inputs in /var/lib/pacemaker/pengine/pe-input-1575.bz2
Jun 09 17:51:03 node2.company.corp crmd[4721]:   notice: Transition 5469 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-1575.bz2): Complete
Jun 09 17:51:03 node2.company.corp crmd[4721]:   notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd
Jun 09 17:51:38 node2.company.corp crmd[4721]:   notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph
Jun 09 17:51:38 node2.company.corp pengine[4720]:   notice: On loss of CCM Quorum: Ignore
Jun 09 17:51:38 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:51:38 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:51:38 node2.company.corp pengine[4720]:   notice: Calculated transition 5470, saving inputs in /var/lib/pacemaker/pengine/pe-input-1576.bz2
Jun 09 17:51:39 node2.company.corp pengine[4720]:   notice: On loss of CCM Quorum: Ignore
Jun 09 17:51:39 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:51:39 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:51:39 node2.company.corp pengine[4720]:   notice: Calculated transition 5471, saving inputs in /var/lib/pacemaker/pengine/pe-input-1577.bz2
Jun 09 17:51:39 node2.company.corp crmd[4721]:   notice: Transition 5471 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-1577.bz2): Complete
Jun 09 17:51:39 node2.company.corp crmd[4721]:   notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd
Jun 09 17:52:00 node2.company.corp crmd[4721]:  warning: No reason to expect node 2 to be down
Jun 09 17:52:00 node2.company.corp crmd[4721]:   notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph
Jun 09 17:52:01 node2.company.corp pengine[4720]:   notice: On loss of CCM Quorum: Ignore
Jun 09 17:52:01 node2.company.corp pengine[4720]:   notice: Start   FTP_SERVICE        (node2.company.corp)
Jun 09 17:52:01 node2.company.corp pengine[4720]:   notice: Calculated transition 5472, saving inputs in /var/lib/pacemaker/pengine/pe-input-1578.bz2
Jun 09 17:52:01 node2.company.corp crmd[4721]:   notice: Initiating monitor operation FTP_SERVICE_monitor_0 locally on node2.company.corp | action 5
Jun 09 17:52:01 node2.company.corp crmd[4721]:   notice: Result of probe operation for FTP_SERVICE on node2.company.corp: 196 (OCF_PENDING) | call=266 key=FTP_SERVICE_monitor_0 confirmed=true cib-update=6016
Jun 09 17:52:01 node2.company.corp crmd[4721]:  warning: Action 5 (FTP_SERVICE_monitor_0) on node2.company.corp failed (target: 7 vs. rc: 196): Error
Jun 09 17:52:01 node2.company.corp crmd[4721]:   notice: Transition aborted by operation FTP_SERVICE_monitor_0 'create' on node2.company.corp: Event failed | magic=0:196;5:5472:7:bf959758-83cf-4e4b-bd33-a360ffa08557 cib=0.454.1 source=match_graph_event:310 complete=false
Jun 09 17:52:01 node2.company.corp crmd[4721]:  warning: Action 5 (FTP_SERVICE_monitor_0) on node2.company.corp failed (target: 7 vs. rc: 196): Error
Jun 09 17:52:01 node2.company.corp crmd[4721]:   notice: Transition 5472 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=2, Source=/var/lib/pacemaker/pengine/pe-input-1578.bz2): Complete
Jun 09 17:52:01 node2.company.corp pengine[4720]:   notice: On loss of CCM Quorum: Ignore
Jun 09 17:52:01 node2.company.corp pengine[4720]:  warning: Processing failed op monitor for FTP_SERVICE on node2.company.corp: OCF_PENDING (196)
Jun 09 17:52:01 node2.company.corp pengine[4720]:  warning: Processing failed op monitor for FTP_SERVICE on node2.company.corp: OCF_PENDING (196)
Jun 09 17:52:01 node2.company.corp pengine[4720]:   notice: Recover FTP_SERVICE        (Started node2.company.corp)
Jun 09 17:52:01 node2.company.corp pengine[4720]:   notice: Calculated transition 5473, saving inputs in /var/lib/pacemaker/pengine/pe-input-1579.bz2
Jun 09 17:52:01 node2.company.corp crmd[4721]:   notice: Initiating stop operation FTP_SERVICE_stop_0 locally on node2.company.corp | action 4
Jun 09 17:52:01 node2.company.corp systemd[1]: Reloading.
Jun 09 17:52:21 node2.company.corp lrmd[4718]:   notice: Giving up on FTP_SERVICE stop (rc=196): timeout (elapsed=19988ms, remaining=12ms)
Jun 09 17:52:21 node2.company.corp crmd[4721]:    error: Result of stop operation for FTP_SERVICE on node2.company.corp: Timed Out | call=267 key=FTP_SERVICE_stop_0 timeout=20000ms
Jun 09 17:52:21 node2.company.corp crmd[4721]:  warning: Action 4 (FTP_SERVICE_stop_0) on node2.company.corp failed (target: 0 vs. rc: 198): Error
Jun 09 17:52:21 node2.company.corp crmd[4721]:   notice: Transition aborted by operation FTP_SERVICE_stop_0 'modify' on node2.company.corp: Event failed | magic=2:198;4:5473:0:bf959758-83cf-4e4b-bd33-a360ffa08557 cib=0.454.2 source=match_graph_event:310 complete=false
Jun 09 17:52:21 node2.company.corp crmd[4721]:  warning: Action 4 (FTP_SERVICE_stop_0) on node2.company.corp failed (target: 0 vs. rc: 198): Error
Jun 09 17:52:21 node2.company.corp crmd[4721]:   notice: Transition 5473 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=3, Source=/var/lib/pacemaker/pengine/pe-input-1579.bz2): Complete
Jun 09 17:52:21 node2.company.corp pengine[4720]:   notice: On loss of CCM Quorum: Ignore
Jun 09 17:52:21 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:52:21 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:52:21 node2.company.corp pengine[4720]:   notice: Calculated transition 5474, saving inputs in /var/lib/pacemaker/pengine/pe-input-1580.bz2
Jun 09 17:52:21 node2.company.corp pengine[4720]:   notice: On loss of CCM Quorum: Ignore
Jun 09 17:52:21 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:52:21 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:52:21 node2.company.corp pengine[4720]:  warning: Forcing FTP_SERVICE away from node2.company.corp after 1000000 failures (max=1000000)
Jun 09 17:52:21 node2.company.corp pengine[4720]:   notice: Calculated transition 5475, saving inputs in /var/lib/pacemaker/pengine/pe-input-1581.bz2
Jun 09 17:52:21 node2.company.corp crmd[4721]:   notice: Transition 5475 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-1581.bz2): Complete
Jun 09 17:52:21 node2.company.corp crmd[4721]:   notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd
Jun 09 17:52:58 node2.company.corp crmd[4721]:   notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph
Jun 09 17:52:58 node2.company.corp pengine[4720]:   notice: On loss of CCM Quorum: Ignore
Jun 09 17:52:58 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:52:58 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:52:58 node2.company.corp pengine[4720]:  warning: Forcing FTP_SERVICE away from node2.company.corp after 1000000 failures (max=1000000)
Jun 09 17:52:58 node2.company.corp pengine[4720]:   notice: Calculated transition 5476, saving inputs in /var/lib/pacemaker/pengine/pe-input-1582.bz2
Jun 09 17:52:58 node2.company.corp crmd[4721]:   notice: Transition 5476 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-1582.bz2): Complete
Jun 09 17:52:58 node2.company.corp crmd[4721]:   notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd
Jun 09 17:53:09 node2.company.corp crmd[4721]:   notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph
Jun 09 17:53:09 node2.company.corp pengine[4720]:   notice: On loss of CCM Quorum: Ignore
Jun 09 17:53:09 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:53:09 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:53:09 node2.company.corp pengine[4720]:  warning: Forcing FTP_SERVICE away from node2.company.corp after 1000000 failures (max=1000000)
Jun 09 17:53:09 node2.company.corp pengine[4720]:   notice: Calculated transition 5477, saving inputs in /var/lib/pacemaker/pengine/pe-input-1583.bz2
Jun 09 17:53:09 node2.company.corp crmd[4721]:   notice: Transition 5477 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-1583.bz2): Complete
Jun 09 17:53:09 node2.company.corp crmd[4721]:   notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd
Jun 09 17:53:28 node2.company.corp crmd[4721]:   notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph
Jun 09 17:53:28 node2.company.corp pengine[4720]:   notice: On loss of CCM Quorum: Ignore
Jun 09 17:53:28 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:53:28 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:53:28 node2.company.corp pengine[4720]:  warning: Forcing FTP_SERVICE away from node2.company.corp after 1000000 failures (max=1000000)
Jun 09 17:53:28 node2.company.corp pengine[4720]:   notice: Calculated transition 5478, saving inputs in /var/lib/pacemaker/pengine/pe-input-1584.bz2
Jun 09 17:53:28 node2.company.corp pengine[4720]:   notice: On loss of CCM Quorum: Ignore
Jun 09 17:53:28 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:53:28 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:53:28 node2.company.corp pengine[4720]:  warning: Forcing FTP_SERVICE away from node2.company.corp after 1000000 failures (max=1000000)
Jun 09 17:53:28 node2.company.corp pengine[4720]:   notice: Calculated transition 5479, saving inputs in /var/lib/pacemaker/pengine/pe-input-1585.bz2
Jun 09 17:53:28 node2.company.corp crmd[4721]:   notice: Transition 5479 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-1585.bz2): Complete
Jun 09 17:53:28 node2.company.corp crmd[4721]:   notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd
Jun 09 17:53:31 node2.company.corp systemd[1]: vsftpd.service stop-final-sigterm timed out. Killing.
Jun 09 17:53:31 node2.company.corp systemd[1]: Stopped Vsftpd ftp daemon.
-- Subject: Unit vsftpd.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit vsftpd.service has finished shutting down.
Jun 09 17:53:31 node2.company.corp systemd[1]: Unit vsftpd.service entered failed state.
Jun 09 17:53:31 node2.company.corp systemd[1]: vsftpd.service failed.

Pacemaker configuration

Cluster Name: FTP_CLUSTER
Corosync Nodes:
 node1 node2
Pacemaker Nodes:
 node1 node2

 

Resources:
 Resource: INTERNAL_FTP_VIRTUAL_IP (class=ocf provider=heartbeat type=IPaddr2)
  Attributes: ip=111.55.3.25 cidr_netmask=24
  Operations: start interval=0s timeout=20s (INTERNAL_FTP_VIRTUAL_IP-start-interval-0s)
              stop interval=0s timeout=20s (INTERNAL_FTP_VIRTUAL_IP-stop-interval-0s)
              monitor interval=10s (INTERNAL_FTP_VIRTUAL_IP-monitor-interval-10s)
 Resource: INTERNAL_FTP (class=systemd type=vsftpd)
  Operations: monitor interval=10s (INTERNAL_FTP-monitor-interval-10s)
 Resource: TOMCAT (class=ocf provider=heartbeat type=tomcat)
  Attributes: java_home=/usr/java/jdk1.8.0_181 catalina_home=/usr/local/tomcat
  Operations: start interval=0s timeout=60s (TOMCAT-start-interval-0s)
              stop interval=0s timeout=120s (TOMCAT-stop-interval-0s)
              monitor interval=15s (TOMCAT-monitor-interval-15s)
 Resource: INTERNAL_FTP_VIRTUAL_IP_APACHE (class=ocf provider=heartbeat type=IPaddr2)
  Attributes: ip=111.99.3.25 cidr_netmask=32 nic=bond0.1104
  Operations: start interval=0s timeout=20s (INTERNAL_FTP_VIRTUAL_IP_APACHE-start-interval-0s)
              stop interval=0s timeout=20s (INTERNAL_FTP_VIRTUAL_IP_APACHE-stop-interval-0s)
              monitor interval=30s (INTERNAL_FTP_VIRTUAL_IP_APACHE-monitor-interval-30s)

 

Stonith Devices:
Fencing Levels:

 

Location Constraints:
  Resource: INTERNAL_FTP
    Enabled on: node2 (score:INFINITY) (role: Started) (id:cli-prefer-INTERNAL_FTP)
  Resource: TOMCAT
    Enabled on: node2 (score:INFINITY) (role: Started) (id:cli-prefer-TOMCAT)
  Resource: INTERNAL_FTP_VIRTUAL_IP
    Enabled on: node1 (score:INFINITY) (id:location-INTERNAL_FTP_VIRTUAL_IP-node1-INFINITY)
    Enabled on: node2 (score:INFINITY) (role: Started) (id:cli-prefer-INTERNAL_FTP_VIRTUAL_IP)
  Resource: INTERNAL_FTP_VIRTUAL_IP_APACHE
    Enabled on: node2 (score:INFINITY) (role: Started) (id:cli-prefer-INTERNAL_FTP_VIRTUAL_IP_APACHE)
Ordering Constraints:
  start INTERNAL_FTP_VIRTUAL_IP then start INTERNAL_FTP (kind:Mandatory) (id:order-INTERNAL_FTP_VIRTUAL_IP-INTERNAL_FTP-mandatory)
Colocation Constraints:
  INTERNAL_FTP with INTERNAL_FTP_VIRTUAL_IP (score:INFINITY) (id:colocation-INTERNAL_FTP-INTERNAL_FTP_VIRTUAL_IP-INFINITY)
Ticket Constraints:

 

Alerts:
 No alerts defined

 

Resources Defaults:
 No defaults set
Operations Defaults:
 No defaults set

 

Cluster Properties:
 cluster-infrastructure: corosync
 cluster-name: FTP_CLUSTER
 dc-version: 1.1.15-11.el7-e174ec8
 have-watchdog: false
 last-lrm-refresh: 1624555372
 no-quorum-policy: ignore
 stonith-enabled: false

 

Quorum:
  Options:

Versions: Oracle Linux Server release 7.2 vsftpd: version 3.0.2 pacemaker-1.1.15-11.el7.x86_64

176
0 + 2
Matt Kereczman avatar
nr flag
Matt Kereczman
How is the Pacemaker cluster configured?
1
Reply
cape avatar
de flag
cape
Hello Matt. Thanks for your answer. I've just updated the post with the pacemaker configuration.
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.