I have already checked the solutions at this post, but I can't find the reason.
This is my situation:
- Running postfix inside a docker container (docker swarm).
- Docker container has no quota.
/var/mail volume mounted from /mnt/shared/postfix/var/mail/ in host./mnt/shared/ is a network logical device using glusterfs (xfs formatted) with 3 nodes.
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda2 20G 12G 7.1G 62% /
/dev/vda2 20G 12G 7.1G 62% /var/spool/postfix
/dev/vda3 60G 31G 30G 51% /gluster
mx1:shared 120G 63G 58G 52% /mnt/shared
# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/vda2 1310720 200164 1110556 16% /
/dev/vda2 1310720 200164 1110556 16% /var/spool/postfix
/dev/vda3 31455744 523000 30932744 2% /gluster
mx1:shared 31455744 523000 30932744 2% /mnt/shared
# df -k
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/vda2 20466256 8816788 10584508 46% /
/dev/vda2 20466256 8816788 10584508 46% /var/spool/postfix
/dev/vda3 62880772 32043020 30837752 51% /gluster
manager1:shared 125761544 65343656 60417888 52% /mnt/shared
# gluster volume status shared detail
Status of volume: shared
---------------------------------------------
Brick : Brick mx1:/gluster/brick
TCP Port : 59830
RDMA Port : 0
Online : Y
Pid : 727880
File System : xfs
Device : /dev/vda3
Mount Options : rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
Inode Size : 512
Disk Space Free : 29.4GB
Total Disk Space : 60.0GB
Inode Count : 31455744
Free Inodes : 30932804
---------------------------------------------
Brick : Brick mx2:/gluster/brick
TCP Port : 55380
RDMA Port : 0
Online : Y
Pid : 3784930
File System : xfs
Device : /dev/vda3
Mount Options : rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
Inode Size : 512
Disk Space Free : 29.4GB
Total Disk Space : 60.0GB
Inode Count : 31455744
Free Inodes : 30932804
---------------------------------------------
Brick : Brick mx3:/gluster/brick
TCP Port : 58943
RDMA Port : 0
Online : Y
Pid : 3905147
File System : xfs
Device : /dev/vda3
Mount Options : rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota
Inode Size : 512
Disk Space Free : 29.4GB
Total Disk Space : 60.0GB
Inode Count : 31455744
Free Inodes : 30932804
# gluster volume status shared inode
Brick : mx1:/gluster/brick
Connection 1:
LRU limit : 16384
Active Inodes : 958
LRU Inodes : 16383
Purge Inodes : 0
---------------------------------------------
Brick : mx2:/gluster/brick
Connection 1:
LRU limit : 16384
Active Inodes : 829
LRU Inodes : 16373
Purge Inodes : 0
---------------------------------------------
Brick : mx3:/gluster/brick
Connection 1:
LRU limit : 16384
Active Inodes : 1087
LRU Inodes : 16370
Purge Inodes : 0
# postconf -n
Most relevant:
mailbox_size_limit = 100000000
message_size_limit = 50000000
Full config:
alias_database = texthash:/etc/aliases
alias_maps = texthash:/etc/aliases
append_dot_mydomain = no
biff = no
bounce_queue_lifetime = 10m
broken_sasl_auth_clients = yes
compatibility_level = 3.6
content_filter = smtp-amavis:amavis:10024
disable_vrfy_command = yes
header_checks = regexp:/etc/postfix/config/header_checks.regex
inet_interfaces = all
inet_protocols = ipv4
mailbox_size_limit = 100000000
maillog_file = /var/log/mail.log
maximal_backoff_time = 10m
maximal_queue_lifetime = 10m
message_size_limit = 50000000
mydestination = localhost
myhostname = mail.example.com
mynetworks = /etc/postfix/config/my.networks.plain
myorigin = $myhostname
policy-spf_time_limit = 3600s
postscreen_access_list = permit_mynetworks cidr:/etc/postfix/config/postscreen_access.cidr
postscreen_blacklist_action = drop
postscreen_dnsbl_action = enforce
postscreen_dnsbl_reply_map = texthash:/etc/postfix/config/dnsbl_reply
postscreen_dnsbl_sites = zen.spamhaus.org=127.0.0.[2..11] b.barracudacentral.org psbl.surriel.com bl.spamcop.net bl.spameatingmonkey.net dnsbl.sorbs.net swl.spamhaus.org*-2 list.dnswl.org=127.[0..255].[0..255].0*-2, list.dnswl.org=127.[0..255].[0..255].1*-4, list.dnswl.org=127.[0..255].[0..255].[2..3]*-6
postscreen_greet_action = drop
queue_directory = /var/spool/postfix
queue_run_delay = 5m
readme_directory = no
recipient_delimiter = +
relayhost =
smtp_tls_exclude_ciphers = EXPORT, LOW, MD5, aDSS, kECDHe, kECDHr, kDHd, kDHr, SEED, IDEA, RC2, RC4, aNULL
smtp_tls_loglevel = $smtpd_tls_loglevel
smtp_tls_mandatory_ciphers = medium
smtp_tls_mandatory_protocols = $smtpd_tls_protocols
smtp_tls_note_starttls_offer = yes
smtp_tls_protocols = $smtpd_tls_protocols
smtp_tls_security_level = $smtpd_tls_security_level
smtpd_banner = $myhostname ESMTP
smtpd_client_connection_rate_limit = 60
smtpd_client_message_rate_limit = 100
smtpd_client_new_tls_session_rate_limit = 60
smtpd_client_recipient_rate_limit = 100
smtpd_client_restrictions = check_client_access texthash:/etc/postfix/config/client_access.hash permit_mynetworks reject_unauth_pipelining permit_sasl_authenticated permit
smtpd_delay_reject = yes
smtpd_error_sleep_time = 1s
smtpd_hard_error_limit = 20
smtpd_helo_required = yes
smtpd_helo_restrictions = permit_mynetworks permit_sasl_authenticated reject_invalid_helo_hostname permit
smtpd_recipient_restrictions = reject_non_fqdn_recipient check_recipient_access texthash:/etc/postfix/config/recipient_access.hash check_sender_access texthash:/etc/postfix/config/sender_access.hash permit_sasl_authenticated check_policy_service unix:private/policy-spf warn_if_reject reject_unknown_recipient_domain permit_mynetworks reject_unauth_destination warn_if_reject reject_unverified_recipient reject_rhsbl_sender dbl.spamhaus.org=127.0.1.[2..99] reject_rhsbl_helo dbl.spamhaus.org=127.0.1.[2..99] reject_rhsbl_reverse_client dbl.spamhaus.org=127.0.1.[2..99] warn_if_reject reject_rbl_client zen.spamhaus.org=127.255.255.[1..255] permit
smtpd_sasl_auth_enable = yes
smtpd_sasl_authenticated_header = yes
smtpd_sasl_local_domain = $myhostname
smtpd_sasl_path = inet:dovecot:999
smtpd_sasl_security_options = noanonymous
smtpd_sasl_type = dovecot
smtpd_sender_login_maps = mysql:/etc/postfix/mysql.users.cf
smtpd_sender_restrictions = check_sender_access texthash:/etc/postfix/config/sender_access.hash permit_sasl_authenticated permit_mynetworks reject_non_fqdn_sender permit
smtpd_soft_error_limit = 10
smtpd_tls_CAfile = /etc/postfix/ssl/chain.pem
smtpd_tls_auth_only = yes
smtpd_tls_cert_file = /etc/postfix/ssl/cert.pem
smtpd_tls_dh1024_param_file = ${config_directory}/dh2048.pem
smtpd_tls_dh512_param_file = ${config_directory}/dh512.pem
smtpd_tls_eecdh_grade = strong
smtpd_tls_exclude_ciphers = EXPORT, LOW, MD5, SEED, IDEA, RC2, RC4, aNULL
smtpd_tls_key_file = /etc/postfix/ssl/privkey.pem
smtpd_tls_loglevel = 1
smtpd_tls_mandatory_ciphers = high
smtpd_tls_mandatory_protocols = $smtpd_tls_protocols
smtpd_tls_protocols = >=TLSv1.2
smtpd_tls_received_header = yes
smtpd_tls_security_level = may
smtpd_tls_session_cache_timeout = 3600s
smtpd_use_tls = yes
tls_preempt_cipherlist = yes
tls_random_source = dev:/dev/urandom
transport_maps = texthash:/etc/postfix/config/transport.hash
virtual_alias_maps = mysql:/etc/postfix/mysql.aliases.cf
virtual_gid_maps = static:5000
virtual_mailbox_base = /var/mail
virtual_mailbox_domains = mysql:/etc/postfix/mysql.domains.cf
virtual_mailbox_maps = mysql:/etc/postfix/mysql.users.cf
virtual_minimum_uid = 5000
virtual_uid_maps = static:5000
The error looks like:
warning: not enough free space in mail queue: 0 bytes < 1.5*message size limit
94AA9C30D6: to=<[email protected]>, orig_to=<[email protected]>, relay=amavis[10.0.5.92]:10024, delay=376, delays=371/0/0/5.3, dsn=4.3.1, status=deferred (host amavis[10.0.5.92] said: 452 4.3.1 id=97454-19 - Temporary MTA failure on relaying, from MTA(smtp:[postfix]:10025): 452 4.3.1 Insufficient system storage (in reply to end of DATA command))
The mail queue is practically empty (reported by mailq)
The free space reported by postfix gradually decreased until it reached 0. Where does that value comes from? Is there any way to get the total free space "seen" by postfix?
Ultimately I restarted the server and I'm not seeing those warnings/errors anymore. However I suspect the problem will be back soon or later. I don't want keep restarting the server each time, so I would like to know why is that happening.
