My network consists of two different locations and in both I have a Postfix server running for outgoing mail on each of them. Mail gets submitted to any of the two round-robin, based on having them both in DNS, e.g. smtp.mydomain.tld points to two A-records. Users submit their mail there and these servers will send the mail out. All good so far in the happy flow.
Now, sometimes it happens that the nexthop mail server to deliver the mail to is unreachable from one of the two networks. Say, for example, that (VPS) host smtp1 is hosted in EU with cloud provider A and smtp2 is hosted in US by cloud provider B. A mail submitted with a recipient on the example.com domain could sometimes only be delivered through smtp2, because of some networking issues or ASN-level blocks by the receiving party for smtp1. This could be a TCP connection timeout, a temporary SMTP error with a message mentioning a ban for our network/location, etc. - I've seen a lot of different type of failures here.
In practice I have seen mail hosts blocking/throttling based on what appears to be based on geolocation, e.g. an American agency's mail server only allowing connections from the US.
What I have tried so far is 'transplanting' these mails manually out of the /var/spool/postfix directory to the other host. And this works. But trying to automate this seems ridiculously complicated.
What I also have tried is setting up transport overrides for known-bad recipient domains. They will then be forwarded to the known-good outgoing smtp server to be delivered OK instead of even attempting to deliver directly to the internet. However, maintaining the list is cumbersome on individual hosts and requires signals from users first together with confirmation what is a known-good path.
What I would like to accomplish is to automate a 'transplant' of these problematic queued messages to the other outbound SMTP server to retry delivery from there. However, I feel that it seems a complicated patch for something that should be fixed in the design.
Ideally, I would like to have a shared outgoing mail queue (database?) for a potential larger amount of outgoing mail servers. This cluster would then attempt to deliver the mail in turn by turn until successful. I just fail to find any non-local directory 'backend' for Postfix as message queue - it seems really limited to just a single host.
Such an ideal design would then also allow us to centralize the outgoing mail queue state to a much more reliable storage and consider the hosts running Postfix as more ephemeral servers that you can scale up and down on demand.
Any attempt to work-around the existing use of a shared queue directory will simply fail, because (1):
The reason is that you cannot share Postfix queues among multiple running Postfix instances.
Anyway, to keep the scope here on the initial issue... how do you handle such deliverability issues with your mail cluster? Looking for a solution that uses Postfix, preferably.
