How to solve issues with Exchange's Autodiscover (xml) and any peculiar clients like Windows Communication Apps [HxTsr.exe] with complex DNS config?

For the search engine, context elements:

• Exchange Online
• Office 365 / Microsoft 365 --> Particular Outlook --> outlook.office.com
• Windows' Mail & Calendar App [HxTsr.exe, "microsoft.windowscommunicationsapps"]
• Apex domain redirect to www.~
• Cloudflare DNS
• Using Cloudflare workers
• HSTS, also for subdomains
• [Azure's] Conditional Access is blocking legacy authentication methods. ('basic authentication'.)
• cname of autodiscover.domain.tld was configured towards autodiscover.outlook.com. as expected, Cloudflare's tricks disabled for this record.
• Cloudflare is providing SSL for most domains, except autodiscover.domain.tld.

Now the undesired behaviour:

In this case, the Autodiscover was not working for the 'windowscommunicationsapps', henceforth named 'problem client' or 'client'. It was working for latest Outlook for Windows most likely due to the more modern built-in autodiscovery methods that skip the legacy autodiscovery ones.

The problem client would load and load and prompt a basic authentication password, and after that, prompt for a username and domain and after that display an error with the option to go to advanced and enter everything again including the server. After that, after I remembered to enter outlook.office365.com as server, it prompted the user with Microsoft's modern authentication window.

What was going wrong:

The client tried to retrieve https://rootdomain.tld/autodiscover/autodiscover.xml. (Seen in cloudflare's firewall's log.)

It got redirected to https://www.rootdomain.tld/autodiscover/autodiscover.xml. (Seen in MS's tool.)

Due to a complex configuration of Cloudflare workers, the client did not get a 404 back in first instance. It just kept loading and loading. In variation config, a dedicated Cloudflare worker returned a 404 page; it did not resolve the issue.

The problem was that there was too much going on.

• http was redirected to https.
• Root (apex) was redirected to www.domain.tld/$1 and
• autodiscover.~ supposed to resolve to autodiscover.outlook.com but for some reason did not deliver the xml. (With Microsoft's tool showed me:

Testing TCP port 443 on host autodiscover.juramento.nl to ensure it's listening and open.

The specified port is either blocked, not listening, or not producing the expected response.

• One of the next step in the Autodiscovery-quest is looking for redirects at autodiscover.domain.tld and that made it find https://autodiscover-s.outlook.com/Autodiscover/Autodiscover.xml

The assumption was that the problematic client was not getting this far and giving up on it's autodiscover-quest too soon. I am unsure if the GET request at autodiscover.domiain.tld:443 is supposed to deliver an error for Exchange Online, but the redirect worked as desired and as soon as Microsoft's tool found https://autodiscover-s.outlook.com/Autodiscover/Autodiscover.xml every thing turned green until the expected basis authentication error due to the blocking of legacy authentication methods in conditional access.

The questions:

1. Why can't this client connect to Exchange / Office 365 quickly?
2. Why is this client asking for basic authentication?
3. Why is autodiscovery not working for this domain?
4. What is autodiscovery actually doing step for step?
5. What can we do to shorten the autodiscovery steps and aid the original problematic client?
6. What can I do to fix autodiscovery for Exchange Online quickly?

Tips

• Do create an extra test account.
• Use https://testconnectivity.microsoft.com/tests/Eas/input as sanity check.
• This test https://testconnectivity.microsoft.com/tests/O365Eas/input is smarter, but did not mimic the problematic client's behaviour.

Solution

I used Cloudflare DNS, added a forwarding rule to forward this particular non-existent resource:

https://apexdomain.tld/autodiscover/autodiscover.xml

to

https://autodiscover-s.outlook.com/Autodiscover/Autodiscover.xml

This shortens the autodiscovery chain significantly. (Confirmed by the Microsoft's tool.) Therefore, it also limits the number of things that can go wrong.

The problematic client now shows the modern authentication window very quickly as desired.

Notes

• I disabled the Cloud worker delivering 404 for https://apexdomain.tld/autodiscover/autodiscover.xml, though I left it working for https://www.domain.tld/autodiscover/autodiscover.xml even though I do not see the chain getting there anymore. The only reason this URI was visited is because of the apex to www. forwarding rule. So I just figured out it can remove this route too ;), but for the record of test case, it has been active during the test.
• I don't know if Cloudflare Rules take priority over Cloudflare Worker routes. You should configure them xOR anyway.

I find the complexity of autodiscover troublesome. Much can go wrong. Ironically the autodisover mechanism is so complex, one could create a autodiscovery's autodiscover config to automatically load where to start the autodiscovery for a particular domain. :/

• Just before posting, I found a reference in the auto-search box to following post. The creative poster has encountered similar problems, including the 443 port error and a similar solution, just a different location (reverse proxy).

It perplexes me the post didn't show up in my google results, but I am happy to see some of our observations and resolutions overlap. ^^

Sources

https://practical365.com/fixing-autodiscover-root-domain-lookup-issues-mobile-devices/

https://www.datarepairtools.com/blog/autodiscover-not-working-while-connecting-to-office-365-account/

Related reading:

https://docs.microsoft.com/en-us/exchange/architecture/client-access/autodiscover?view=exchserver-2019#autodiscover-services-in-outlook