AD FS 3.0 SSO issue in Outlook Desktop Application with Modern Auth for MS365

I have installed and configured AD FS services on a Microsoft Windows Server 2016 Standard.

Through Azure AD Connect we were able to configure our domain as a federated domain on our Microsoft 365 tenant. Besides that the Azure AD Connect also automatically configured a Relying Party Trust for Microsoft Office 365 Identity Platform Worldwide

We created a TestUser in our domain with the UPN [email protected]. This user was synced to the MS365 tenant where we assigned a Exchange Online license to it.

When we started testing we were successfully able to access https://outlook.office.com via Microsoft Edge on the PC of TestUser which verifies that the SSO is working.

However when we open the Outlook 2019 (included via a Microsoft Office 2019 Standard installation) from TestUser the UPN is automatically inserted in the Simplified Account Creation wizard. When we continue in the dialog a Modern Auth window pops up containing our AD FS login page. Here it asks for the password of the TestUser and thus SSO is not available.

In the AD FS Event Viewer we get the following error message:

The Federation Service could not authorize token issuance for caller 'DOMAIN\TestUser
'. The caller is not authorized to request a token for the relying party 'urn:federation:MicrosoftOnline'. See event 501 with the same Instance ID for caller identity. 

Additional Data 
Instance ID: 9c026fe6-4068-4a47-9e89-e4248dd5ca85 
Relying party: urn:federation:MicrosoftOnline 
Exception details: 
Microsoft.IdentityServer.Service.IssuancePipeline.CallerAuthorizationException: MSIS5007: The caller authorization failed for caller identity DOMAIN\TestUser for relying party trust urn:federation:MicrosoftOnline.
   at Microsoft.IdentityModel.Threading.AsyncResult.End(IAsyncResult result)
   at Microsoft.IdentityModel.Protocols.WSTrust.WSTrustServiceContract.ProcessCoreAsyncResult.End(IAsyncResult ar)
   at Microsoft.IdentityModel.Protocols.WSTrust.WSTrustServiceContract.EndProcessCore(IAsyncResult ar, String requestAction, String responseAction, String trustNamespace) 
User Action 
Use the AD FS Management snap-in to ensure that the caller is authorized to request a token for the relying party.

We checked the Issuance Authorization Rules for the Relying Party Trust. This only contains one rule: Permit Access to All Users

Any advice on how we could get SSO working in Outlook desktop app would be greatly appreciated!