self signed certificate vs a certificat authority with email

I use a self signed certificate with my mail server. I use dovecot and postfix startTLS. I also have a DKIM setup for digital signatures and everything is working. However, when sending mail to hotmail or outlook it often if not always falls into the client spam unless they request to trust.

I wouldn't make any difference if I would pay to have a signed certificate authority would it? I mean a certificate guarantees that when thunderbird connects to my server to send the email it's encrypted, once postfix sends this to the target host there is no guarantee it then is still encrypted. So self sign or paid cert I don't gain a "whitelist reputation" with big box companies like outlook or hotmail? Which is always the problem- I never have issues with gmail.

I want to try and get on hotmail/outlook.com's accepted mail reputation and my mail is encrypted with digital signatures. Any advice would be welcome.

My ip/domain has a good reputation etc.

First of all, Let's Encrypt is free so you're in no obligation to pay for a valid certificate.

Second, you seriously need to read up on how PKI works. A self-signed certificate is worthless since it cannot be authenticated, meaning that any MitM could replace it with their own self-signed certificate and you couldn't tell whether it's any more legitimate than yours.

Third, email is incredibly complex when it comes to opaque spam filters. It's unlikely that your certificate matters since it is only used when you receive emails, not when you send them in which case it's actually your email server checking their certificate and not the other way around, but you never know.

I would advise making sure that your DMARC, DKIM and SPF are properly setup since they might be the cause for failure (I know Gmail to be lax in some cases). Beyond that, ask Microsoft what they're basing their spam trigger on (good luck with that) or ask them to remove you from it (good luck with that too).