I don't think outsourcing your key management to a third party makes any difference in the security of your application than doing the key management on your server. However, as you mention, you are not allowed to do the key management on your server.
In both cases, I think it is better to do the key management on your client without storing any data on it.
Assuming any user has a username/id and password, I propose you the following key management process:
- For the key generation, a client generates the key pair locally. Using its user id and password as an input in a KDF a symmetric key is generated which can be used to symmetrically encrypt the private key of the pair. Now the private key can be send back to the server and stored there locally. The public key can be sent plain back to the server to process it as you mention.
- In the case of an invitation, a client can retrieve its private key from the server and decrypt it using its id and password as an input to the KDF to obtain the symmetric key. Then a user can sign an appropriate message and send it to another user. Finally the other user the only thing that has to do it to provide the signed message he obtained to the server that will check if it is signed by a pre-existing user and allow the new user to use the service.