How are the instantiations of RSAES-OAEP and SHA*WithRSAEncryption different in practice?

For the spare-time project I had been working on, I'm evaluating the PKCS#1 padded RSA schemes for implementation.

For PKCS#1 v1.5, encryption doesn't seem to require a hash function, and the signature doesn't need additional mask-generating function (MGF) beyond a digest algorithm for hashing the message.

For PKCS#1 v2.x, both encryption and signature are instantiated with a MGF, a hash function, and an optional label (which has no specified use currently). The MGF which, in turn, is instantiated with another hash function using the (ad hoc) MGF1 construction.

In my opinion, a composite cryptosystem should have as few instantiation parameters as possible, so as to ease implementation burdon and promote interoperability; only parameters necessary for cryptographic agility should be inserted into the design.

Which is why, it comes to me as a suprise, that the hash that instantiates the MGF can be different from that used for hashing the input.

My question here are:

• are there existing recommendations (e.g. CMS, PKIX) on the use and instantiation of PKCS#1 v2.x RSAES-OAEP and RSASSA-PSS, where the selection of hash functions are specified?
• Are there assigned IDs (from IANA or similar organizations)?
• Additional points goes to treatment of NIST Draft FIPS 186-5, where SHAKE-{128,256} are to be approved for use as MGF.

I'll start with a functional comparison of RSAES-PKCS1-v1_5 with RSAES-OAEP.

The later is the modern near-substitute of the former.

First and foremost, it's essentially impossible to make a library implementing RSAES-PKCS1-v1_5 decryption that insures security against side-channel attacks. Many attempts to get application-level security have failed: an adversary able to make a modest number of queries to a device performing decryption and observe it's behavior (error code, packet size, timing, disk accesses, cache, power supply sound…) often succeeds [that is manages to decrypt one message they intercepted, or sign one message if the key is dual use]. Bleichenbacher's CCA attack has so many variations that it's hard to keep track. Contrast with RSAES-OAEP: a limited amount of care in a library implementation prevents the equivalent attack.

Also, RSAES-PKCS1-v1_5 is defined to allow down to 64-bit randomness, which is not enough by modern standards to robustly prevent testing an exact guess of plaintext. The only way to fix this is to prevent encryption of messages too close to the maximum size allowed.

RSAES-OAEP has a proof of security (under the assumption the RSA problem is hard, and the hash can be modeled as a pseudo-random function, and the implementation has no side channel). RSAES-PKCS1-v1_5 does not.

One downside of RSAES-OAEP is that it allows smaller messages for the same modulus, but that's seldom an issue in practice, for when message size grows we use hybrid encryption.

existing recommendations (e.g. CMS, PKIX) on the use and instantiation of RSAES-OAEP and RSASSA-PSS

As pointed in comment to the question, the current best practice is to use MGF1 with the same hash as for the rest of the construction. SHA-256 is common, SHA-512 would be best. The most common deviation from that is use of SHA-1 in MGF1 for RSAES-OAEP, because the Java API makes it an easy mistake (see this). That's still secure as far as we know.

For RSASSA-PSS, a common (but not the best) practice is to leave the salt field empty, that is use $sLen=0$ in EMSA-PSS.

Using SHAKE-{128,256} as MGF would be sound in theory, but has not made it to practice in applications I know, and might never do.

Are there assigned IDs (from IANA or similar organizations)?

There are OIDs for RSAES-OAEP, RSASSA-PSS, MGF1, SHA-256 and SHA-512. Now how should these OIDs be used in combination in e.g. an X.509 certificate, and is there something that will keep on working if that's attempted? I pass.