FIPS 186-4, Use of RSA key pair for Encryption and Signature

Section 5.1 of FIPS 186-4 states that

An RSA key pair used for digital signatures shall only be used for one digital signature scheme (e.g., ANS X9.31, RSASSA-PKCS1 v1.5 or RSASSA-PSS)

and

RSA digital signature key pair shall not be used for other purposes (e.g., key establishment)

What problems can arise from not following these guidelines?

Use of same RSA key pair with RSA-OAEP and RSA-PSS is also vulnerable?

Here is one situation where this could be messy. Let $(N=pq, e, d)$ be an RSA-tuple that is used for signatures and encryption. (Here I use textbook RSA / textbook RSA signatures for the sake of simplicity; there are more robust schemes). Given a signature oracle (i.e. an oracle that signs any message and returns the signature), we can decrypt RSA ciphertexts.

1. Adversary is given ciphertext $c\equiv m^e \;(\text{mod } \phi(N))$.
2. Adversary queries signature oracle to obtain $c^d\equiv m\;(\text{mod } \phi(N))$

How realistic is access to a signature oracle? Any situation where you can explicitly obtain signatures from an authority. For example, a certificate authority gives signatures for certificates. (of course, the schemes are a lot more complicated and domain separated to prevent these types of attacks).

What problems can arise from not following these guidelines

We loose the security reductions (of RSASSA-PSS and RSAES-OAEP) or security endorsements (for others), since they are made under the hypothesis that the RSA key is used only for the operation mode considered. And we know there is potential for problem from actual attacks that can occur when using the same key for signing (any mode), and encryption or signing by textbook RSA (a decryption or signing oracle for textbook RSA can be turned into a signing oracle for any mode).

Also to consider: a poorly conceived signature padding or implementation has the potential to leak the key itself, thus killing the security of a perfectly good system using the same key. However this is not as much to fear with RSA as it is with Rabin signature, where there are a number of attacks on this theme.

What if one uses RSA-OAEP and RSA-PSS with the same RSA key pair?

There's no known attack, but the existing security reductions no longer apply. I'd be surprised if an attack emerged, and not surprised if a security reduction for the combination could be made (e.g. from the reduction for Full Domain Hash), with a lowered proven quantitative security level perhaps.

In any case, it's good practice to stick to the principle: one usage, one key.