Although RSA is not meant for encryption one can use RSA for encryption. If one uses TextBook RSA then it will be insecure since the encryption is free then any attacker can check the values. We call this the encryption oracle and it is free on public-key systems
A simple RSA encryption oracle game...
def Ind_CPA_RSA(adversary, target):
(e,n,d,...) = generate_RSA_key() //keygen part
def RSA_encryption_oracle_PKCS#1_v1.5(plaintext): //Encryption oracle
EM = PKCS#1_v1.5_padding(plaintext)
ciphertext = EM^e mod n
return ciphertext
for each m in possible_message_space: //queries
c = RSA_encryption_oracle_PKCS#1_v1.5(m)
if c == target
print(target)
return succcess
return failure
So, the adversary tries all possible messages as long as they can, to see the equality to win.
In textbook RSA if public exponent $e=3$ then cube-root attack works for all messages suche that $len(m) < \sqrt[3]{n}$.
For all other attacks the Dan Boneh's article is a good starting point;
$$\textbf{Never use Text Book RSA as long as you know what you do!}$$
To be secure one has to use RSA encryption with proper paddings PKCS#1 v1.5 (RSAES-PKCS1-v1_5) or OAEP (RSAES-OAEP) padding. These paddings add randomization to achieve probabilistic encryption.
Each uses special encodings to achieve this like PKCS#1 v1.5 padding;
EM = 0x00 || 0x02 || PS || 0x00 || M.
M
is the message. The PS
consist of the randomization part
Generate an octet string PS of length k - mLen - 3
consisting of pseudo-randomly generated nonzero octets.
The length of PS will be at least eight octets.
For example for 2048-bit RSA; $k = 256$, $mLen=4$ then PS
length is 249 bytes of randomness for one letter-sized message. Therefore the attacker cannot test the values with the encryption oracle. The rest is attacking the RSA problem.
Similarly, OAEP has randomness and OAEP has proven to have IND-CCA1 security. Prefer OAEP to PKCS#1 v1.5 since it has many attacks due to improper implementations.
If anyone wants an academic article about metrics of RSA encryption here is the paywalled article;