I am familiar with any algorithm that is based on the number of bits in a key (n-bit) and how to restore a key, such as the Discrete Logarithm Problem or Factorization.
There are hundreds if not thousands of encryption algorithms. What you are naming are hard problems that we use to build algorithms such as Diffie-Hellman which uses the DLP, or RSA which relies on the fact that it is hard to factor the modulus (the RSA problem).
When I use any algorithm for encrypting and decrypting images, such as RSA
Nobody uses RSA directly to encrypt images. And modern encryption algorithms operate on bits / bytes, which can represent any kind of "message". Although RSA itself operates on numbers, the PKCS#1 specifications - that define two secure ways of padding for encryption - will require the message to consist of bytes.
how can I estimate this algorithm's security and work efficiently with image encryption.
You can look at keylength.com for recommendations of for the key size and the relative security or strength (also usually shown in bits) that a particular key size can offer.
As for efficiency, usually key encapsulation is used to encrypt a random symmetric data key which can then encrypt the image. This way of combining asymmetric and symmetric algorithms is called a hybrid cryptosystem.
Generally symmetric algorithms are much faster and use less overhead to encrypt data.
