Authentication-: I understand that authentication is basically digital signature.

You can **use** a digital signature for authentication, be it entity authentication (e.g. in the TLS protocol) or message authentication (e.g. in the PGP protocol). It is however also possible to use other means, e.g. a MAC if you share a secret key: the digital signature is a means to an end.

In RSA encryption, we use public key(of whose sender or receiver?) for encryption and private key(of whom?) for decryption.

You'd use the public key of the receiver. You first need to establish trust in the public key though. This is why you can e.g. sign keys in PGP so that you can utilize it's *web of trust*.

They say hash is encrypted using RSA. But why are we using PRIVATE KEY (of sender) here (instead of public key) and public key of sender instead of private key?

The private key of the sender used in signature generation. It is possible to show that that the private key of the sender is used by verifying the signature. The hash over the data is used so that the *integrity and authenticity of the message* is maintained.

It's best *not* to think of that as encryption with a private key, so they got that wrong. Both signature generation and encryption in RSA depend on modular exponentiation. However, that's where the similarities end. For more information see my self answered question here.

Note that *two* key pairs are used if you want to encrypt and sign. The encryption part is performed using the key pair of the receiver, while the signature generation is performed using the key pair of the sender. Encryption is always performed using the public key, decryption with the private key. For signature generation the private key is used, for verification the public key.

Often a data or session key is encrypted instead of the message directly. That's just because symmetric encryption is more efficient (not just in compute time but also the resulting ciphertext size). Similarly usually you sign the hash instead of the message - although there are also some security related reasons for that.