Some exchanges use the signature generated by hashing the payload with the private key.
About right: typically the a signature of a payload is generated by hashing the payload, then processing the hash and the private key to form the signature. The signature can then be verified against the public key (assumed authentic) and payload. The verification is by hashing the payload, then processing the hash, signature and public key to form a verification result: true if the payload, signature and public key are unaltered; and (hopefully) false otherwise. Hashing is considered an integral part of signing and verifying a payload.
A public/private key pair is generated for ciphering, the payload is encrypted with the private key by the sender and it's decrypted with the public key by the receiver to ensure the authenticity.
Uh, "payload encrypted with the private key" and "decrypted with the public key" are incorrect terminology: encryption aims at making it's payload confidential, and since the public key is known to all, and (in the procedure outlined) assumed able to undo what was done with the the private key, the goal of encryption wouldn't be met in the procedure outlined. It should be said "sign with the private key", "verify with the public key".
Secure RSA signature used in practice can be constructed from textbook RSA encryption about by the procedure outlined. But not all asymmetric encryption schemes can be turned into signature schemes in this way, and many widely used signature schemes are not constructed in this way.
A hash is supposed to not be reversible, so it can't be decrypted, only regenerated using the same key.
A hash does not involve a key. It's not encryption, and finding the input of a hash from it's output would not be finding the "decrypted" input: that's a first preimage attack. Such attack is totally possible if the input/payload is from a small enough set (e.g. the amount of a transaction). We do NOT need a hash to be resistant to first preimage attack to make it usable in signature.
The plaintext public key is used to identify the sender, and the signature to ensure that the payload as been sent by them. But if the exchange doesn't know the private key of the sender it won't be able to regenerate the signature.
Correct, if we read "exchange" as receiver.
If the signature is generated with the private key, the exchange has to know the private key of the sender, thus defeating the point of the private key being private.
Yes the signature is generated with the private key. The receiver verifies the signature using the sender's public key, assumed authentic. It does not need to know the private key, which indeed would defeat the purpose of signature.
If the signature is generated with the public key…
It's not. A signature is generated with a private key, and verified with the matching public key.
Notice that verifying the signature only proves that the payload has been signed by the holder of the private key at some moment. To prove that the holder of the private key really participated in the current exchange, we need something extra; like the payload contains a challenge that was just generated by the receiver.
How is the signature verified using the public key?
Let's take the example of RSASSA-PKCS1-v1_5 signature, which is close to what the question has in mind. For RSA private key $(n,d)$, and hash $H$ (e.g. SHA-256), the signature of payload $M$ is defined as $S=(K\mathbin\|H(M))^d\bmod n$ where we assimilate bitstrings and integers, and $K$ is a public bitstring (specified in EMSA-PKCS1-v1_5) depending only on the size of $n$ and such that the integer $K\mathbin\|H(M)$ is below, but close enough to $n$.
To verify signature $S$ against payload $M$ and the public key $(n,e)$ matching private key $(n,d)$, after checking that $0\le S<n$, we compute $S^e\bmod n$, compute $K\mathbin\|H(M)$, and compare the two. They match if $S$, $M$ and public key $(n,e)$ are genuine. Proof of that is easy: by construction of $(n,e,d)$ it holds $\forall X\in[0,n),\ {(X^d)}^e\bmod n=X$. We apply this with $X=K\mathbin\|H(M)$.
Notice that this verification process did not use the private key $(n,d)$.
Without proof (we don't know one), it's hard to come up with $(S',M')$ that passes this verification other than by copying an existing $(S,M)$, or using $(n,d)$ or/and an equivalent to the factorization of $n$.
Other signature schemes work quite differently; e.g. Schnoor signature.
