Must a server hash (a) all own sent messages, enumerate in the section 4.4.1, or (b) own sent and received from client ?
The transcript hash (or hashes) is the hash of all messages exchanged so far. So, the server and the client compute these transcript hashes in the prescribed way. Loosely speaking, the transcript hash is used to "authenticate" the key exchange protocol. Therefore, a transcript hash based solely on the server's message could be highly problematic since it does not help against any potential shenanigans an attack may mount against a client. There are some expectations when dealing with HelloRetryRequest
messages.
Must a server send the calculated transcript hash in any message separately or the hash is used only while deriving a keys using HKDF-Extract/Expand procedures ?
The transcript hash is never sent explicitly, since it should be computed from what each party believes to be the conversation that took place. The transcript hash is used during authentication and key derivation. If an attack occurs, the transcript hash is likely to differ; hence the signature that the server sends in CertificateVerify
is computed on a transcript hash that is different from what the client has, so certificate validation is likely to fail here. However, this is not enough for authentication: without going into the details, authentication also requires a MAC on the transcript hash (after the signature). As you noted, the transcript hash goes into the key derivation process; therefore the MAC sent by the server (or the client later) in the Finished
message is unlikely to match what the client (server) expects.