I'm a newbie that is studying about JWT using RS256 algorithm for signing and verification. I have drawn a diagram that represents my understanding of how RS256 works.

Basically, below is what I have by far:

1. header and payload after being trimmed are both base64url encoded
2. base64url encoded header and payload are hashed using SHA256
3. 2 resulting hashes from above will be joined by a dot "." to produce M -> M = "hash_header.hash_payload"
4. M (string) and a RSA generated private key will all be converted to binary form
5. RSA receive message M (binary) and private key (binary) as input and produce an output C (binary)
6. C (binary) is converted to hex and then base64url encoded to produce the final signature for the JWT token

Am I getting something wrong? I'm all ears for any comments!

Update 1

After having thoroughly read through the answer provided, I still couldn't grasp the overall picture of how RS256 should work. I appreciate the details but for a beginner like me without many resources to learn from, I just want to first understand the gist of the algorithm. Below is my revised diagram for how to create a JWT token and what I have got so far:

1. Generate public and private key pair with OpenSSL for example
2. Signature is generated using RS256 with BASE64URL(HEADER), BASE64URL(PAYLOAD) and RSA Private Key
3. Public Key is included in Protected Header
4. JWT in JWS format is formed by BASE64URL(UTF8(JWS Protected Header)) concat "." concat BASE64URL(JWS Payload) concat "." concat BASE64URL(JWS Signature)

Is this the correct interpretation? Much appreciation for your insight!

Update 2

Thanks to dave_thompson_085, I have modified the diagram and the result is as follows:

1. Generate public and private key pair with OpenSSL for example
2. Signature is generated using RS256 with BASE64URL(HEADER) period BASE64URL(PAYLOAD) period RSA Private Key
3. Public Key ID (kid) is included in Protected Header
4. JWT in JWS format is formed by BASE64URL(UTF8(JWS Protected Header)) concat "." concat BASE64URL(JWS Payload) concat "." concat BASE64URL(JWS Signature)

Your step 1 is correct and pretty much everything else is wrong, although it depends some on what your ill-defined "RSA" box includes.

The general structure for all signature (JWS) methods used for JWT is in rfc7515 and is in brief:

1. concatenate base64url(header), a period, and base64url(payload)

2. apply the selected signature algorithm to #1 treated as a byte* string, producing (at least for currently used algorithms) a byte string

3. base64url #2 (not the hex of #2), and for the Compact serialization (which is what people mostly use), effectively concatenate #1, another period, and base64url(#2)

* Formally standards such as RFCs call these 'octet' strings, meaning exactly 8 bits, because 'byte' historically has not always been 8 bits. However, all modern systems encountered by most programmers and users do have 8-bit bytes, and in practice now we treat bytes and octets as the same thing, so I use the more familiar term.

For the "RSnnn" signature algorithms, rfc 7518 section 3 simply references PKCS#1 rfc3447 which itself uses/references a padding method; combining these the signing operation is:

1. hash the input (for JWS from #1 above, treated as a byte string) with the selected hash (for RS256 this is SHA-256 from FIPS180-2 or later; 7518 references 180-4)

2. encode the identity of the hash algorithm (SHA-256) plus the hash value (#1) in an ASN.1 structure DigestInfo

3. prefix (pad) the DigestInfo (#2) with 00 01 FF... 00 to match the size of the RSA key's modulus

4. convert #3 to an integer, perform the 'raw' or 'naive' RSA primitive (RSASP1) equivalent to $r ^ d \bmod n$, and convert to a byte string (for JWS/JWT used in #3 above)

The/an RSA privatekey is semantically an integer $d$ combined with an integer $n$ but in practice usually stored with added "CRT" (Chinese Remainder Theorem) parameters allowing a more efficient implementation -- see all of rfc3447, or wikipedia and details of this storage and implementation vary.

(You may notice on the rfc-editor site that rfc3447 PKCS#1 v2.1 has been superseded by rfc8017 PKCS#1 v2.2; however, this area hasn't changed so it doesn't matter which you read. In fact the PKCS1-v1_5 operations in v2.2, v2.1, and v2.0 are substantively the same as 'block type 1' in rfc2313 PKCS#1 v1.5, but expressed differently.)