Your step 1 is correct and pretty much everything else is wrong, although it depends some on what your ill-defined "RSA" box includes.

The general structure for all signature (JWS) methods used for JWT is in rfc7515 and is in brief:

concatenate base64url(header), a period, and base64url(payload)

apply the selected signature algorithm to #1 treated as a byte* string, producing (at least for currently used algorithms) a byte string

base64url #2 (not the hex of #2), and for the Compact serialization (which is what people mostly use), effectively concatenate #1, another period, and base64url(#2)

* Formally standards such as RFCs call these 'octet' strings, meaning exactly 8 bits, because 'byte' historically has not always been 8 bits. However, all modern systems encountered by most programmers and users do have 8-bit bytes, and in practice now we treat bytes and octets as the same thing, so I use the more familiar term.

For the "RSnnn" signature algorithms, rfc 7518 section 3 simply references PKCS#1 rfc3447 which itself uses/references a padding method; combining these the signing operation is:

hash the input (for JWS from #1 above, treated as a byte string) with the selected hash (for RS256 this is SHA-256 from FIPS180-2 or later; 7518 references 180-4)

encode the identity of the hash algorithm (SHA-256) plus the hash value (#1) in an ASN.1 structure DigestInfo

prefix (pad) the DigestInfo (#2) with 00 01 FF... 00 to match the size of the RSA key's modulus

convert #3 to an integer, perform the 'raw' or 'naive' RSA primitive (RSASP1) equivalent to $r ^ d \bmod n$, and convert to a byte string (for JWS/JWT used in #3 above)

The/an RSA privatekey is *semantically* an integer $d$ combined with an integer $n$ but in practice usually stored with added "CRT" (Chinese Remainder Theorem) parameters allowing a more efficient implementation -- see all of rfc3447, or wikipedia and details of this storage and implementation vary.

(You may notice on the rfc-editor site that rfc3447 PKCS#1 v2.1 has been superseded by rfc8017 PKCS#1 v2.2; however, this area hasn't changed so it doesn't matter which you read. In fact the PKCS1-v1_5 operations in v2.2, v2.1, and v2.0 are substantively the same as 'block type 1' in rfc2313 PKCS#1 v1.5, but expressed differently.)