OpenSSL Error: lib(128):capi_rsa_priv_enc:function not supported in client Auth

My scripts to sign file via API was working properly fine when my previous server setup was Ubuntu 20.04 and openssl version is 1.1.1b.

But after upgrade, I am getting this issue. Client environment is same before and after the server upgrade.

Now
Current Server Env :

    Ubuntu 22..04  
    Openssl 3.0.2  
    Apache 2.4.52  
    SSL x509 certificate is installed on the Apache Server 

Client Env:

    Windows 2016  
    Curl 7.78.0
    openssl 1.1.1f

Below command has been executed to sign a file via web API running on Server over https.

curl.exe --request POST --cacert "ca.crt" --cert "user.crt" --engine capi --key-type ENG --key "user.key" --verbose -L "https://webapi:port" --header "content-type: multipart/form-data" --form "file=@file.txt" --output "out_file.txt"

tailed output:

...
..

[5 bytes data]

• TLSv1.2 (IN), TLS handshake, Hello request (0): { [4 bytes data]
• TLSv1.2 (OUT), TLS handshake, Client hello (1): } [232 bytes data]
• TLSv1.2 (IN), TLS handshake, Server hello (2): { [117 bytes data]
• TLSv1.2 (IN), TLS handshake, Certificate (11): { [1319 bytes data]
• TLSv1.2 (IN), TLS handshake, Server key exchange (12): { [556 bytes data]
• TLSv1.2 (IN), TLS handshake, Request CERT (13): { [1979 bytes data]
• TLSv1.2 (IN), TLS handshake, Server finished (14): { [4 bytes data]
• TLSv1.2 (OUT), TLS handshake, Certificate (11): } [4292 bytes data]
• TLSv1.2 (OUT), TLS handshake, Client key exchange (16): } [37 bytes data]
• TLSv1.2 (OUT), TLS alert, internal error (592): } [2 bytes data]
• OpenSSL SSL_read: error:8006F074:lib(128):capi_rsa_priv_enc:function not supported, errno 0 0 82.4M 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
• Closing connection 0 curl: (56) OpenSSL SSL_read: error:8006F074:lib(128):capi_rsa_priv_enc:function not supported, errno 0

I tried to disable tlsv1.2 and tlsv1.3 on server ssl.conf but still issue remains the same.

My understanding on openssl and x509 certificate troubleshooting is limited.

The issue seems to be related to CAPI engine use. My suspicion is that the OpenSSL version that is being used does not support the "capi_rsa_priv_enc" function Openssl 1.1.1f capi_rsa_priv_enc link.

And it seems that curl is compiled with an older version of OpenSSL that does support the function.

Check what version curl uses:

curl --version

Try to recompile the curl with OpenSSL 1.1.1f on the Windows client and check if the error is fixed.

EDIT:

OpenSSL with CAPI and Mutual Auth at client side will not work if the OpenSSL server version is >=1.1.1. TLS 1.2 won't work on Windows

The older CAPI engine won't support RSA-PSS for both TLS 1.2 and 1.3. And as per your server configuration, it uses OpenSSL-3, which will cause negotiation to fail.

Also, the OpenSSL-3 version won't support the CAPI engine anyway, so my recommendation is to upgrade the client system to use the next-gen Windows Crypto API for accessing the WebAPI.

You can also look into openssl-cng-engine