TLDR: Is there a way to change ADH from ADH:@STRENGTH to ADH:@SECLEVEL=0 either by using a config file (supported on x11vnc), or by adding some commands on the terminal line I use to start VNC (see below)?
I'm trying to have a working encrypted connection to my server through vnc. I'm using x11vnc 0.9.16. I am running Ubuntu 21.04.
When I try running it through that command:
sudo /usr/bin/x11vnc -auth guess -forever -loop -noxdamage -repeat -rfbauth /home/fairbanks/.vnc/passwd -rfbport 5903 -shared -ssl
I am getting this error:
27/02/2022 03:38:12 SSL: accept_openssl(OPENSSL_VNC)
27/02/2022 03:38:12 SSL: spawning helper process to handle: 192.168.3.30:51494
27/02/2022 03:38:12 SSL: helper for peerport 51494 is pid 19343:
27/02/2022 03:38:12 connect_tcp: trying: 127.0.0.1 20000
27/02/2022 03:38:13 check_vnc_tls_mode: waited: 1.418020 / 1.40 input: (future) RFB Handshake
27/02/2022 03:38:14 check_vnc_tls_mode: version: 3.8
27/02/2022 03:38:14 SSL: ssl_helper[19343]: exit case 2 (ssl_init failed)
27/02/2022 03:38:14 SSL: accept_openssl: cookie from ssl_helper[19343] FAILED.
in vnc viewer on windows, when I try to connect, it throws me some sort of security level error.
From the research I made, it seems the -ssl option, uses openssl to work, my openssl version is 1.1.11. From what I've googled, it has to do with a default option that conflicts. it seems somewhere, I have to make a change from ADH:@STRENGTH to ADH:@SECLEVEL=0. As it can be seen in those links:
https://github.com/vishnumotghare/x11vnc/blob/master/src/sslhelper.c
https://githubmemory.com/repo/LibVNC/x11vnc/issues/129
But I can't, for the life of me, find out where to do that.
from the first link, I think it might be needed to be done when compiling the x11vnc. And I don't have a clue on how to go about that, because even if I achieve the goal of compiling it, doing so on my own might get me a lot of new errors as I'm far from knowing what I'm doing in that situation.
I also found out here:
https://manpages.ubuntu.com/manpages/bionic/man1/x11vnc.1.html
that: "Config file support: if the file $HOME/.x11vncrc exists then each line in it is treated as a single command line option."
so I'm wondering if there is a way to make that change on ADH either by using that config file support, or by adding some commands on the terminal line I use to start VNC. If so, how would I go about it?
