Setup Jenkins permanent agent over internal network

I'm trying to connect a permanent Jenkins node to a Jenkins master instance inside of the same AWS VPC.

The setup:

• Jenkins master instance inside AWS, in a private subnet
• fronted by an internet load-balancer, restricted to office IPs
• external DNS entry that points to said load balancer
• internal load balancer allowing connections from inside the network
  • internal DNS different from the external one

Because of how Jenkins configures the agent, I cannot get the node to connect to the master over the VPC network - it always tries to go over the internet and then gets blocked by the ACL. Changing the JNLP URL in the agent initialization doesn't work, it still goes to the external address to get the agent port info.

java -jar agent.jar -jnlpUrl http://internal-jenkins-load-balancer.us-west-2.elb.amazonaws.com/computer/test-agent/slave-agent.jnlp -secret ******** -workDir "/home/ubuntu"
Mar 10, 2022 3:34:13 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using /home/ubuntu/remoting as a remoting work directory
Mar 10, 2022 3:34:13 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Both error and output logs will be printed to /home/ubuntu/remoting
Mar 10, 2022 3:34:14 PM hudson.remoting.jnlp.Main createEngine
INFO: Setting up agent: test-agent
Mar 10, 2022 3:34:14 PM hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Jenkins agent is running in headless mode.
Mar 10, 2022 3:34:14 PM hudson.remoting.Engine startEngine
INFO: Using Remoting version: 4.5
Mar 10, 2022 3:34:14 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using /home/ubuntu/remoting as a remoting work directory
Mar 10, 2022 3:34:14 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [https://jenkins-master.example.com/]
SEVERE: Failed to connect to https://jenkins-master.example.com/tcpSlaveAgentListener/: connect timed out

Changing the Jenkins URL parameter to the internal load balancer name works, maybe there's another parameter I can set?

Note: I would like to avoid creating a private hosted zone for this.