Tomcat cluster in Kubernetes: fetchMembers Failed to open stream

Tomcat cluster in K8s:

Following https://cwiki.apache.org/confluence/display/tomcat/ClusteringCloud

added the following in tomcat server.xml:

<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster">
<Channel className="org.apache.catalina.tribes.group.GroupChannel">
<Membership className="org.apache.catalina.tribes.membership.cloud.CloudMembershipService"/>
</Channel>
</Cluster>

Error log from tomcat POD:

26-May-2023 22:38:50.993 SEVERE [Catalina-utility-1] org.apache.catalina.tribes.membership.cloud.KubernetesMembershipProvider.fetchMembers Failed to open stream
        java.io.IOException: Failed connection to [https://10.96.0.1:443/api/v1/namespaces/tomcat/pods] with token [...]
                at org.apache.catalina.tribes.membership.cloud.TokenStreamProvider.openStream(TokenStreamProvider.java:56)
                at org.apache.catalina.tribes.membership.cloud.KubernetesMembershipProvider.fetchMembers(KubernetesMembershipProvider.java:136)
                at org.apache.catalina.tribes.membership.cloud.CloudMembershipProvider.heartbeat(CloudMembershipProvider.java:127)
                at org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:211)
                at org.apache.catalina.tribes.group.GroupChannel$HeartbeatRunnable.run(GroupChannel.java:847)
                at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
                at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
                at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
                at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
                at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
                at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
                at java.base/java.lang.Thread.run(Thread.java:829)
        Caused by: java.io.IOException: Server returned HTTP response code: 403 for URL: https://10.96.0.1:443/api/v1/namespaces/tomcat/pods
                at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1924)
                at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1520)
                at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:250)
                at org.apache.catalina.tribes.membership.cloud.AbstractStreamProvider.openStream(AbstractStreamProvider.java:117)
                at org.apache.catalina.tribes.membership.cloud.TokenStreamProvider.openStream(TokenStreamProvider.java:53)
                ... 11 more

Shell into the tomcat instance:

sh-4.2# curl -v https://10.96.0.1:443/api/v1/namespaces/tomcat/pods
*   Trying 10.96.0.1:443...
* Connected to 10.96.0.1 (10.96.0.1) port 443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none
* [CONN-0-0][CF-SSL] TLSv1.2 (OUT), TLS header, Certificate Status (22):
* [CONN-0-0][CF-SSL] TLSv1.2 (OUT), TLS handshake, Client hello (1):
* [CONN-0-0][CF-SSL] TLSv1.2 (IN), TLS handshake, Server hello (2):
* [CONN-0-0][CF-SSL] TLSv1.2 (IN), TLS handshake, Certificate (11):
* [CONN-0-0][CF-SSL] TLSv1.2 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html