kubeadm init failing to connect through proxy

I have this version of kubeadm

[root@megatron ~]# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-03T13:44:24Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/amd64"}

My docker is setup and working properly, and can easily and properly pull all the needed images through the proxy I am using.

I have the HTTP proxy configurations across the board in profile and bashrc and environment etc..

When I try to run kubeadm and have it pull images it times out

I0520 10:56:32.085975  260833 version.go:186] fetching Kubernetes version from URL: https://dl.k8s.io/release/stable-1.txt
exit status 1
output: time="2022-05-20T10:59:03-04:00" level=fatal msg="pulling image: rpc error:                                                                                        code = Unknown desc = failed to pull and unpack image \"k8s.gcr.io/kube-apiserver:                                                                                       v1.24.0\": failed to resolve reference \"k8s.gcr.io/kube-apiserver:v1.24.0\": faile                                                                                       d to do request: Head \"https://k8s.gcr.io/v2/kube-apiserver/manifests/v1.24.0\": d                                                                                       ial tcp 172.253.63.82:443: i/o timeout"
, error
k8s.io/kubernetes/cmd/kubeadm/app/util/runtime.(*CRIRuntime).PullImage
        cmd/kubeadm/app/util/runtime/runtime.go:121
k8s.io/kubernetes/cmd/kubeadm/app/cmd.PullControlPlaneImages
        cmd/kubeadm/app/cmd/config.go:340
k8s.io/kubernetes/cmd/kubeadm/app/cmd.newCmdConfigImagesPull.func1
        cmd/kubeadm/app/cmd/config.go:312
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute
        vendor/github.com/spf13/cobra/command.go:856
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC
        vendor/github.com/spf13/cobra/command.go:974
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute
        vendor/github.com/spf13/cobra/command.go:902
k8s.io/kubernetes/cmd/kubeadm/app.Run
        cmd/kubeadm/app/kubeadm.go:50
main.main
        cmd/kubeadm/kubeadm.go:25
runtime.main
        /usr/local/go/src/runtime/proc.go:250
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1571
failed to pull image "k8s.gcr.io/kube-apiserver:v1.24.0"
k8s.io/kubernetes/cmd/kubeadm/app/cmd.PullControlPlaneImages
        cmd/kubeadm/app/cmd/config.go:341
k8s.io/kubernetes/cmd/kubeadm/app/cmd.newCmdConfigImagesPull.func1
        cmd/kubeadm/app/cmd/config.go:312
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute
        vendor/github.com/spf13/cobra/command.go:856
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC
        vendor/github.com/spf13/cobra/command.go:974
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute
        vendor/github.com/spf13/cobra/command.go:902
k8s.io/kubernetes/cmd/kubeadm/app.Run
        cmd/kubeadm/app/kubeadm.go:50
main.main
        cmd/kubeadm/kubeadm.go:25
runtime.main
        /usr/local/go/src/runtime/proc.go:250
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1571

I have even manually pulled the necessary images

[root@megatron ~]# kubeadm config images list
k8s.gcr.io/kube-apiserver:v1.24.0
k8s.gcr.io/kube-controller-manager:v1.24.0
k8s.gcr.io/kube-scheduler:v1.24.0
k8s.gcr.io/kube-proxy:v1.24.0
k8s.gcr.io/pause:3.7
k8s.gcr.io/etcd:3.5.3-0
k8s.gcr.io/coredns/coredns:v1.8.6
[root@megatron ~]# docker pull k8s.gcr.io/coredns/coredns:v1.8.6
v1.8.6: Pulling from coredns/coredns
Digest: sha256:5b6ec0d6de9baaf3e92d0f66cd96a25b9edbce8716f5f15dcd1a616b3abd590e
Status: Image is up to date for k8s.gcr.io/coredns/coredns:v1.8.6
k8s.gcr.io/coredns/coredns:v1.8.6
[root@megatron ~]#

I need help understanding why kubeadm is not using the proper http proxy which seems to be the case when trying to get https://dl.k8s.io/release/stable-1.txt

There are no problems getting that file, why isn't kubeadm getting it?

[root@starscream ~]# wget https://dl.k8s.io/release/stable-1.txt
--2022-05-20 11:09:46--  https://dl.k8s.io/release/stable-1.txt
Connecting to [PROXY]:8080... connected.
Proxy request sent, awaiting response... 302 Moved Temporarily
Location: https://storage.googleapis.com/kubernetes-release/release/stable-1.txt [following]
--2022-05-20 11:09:47--  https://storage.googleapis.com/kubernetes-release/release/stable-1.txt
Connecting to [PROXY]:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 7 [text/plain]
Saving to: ‘stable-1.txt’

stable-1.txt                               100%[======================================================================================>]       7  --.-KB/s    in 0s

2022-05-20 11:09:48 (331 KB/s) - ‘stable-1.txt’ saved [7/7]

Update:

After looking at the forced version option I tried that

kubeadm config images pull --kubernetes-version 1.24.0 --v=5

Now it doesn't try to retrieve the stable-1.txt. I suspect I missed the fact that it may have been able to retrieve it regardless.

Now it is STILL trying to pull images that docker already has. Why is kubeadm trying to pull images that already exist?

I0520 12:57:24.402436  228499 kubelet.go:214] the value of KubeletConfiguration.cgroupDriver is empty; setting it to "systemd"
exit status 1
output: time="2022-05-20T12:59:54-04:00" level=fatal msg="pulling image: rpc error: code = Unknown desc = failed to pull and unpack image \"k8s.gcr.io/kube-apiserver:v1.24.0\": failed to resolve reference \"k8s.gcr.io/kube-apiserver:v1.24.0\": failed to do request: Head \"https://k8s.gcr.io/v2/kube-apiserver/manifests/v1.24.0\": dial tcp 142.250.145.82:443: i/o timeout"
, error

It still doesn't help that the proxy is not being used.I can accept that but why isn't kubeadm using the existing images?