I have upgraded the k8s from 1.23.4 to 1.24.13 and also migrated the CRI from docker to crio. I am deploying the ingress controller ver1.7 , pods are not coming up with error permission denied. Any suggestions how to fix this issue.
NGINX Ingress controller
Release: v1.7.0
Build: 72ff21ed9e26cb969052c753633049ba8a87ecf9
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.21.6
---------------------------------------------
W0426 14:40:48.575130 2 client_config.go:618] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0426 14:40:48.575402 2 main.go:209] "Creating API client" host="https://10.96.0.1:443"
I0426 14:40:48.598442 2 main.go:253] "Running in Kubernetes cluster" major="1" minor="24" git="v1.24.13" state="clean" commit="49433308be5b958856b6949df02b716e0a7cf0a3" platform="linux/amd64"
I0426 14:40:48.918593 2 main.go:104] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I0426 14:40:48.949455 2 ssl.go:533] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I0426 14:40:48.964008 2 nginx.go:261] "Starting NGINX Ingress controller"
I0426 14:40:48.967711 2 store.go:524] "ignoring ingressclass as the spec.controller is not the same of this ingress" ingressclass="ingress-nginx-class"
I0426 14:40:48.973448 2 event.go:285] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-controller", UID:"c6f89cbd-4379-4209-8016-69360edf584a", APIVersion:"v1", ResourceVersion:"23095872", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-controller
I0426 14:40:50.073154 2 store.go:429] "Ignoring ingress because of error while validating ingress class" ingress="oaans/oaainstall-oaa-admin-ui" error="no object matching key \"ingress-nginx-class\" in local store"
I0426 14:40:50.073205 2 store.go:429] "Ignoring ingress because of error while validating ingress class" ingress="oaans/oaainstall-risk-admin" error="no object matching key \"ingress-nginx-class\" in local store"
I0426 14:40:50.073221 2 store.go:429] "Ignoring ingress because of error while validating ingress class" ingress="oaans/oaainstall-sms" error="no object matching key \"ingress-nginx-class\" in local store"
I0426 14:40:50.073240 2 store.go:429] "Ignoring ingress because of error while validating ingress class" ingress="oaans/oaainstall-spui" error="no object matching key \"ingress-nginx-class\" in local store"
I0426 14:40:50.073253 2 store.go:429] "Ignoring ingress because of error while validating ingress class" ingress="oaans/oaainstall-yotp" error="no object matching key \"ingress-nginx-class\" in local store"
I0426 14:40:50.073280 2 store.go:429] "Ignoring ingress because of error while validating ingress class" ingress="oaans/oaainstall-email" error="no object matching key \"ingress-nginx-class\" in local store"
I0426 14:40:50.073327 2 store.go:429] "Ignoring ingress because of error while validating ingress class" ingress="oaans/oaainstall-fido" error="no object matching key \"ingress-nginx-class\" in local store"
I0426 14:40:50.073363 2 store.go:429] "Ignoring ingress because of error while validating ingress class" ingress="oaans/oaainstall-oaa-kba" error="no object matching key \"ingress-nginx-class\" in local store"
I0426 14:40:50.073415 2 store.go:429] "Ignoring ingress because of error while validating ingress class" ingress="oaans/oaainstall-oaa-policy" error="no object matching key \"ingress-nginx-class\" in local store"
I0426 14:40:50.073481 2 store.go:429] "Ignoring ingress because of error while validating ingress class" ingress="oaans/oaainstall-oaa-svc" error="no object matching key \"ingress-nginx-class\" in local store"
I0426 14:40:50.073499 2 store.go:429] "Ignoring ingress because of error while validating ingress class" ingress="oaans/oaainstall-push" error="no object matching key \"ingress-nginx-class\" in local store"
I0426 14:40:50.073510 2 store.go:429] "Ignoring ingress because of error while validating ingress class" ingress="oaans/oaainstall-risk-runtime" error="no object matching key \"ingress-nginx-class\" in local store"
I0426 14:40:50.073523 2 store.go:429] "Ignoring ingress because of error while validating ingress class" ingress="oaans/oaainstall-totp" error="no object matching key \"ingress-nginx-class\" in local store"
I0426 14:40:50.165746 2 nginx.go:304] "Starting NGINX process"
I0426 14:40:50.165874 2 leaderelection.go:248] attempting to acquire leader lease ingress-nginx/ingress-nginx-leader...
I0426 14:40:50.166453 2 nginx.go:324] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"
**I0426 14:40:50.166617 2 controller.go:189] "Configuration changes detected, backend reload required"
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)**
I0426 14:40:50.187671 2 leaderelection.go:258] successfully acquired lease ingress-nginx/ingress-nginx-leader
I0426 14:40:50.187763 2 status.go:84] "New leader elected" identity="ingress-nginx-controller-7946bb97cd-lsnps"
I0426 14:40:50.193685 2 status.go:215] "POD is not ready" pod="ingress-nginx/ingress-nginx-controller-7946bb97cd-lsnps" node="XXX.XXX.XXX.XXX"
I0426 14:40:50.302080 2 controller.go:206] "Backend successfully reloaded"
I0426 14:40:50.302175 2 controller.go:217] "Initial sync, sleeping for 1 second"
I0426 14:40:50.302298 2 event.go:285] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-7946bb97cd-lsnps", UID:"0e1fd710-bcf0-4174-b887-818b85babdd3", APIVersion:"v1", ResourceVersion:"23097156", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
2023/04/26 14:40:50 [emerg] 21#21: open() "/var/log/nginx/access.log" failed (13: Permission denied)
W0426 14:40:51.304107 2 controller.go:237] Dynamic reconfiguration failed (retrying; 15 retries left): Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
W0426 14:40:52.333679 2 controller.go:237] Dynamic reconfiguration failed (retrying; 14 retries left): Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
W0426 14:40:53.723856 2 controller.go:237] Dynamic reconfiguration failed (retrying; 13 retries left): Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
W0426 14:40:55.522313 2 controller.go:237] Dynamic reconfiguration failed (retrying; 12 retries left): Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
W0426 14:40:57.864167 2 controller.go:237] Dynamic reconfiguration failed (retrying; 11 retries left): Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
W0426 14:41:00.759655 2 controller.go:237] Dynamic reconfiguration failed (retrying; 10 retries left): Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
W0426 14:41:04.577711 2 controller.go:237] Dynamic reconfiguration failed (retrying; 9 retries left): Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
W0426 14:41:09.843050 2 controller.go:237] Dynamic reconfiguration failed (retrying; 8 retries left): Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
W0426 14:41:16.501643 2 controller.go:237] Dynamic reconfiguration failed (retrying; 7 retries left): Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
W0426 14:41:25.410662 2 controller.go:237] Dynamic reconfiguration failed (retrying; 6 retries left): Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
W0426 14:41:36.552688 2 controller.go:237] Dynamic reconfiguration failed (retrying; 5 retries left): Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
I0426 14:41:47.204812 2 sigterm.go:36] "Received SIGTERM, shutting down"
I0426 14:41:47.204849 2 nginx.go:380] "Shutting down controller queues"
I0426 14:41:47.223830 2 status.go:215] "POD is not ready" pod="ingress-nginx/ingress-nginx-controller-7946bb97cd-lsnps" node="XXX.XXX.XXX.XXX"
I0426 14:41:47.228742 2 status.go:131] "removing value from ingress status" address=[]
I0426 14:41:47.228830 2 nginx.go:388] "Stopping admission controller"
E0426 14:41:47.228962 2 nginx.go:327] "Error listening for TLS connections" err="http: Server closed"
I0426 14:41:47.228999 2 nginx.go:396] "Stopping NGINX process"
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2023/04/26 14:41:47 [warn] 31#31: the "http2_max_field_size" directive is obsolete, use the "large_client_header_buffers" directive instead in /etc/nginx/nginx.conf:145
2023/04/26 14:41:47 [warn] 31#31: the "http2_max_header_size" directive is obsolete, use the "large_client_header_buffers" directive instead in /etc/nginx/nginx.conf:146
2023/04/26 14:41:47 [warn] 31#31: the "http2_max_requests" directive is obsolete, use the "keepalive_requests" directive instead in /etc/nginx/nginx.conf:147
2023/04/26 14:41:47 [notice] 31#31: signal process started
I0426 14:41:48.282863 2 nginx.go:409] "NGINX process has stopped"
I0426 14:41:48.282894 2 sigterm.go:44] Handled quit, delaying controller exit for 10 seconds
E0426 14:41:50.188870 2 queue.go:78] "queue has been shutdown, failed to enqueue" key="&ObjectMeta{Name:sync status,GenerateName:,Namespace:,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[]OwnerReference{},Finalizers:[],ManagedFields:[]ManagedFieldsEntry{},}"
W0426 14:41:50.983743 2 controller.go:237] Dynamic reconfiguration failed (retrying; 4 retries left): Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
I0426 14:41:58.283336 2 sigterm.go:47] "Exiting" code=0
