Score:2

Argocd Ingress not working on a Ubuntu Microk8s cluster

is flag

I have a working Microk8s cluster. After enabling the argocd community addon, the recommended ingress for argocd sever doesn't seem to be working.

Here's a note I got after enabling the addon:

Infer repository community for addon argocd
Infer repository core for addon helm3
Addon core/helm3 is already enabled
Installing ArgoCD (Helm v4.6.3)
"argo" already exists with the same configuration, skipping
Release "argo-cd" does not exist. Installing it now.
NAME: argo-cd
LAST DEPLOYED: Thu Oct 20 17:34:33 2022
NAMESPACE: argocd
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
In order to access the server UI you have the following options:

1. kubectl port-forward service/argo-cd-argocd-server -n argocd 8080:443

    and then open the browser on http://localhost:8080 and accept the certificate

2. enable ingress in the values file `server.ingress.enabled` and either
      - Add the annotation for ssl passthrough: https://github.com/argoproj/argo-cd/blob/master/docs/operator-manual/ingress.md#option-1-ssl-passthrough
      - Add the `--insecure` flag to `server.extraArgs` in the values file and terminate SSL at your ingress: https://github.com/argoproj/argo-cd/blob/master/docs/operator-manual/ingress.md#option-2-multiple-ingress-objects-and-hosts


After reaching the UI the first time you can login with username: admin and the random password generated during the installation. You can find the password by running:

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

(You should delete the initial secret afterwards as suggested by the Getting Started Guide: https://github.com/argoproj/argo-cd/blob/master/docs/getting_started.md#4-login-using-the-cli)
ArgoCD is installed

Additionally, here's the Ingress that I defined:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argocd-server-ingress
  namespace: argocd
  annotations:
    cert-manager.io/cluster-issuer: lets-encrypt
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    # If you encounter a redirect loop or are getting a 307 response code
    # then you need to force the nginx ingress to connect to the backend using HTTPS.
    #
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
  rules:
  - host: argocd.DOMAIN.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: argo-cd-argocd-server
            port:
              name: https
  tls:
  - hosts:
    - argocd.DOMAIN.com
    secretName: argocd-secret # do not change, this is provided by Argo CD

When I visit the host address, I get this:

enter image description here

From firefox, here's the certificate being issued.

enter image description here

enter image description here

From the notes I got after enabling this addon, how do I do this part enable ingress in the values file server.ingress.enabled this or get my ingress to correctly work?

UPDATE:

Here's what the ingress description looks like: enter image description here

dummyuser avatar
uy flag
is the cert-manager installed and properly configured for lets-encrypt?
Mysterio avatar
is flag
@dummyuser yes. It's configured and the same issuer is working well with other ingresses.
dummyuser avatar
uy flag
Please do the following command: `curl -v https://<ARGOCD.URL>` (optionally with -k ) check the details of the certificate which is presented by argocd. Does it match your expectations or is it a selfsigned? Use alternativly Firefox, which allows to have a look at the certificate.
Mysterio avatar
is flag
@dummyuser updated the question with images from firefox. It seems "enable ingress in the values file `server.ingress.enabled`" thingy has to be done to terminate the internal certificate from the addon before the lets-encrypt issuer can do its thing
dummyuser avatar
uy flag
so ... fist finding ... ingress seems to present a selfsinged Cert which is not trusted by your fist Browser. I'm not that much in nginx ingress, but the lines `kubernetes.io/tls-acme: "true"` and `nginx.ingress.kubernetes.io/ssl-passthrough: "true"` together do not really make sense to me. I guess the first one can be false. pls check Documentation ( e.g. https://cert-manager.io/docs/usage/ingress/). What happens if you click `Accept the Risk and continue`?
Mysterio avatar
is flag
@dummyuser I just get an nginx 404 Not Found error.
dummyuser avatar
uy flag
is there a service named `argo-cd-argocd-server` up in the NS argocd and listens to 443? and additionally is there a POD up and running connected to the Service ?
dummyuser avatar
uy flag
the line `kubernetes.io/tls-acme: "true"` should be removed from ingress.
dummyuser avatar
uy flag
I guess the servicename in the ingress should be `argocd-server` instead of `argo-cd-argocd-server`
Mysterio avatar
is flag
@dummyuser check the løg info. Service name is the one specified and the service seem to be working as expected
dummyuser avatar
uy flag
souds like you have a big argocd playground now. I'll write an official answer
Score:1
uy flag

The Error message did indicate an invalid Certificate which was not accepted by the browser. The ingess configuration requested a certificate and ssl-passthrough which does not fit together. The line kubernetes.io/tls-acme: "true" had to be removed from Ingress and a minor change of the service name.

The TLS connection to ArgoCD is terminated at the ArgoCD Server and not at the ingress GW. ArgoCD uses the Certificate (and Priv Key) stored in the argocd-secret. ingess should look like

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argocd-server-ingress
  namespace: argocd
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
  rules:
  - host: argocd.<domain>
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service: 
            name: argocd-server
            port:
              name: https
  tls:
  - hosts:
    - argocd.<url>
    secretName: argocd-secret # not relevant

and a kubectl get services must contain a line like

argocd-server     ClusterIP   10.99.19.178   <none>        80/TCP,443/TCP               179d

a kubectl get Ingress -n argocdresults in

NAME                    CLASS    HOSTS                 ADDRESS         PORTS     AGE
argocd-server-ingress   <none>   argocd.k3sxx.xx   192.168.xx.xx   80, 443   15m

and details can be seen with kubectl get Ingress -n argocd -o yaml

Mysterio avatar
is flag
Hmmm so, I deleted addon and argocd namespace. I installed argocd via the official install guide (https://argo-cd.readthedocs.io/en/stable/getting_started/#1-install-argo-cd) and removed `kubernetes.io/tls-acme: "true"` from the ingress setup and service name set to `argocd-server` BUT I now get a 404 Not Found nginx error in the browser when I got to the URL generated.
Mysterio avatar
is flag
The ingress description shows that it's correctly hooked into the `argocd-server` service.
dummyuser avatar
uy flag
please do a `curl -v -k https://argocd-server.argocd.svc.cluster.local` from any other POD in your cluster (does not work frm argocd NS, as these PODs do not have curl). if you get a http 200 Argos cd is set up properly and there is a ingress issue. I copied my ingress file to my answer.
Mysterio avatar
is flag
I got an HTTP 200 alright. So argocd is working just fine. It's the ingress which for some reason is fighting over certificates.
Mysterio avatar
is flag
The only unexplored option is the one from the logs to enable ingress via values file: "enable ingress in the values file `server.ingress.enabled`". Do u have an idea of how to generate and pass this value file?
Mysterio avatar
is flag
This response could also be a reason for this error: https://stackoverflow.com/a/67263155/8194007
dummyuser avatar
uy flag
the link just shows how to install argocd without TLS. the option `server.ingress.enabled` defines if the ingress ruel should be enabled at chart installation , then you need to add some parameter to the value file anyway. to me it is a bit easier to deploy the ingess rule separatly. may you update the `ingress file` in your question and show the output of `kubectl get Ingress -n argocd` (should look like `argocd-server-ingress <none> argocd.k3s.xxx.xxx 192.168.xx.xx 80, 443 2m8s` )
Mysterio avatar
is flag
updated the question with the description of the ingress as well as the `get` command output.
dummyuser avatar
uy flag
is `server.ingress.enabled` set to false now and ingress is deployed separately ? looks like you still get a certificate from the ingress controller before the 404
Mysterio avatar
is flag
@dummuser, where do I check this? the deployment? or service?
dummyuser avatar
uy flag
lets do a quick summery: you installed argo using the file `https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml`, not the helm now? This file does not contain ingress definitions. If so values of the hlemchart (server.ingress.enabledare out of scope) Argo does work, checked with curl, you use an ingress-nginx. Right now, you get a certificate from ingress if trying to access argocd an afterwards a 404. I have a nginx-ingess config with passthrough Ingress running with the config in my answer.
dummyuser avatar
uy flag
Looks like there is anything in the nginx-ingress / Ingress definition which I cannot find remotely. Do you have other (working) ingress definitions on this Server ?
Mysterio avatar
is flag
yes, I have 4 other ingress definitions working fine using the same certificate issued. The only difference is this service port points to HTTPS instead of some service port number.
dummyuser avatar
uy flag
ok in this case, add the `--insecure` option to the argocd deployment and use one of the other ingress as template - means we move tls termination from Argo to the ingress gw.
Mysterio avatar
is flag
Which of the command should get this `--insecure` option? Not that well seasoned with the world of Kubernetes as you may have noticed.
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.