Score:0

How can I know request waiting time in Nginx Ingress Controller?

th flag

We use Kubernetes with Nginx Ingress Controller to run our platform with various backend services. We also use New Relic (& Prometheus, Grafana) for our Observability dashboards & alerts. Nginx Ingress Controller is the entrypoint for all our requests.

As we keep getting requests, if all threads of our Backend services are busy, then I am assuming that the requests start getting queued up and they get eventually serviced as the threads / resources start freeing up.

Currently, I am unable to find a metric that will give me the amount of time requests are waiting before being picked up by Nginx and eventually being serviced by our Backend.

Nginx Ingress Controller Prometheus metrics contain the following 4 latency metrics:

  • nginx_ingress_controller_request_duration_seconds (nginx var: request_time)
  • nginx_ingress_controller_response_duration_seconds (nginx var: upstream_response_time)
  • nginx_ingress_controller_connect_duration_seconds (nginx var: upstream_connect_time)
  • nginx_ingress_controller_header_duration_seconds (nginx var: upstream_header_time)

On plotting these metrics and trying various load tests, I have figured out that all of these metrics are giving me the latency of requests once they are picked up by Nginx but none of them seem to include the waiting time of requests before being picked up by Nginx.

I am looking for answers to the following questions:

  • Does upstream_connect_time include the waiting time of requests (before being picked up by Nginx)?
  • Does any of the above 4 metrics include the waiting time of requests (before being picked up by Nginx)?
  • I have found a variable called upstream_queue_time in ngx_http_upstream_module mentioned here. Does this metric give me waiting time of requests (before being picked up by Nginx)? If so, is this available only for Nginx Plus and which metric can I use?
  • Is there any other way I can get insight into the waiting time of requests (before being picked up by Nginx)?

Please let me know if my above understanding is incorrect.

You time and expertise in answering this is really appreciated. Thanks a lot.

I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.