We use Kubernetes with Nginx Ingress Controller to run our platform with various backend services. We also use New Relic (& Prometheus, Grafana) for our Observability dashboards & alerts. Nginx Ingress Controller is the entrypoint for all our requests.
As we keep getting requests, if all threads of our Backend services are busy, then I am assuming that the requests start getting queued up and they get eventually serviced as the threads / resources start freeing up.
Currently, I am unable to find a metric that will give me the amount of time requests are waiting before being picked up by Nginx and eventually being serviced by our Backend.
Nginx Ingress Controller Prometheus metrics contain the following 4 latency metrics:
nginx_ingress_controller_request_duration_seconds
(nginx var:
request_time
)
nginx_ingress_controller_response_duration_seconds
(nginx var: upstream_response_time
)
nginx_ingress_controller_connect_duration_seconds
(nginx var:
upstream_connect_time
)
nginx_ingress_controller_header_duration_seconds
(nginx var:
upstream_header_time
)
On plotting these metrics and trying various load tests, I have figured out that all of these metrics are giving me the latency of requests once they are picked up by Nginx but none of them seem to include the waiting time of requests before being picked up by Nginx.
I am looking for answers to the following questions:
- Does
upstream_connect_time
include the waiting time of requests (before being picked up by Nginx)?
- Does any of the above 4 metrics include the waiting time of requests (before being picked up by Nginx)?
- I have found a variable called
upstream_queue_time
in ngx_http_upstream_module
mentioned here. Does this metric give me waiting time of requests (before being picked up by Nginx)? If so, is this available only for Nginx Plus and which metric can I use?
- Is there any other way I can get insight into the waiting time of requests (before being picked up by Nginx)?
Please let me know if my above understanding is incorrect.
You time and expertise in answering this is really appreciated. Thanks a lot.