Join Log in
Score:0
cyberrspiritt avatar Ubuntu

Temporary failure in Name Resolution PHP-Ubuntu

in flag
cyberrspiritt

I keep getting this error very frequent:

Severity: Warning --> mysqli::real_connect(): php_network_getaddresses: getaddrinfo failed: Temporary failure in name resolution

This issue gets resolved only after Server restart. Considering this happens over a live server having millions of requests daily, restarting isn't a best option.

[EDITED] I'm using Ubuntu 18.04 and connecting to an Aurora Cluster on AWS and the backend code resides on EC2

506
0 + 0
php
aws
in flag
matigo
There is an insufficient amount of information here to provide any useful suggestions beyond generic tips that can be found on the first page of Google. Could you [edit] your question to include: (1) the version of Ubuntu you're using (2) the hostname that you're using to connect to MySQL
0
Reply
cyberrspiritt avatar
in flag
cyberrspiritt
updated @matigo
0
Reply
Score:2
avatar Ubuntu
in flag
matigo

Taking into consideration the following details:

  1. millions of requests daily
  2. Aurora Cluster on AWS

You may want to take a look at your system to make sure you're not exceeding the DNS quota for your account.

One of the stand-out items from the quota documentation is this:

Each Amazon EC2 instance limits the number of packets that can be sent to the Amazon-provided DNS server to a maximum of 1024 packets per second per network interface. This quota cannot be increased. The number of DNS queries per second supported by the Amazon-provided DNS server varies by the type of query, the size of response, and the protocol in use. For more information and recommendations for a scalable DNS architecture, see the Hybrid Cloud DNS Solutions for Amazon VPC whitepaper.

If you reach the quota, the Amazon Route 53 Resolver rejects traffic [...]

Note: Emphasis mine.

The "maximum of 1024 packets per second" bit is important because the actual number of packets per query can vary and there are typically multiple packets per DNS query.

If your server(s) are receiving millions of requests per day, then there is a high possibility that your server(s) are hitting that packet maximum:

  • 1,000,000 requests / 86,400 seconds = 11.574 Requests per second
  • 11.574 * 4 Packets¹ = 46-ish packets per second
  • 1024 / 46 = 22.26-ish DNS calls per second

I cannot say this is definitely the problem, but this is a good place to start looking, particularly if your web servers have regular rush periods where traffic is not operating at a nice, flat average.

¹ Having been bitten by this problem in the past, I've measured that many DNS requests require an average of 4 packets per call

0 + 0
cyberrspiritt avatar
in flag
cyberrspiritt
Thanks for your response. This I'm aware of, If I'm hitting the max requests per second, how do i resolve it? Usually restarting the server resolves it. What are the workarounds to avoid that limit?
0
Reply
in flag
matigo
I’ve not found a good, HA-friendly way to resolve the issue. I’ve done things like hybrid DNS and whatnot but, if the cluster fails over, a local DNS can result in connection failures for several seconds/minutes until the local TTL expires … which isn’t great. Generally I’ve had to go into the application code and look for SQL queries that are being needlessly duplicated or are unnecessarily myopic.
0
Reply
cyberrspiritt avatar
in flag
cyberrspiritt
would using nscd services or dnsmasq help in not hitting the limit? since they would cache the NS queries for some time?
0
Reply
in flag
matigo
Yes, so long as the cluster never has to fail over to another instance. Otherwise you’re still waiting for the local TTL
1
Reply
cyberrspiritt avatar
in flag
cyberrspiritt
I tried to debug this using an AWS article: https://aws.amazon.com/premiumsupport/knowledge-center/vpc-find-cause-of-failed-dns-queries/ The most number of packets per second i could find was 26. But when i actually faced that php error today again, the packet count was less than 15 during that time frame.
0
Reply
pn flag
Manuel Kießling
I've run into similar problems, and blogged about my solution here: https://manuel.kiessling.net/2022/01/27/getting-rid-of-temporary-failures-in-name-resolution-on-aws-ec2/
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.