Score:2

Linux ports become unavailable

cn flag

We have some Ubuntu 18.04 servers with hard-to-diagnose problems that have some similarities. The servers run a hosted VOIP PBX service with various component applications.

  1. Kamailio occasionally starts giving "Lost connection to MySQL" errors. (these connections are TCP)
  2. Kamailio occasionally gives "more than 'max_user_connections' active connections" MySQL errors. (these connections are TCP)
  3. Asterisk occasionally starts giving "couldn't allocate a port for RTP" errors. (these connections are UDP)

Although there are other possible explanations individually (and we've looked into those), they could conceivably all be caused by the same one problem of ports not being released back to the available pool after they've been used.

Would anyone have any hints about where to look to find out if this theory is true, or possible solutions if it is? Thanks in advance.

Nikita Kipriyanov avatar
za flag
The simpler explanation could be that your *asterisk user* runs out of available file descriptors, set too low in `/etc/limits.conf` or some file included into it. Before thinking there are problems in the system, diagnose; in particular, `lsof` for that user and `ss -lnptu` could reveal is there really no "unused" ports left or you hit the said limit.
David Cunningham avatar
cn flag
Thanks Nikita, using "lsof" showed ports were in use, but the question is why. We have many other servers with similar software which don't have the issue. All these problems happen to just one group of servers, and none on any others. That's why we're looking at what could be a system-wide issue on these servers, affecting more than one application on those servers, for example if something could prevent ports from being released.
Nikita Kipriyanov avatar
za flag
*All 65534* ports were in use or only the subset that was configured in the Asterisk or only 1024 or whatever the limit is set for the asterisk user? I asked you to do `lsof` exactly to distinguish those cases.
Wilson Hauck avatar
jp flag
@DavidCunningham Additional DB information request from FAILING srv, please. RAM size, # cores, any SSD or NVME devices on MySQL Host server? Post TEXT data on justpaste.it and share links. From your SSH login root, Text results of: A) SELECT COUNT(*), sum(data_length), sum(index_length), sum(data_free) FROM information_schema.tables; B) SHOW GLOBAL STATUS; after minimum 24 hours UPTIME C) SHOW GLOBAL VARIABLES; D) SHOW FULL PROCESSLIST; E) STATUS; not SHOW STATUS, just STATUS; G) SHOW ENGINE INNODB STATUS; for server workload tuning analysis to provide suggestions.
Wilson Hauck avatar
jp flag
From one failing server - please, Post TEXT data on justpaste.it and share the links. Additional very helpful OS information includes - please, htop 1st page, if available, TERMINATE, top -b -n 1 for most active apps, top -b -n 1 -H for details on your mysql threads memory and cpu usage, ulimit -a for list of limits, iostat -xm 5 3 for IOPS by device & core/cpu count, df -h for Used - Free space by device, df -i for inode info by device, cat /proc/meminfo includes VMallocUused, for server workload tuning analysis to provide suggestions.
Score:0
in flag

Check below links as you haven't shared complete logs or what you have tried to fix it, so I am sharing similar issue.

Lost connection to MySQL -- Lost Connection

More than 'max_user_connections' active connections -- max_user_connections

Couldn't allocate a port for RTP -- Port

I am sure these links will give you pointer to either towards solution or improve your question.

David Cunningham avatar
cn flag
Hello, thank you for that, however my question is not about each one individually. I'm asking about the possibility that there is some system-wide problem about ports not being released. We manage many other servers with similar software, and all these problems happen to just one group of servers, and none on any others. That's why we're looking at what could be a system-wide issue on these servers, affecting more than one application on these servers.
asktyagi avatar
in flag
If you check each links, it shows many problems with solution. If you have any config management tool which is managing config across your environment then problem also replicated. These issues are generic and can be related to software or os settings.
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.