Score:0

network-latency/ping degradation over time correlated with memory fragmentation

in flag

There is an Online Charging System (OCS) which handles Diameter Ro and Gy traffic. The OCS receives CCR Diameter messages and answers with CCA Diameter messages. And this response time - latency - degrades over time.

Normal average latency is 20 ms, but in one week it degrades to 70 ms with increased frequency of spikes more than few seconds. The cluster consists of several physical hosts, on each physical host there are two VMs (Redhat KVM, RHEL) - Application (C++) VM and Database (Java) VM.

There is a strong correlation between latency degradation and memory fragmentation. For example, after cache drop + memory compaction or VM restart the latency becomes normal, but in one week it degrades to the numbers I wrote above.

If there is a VM backup, then latency degrades immediately after it.

Apart from Diameter latency degradation, there is a correlated ping RTT degradation with spikes up to 30 seconds. Ping RTT degradation between VMs on the same physical host is worse than between VMs on different hosts. There is no ping RTT degradation between physical hosts.

Any ideas how to fix this latency issue are welcome!

I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.