Join Log in
Score:1
cybermizz avatar Server

High Load Average in Solaris Operating System - Due to high read I/O

br flag
cybermizz

In Linux OS, the load average is dependent on multiple factors like CPU utilization, number of uninterruptible sleep processes, and others. On the other hand, Solaris OS load average is dependent on the number of runnable and running processes.

I have faced a high load average of about 250 (1min), 230 (5min), 219 (15min) on Solaris bare metal machine having the resources as

vCPUs: 256 RAM: 512GB DISK: SAN

During this high load, I figure out that the CPU idle time (87% idle) and the free RAM was is about >100GB. It completely shows that there is no any bottleneck at these two resources. As the backup process running on the system and a lot of reading requests are going to happen on my SAN filesystem but the response time is about 0.25-0.35ms which is far less or good. From the iostat utility, I found that those filesystems were about 40-50% busy. From vmstat, after a few seconds, there are about 40-120 runnable processes means they get into runqueue but in the next second the value is 0. From the graphs and stats, it was visible that the read request caused this issue.

Questions:

  1. Does these runnable processes could cause such a high load average? If its, so How
  2. If the read request response time from SAN is much good and filesystem busy value is 50 percent not 100 percent then why this causing load? How can it be relatable.

Note: In case of any issue regarding this scenario please let me know.

87
0 + 0
Andrew Henle avatar
ph flag
Andrew Henle
*It completely shows that there is no any bottleneck at these two resources* No, it does not. In fact, quite the opposite could be happening. Post the output from `echo ::memstat | mdb -k`, run as root. First, "free memory" counts memory in filesystem and ZFS ARC caches as "free". If you have a sudden need for RAM, and it's tied up in the ZFS ARC, you're screwed. Sure, the ZFS ARC will release its memory, but that can be timed with a sun dial as it's single-threaded and S-L-O-W. When that happens, your run queues go up while your CPUs are idle and your memory is "free".
0
Reply
Andrew Henle avatar
ph flag
Andrew Henle
And it's even worse if your sudden demand for RAM is for huge pages - then the memory pages not only have to be pried free from the ZFS ARC, then the 4 kb pages have to be coalesced into the huge pages needed. (Then the huge pages get released and reclaimed by the ZFS ARC and splintered back into 4 kb pages - repeat *ad naseum*...)
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.