we have a RHEL 7.6 VM machine , server with 16 GIGA RAM and 8 CPU's
we notices that CPU utilization is high on this server , and also the CPU load average
load average is around - 50 - 60 , and the behavior is like machine most of the time cant access VIA SSH ( like machine is freezes )
therefore we decided to stop all applications on this server , but seems that application isn't part of the problem because CPU utilization still high and the same is the CPU load average
from the dmesg logs we can see the following:
[Sun Oct 11 11:47:10 2021] pci 0000:00:15.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 03] add_size 200000 add_align 100000
[Sun Oct 11 11:47:10 2021] pci 0000:00:15.3: bridge window [io 0x1000-0x0fff] to [bus 06] add_size 1000
[Sun Oct 11 11:47:10 2021] pci 0000:00:15.4: bridge window [io 0x1000-0x0fff] to [bus 07] add_size 1000
[Sun Oct 11 11:47:10 2021] pci 0000:00:15.5: bridge window [io 0x1000-0x0fff] to [bus 08] add_size 1000
[Sun Oct 11 11:47:10 2021] pci 0000:00:15.6: bridge window [io 0x1000-0x0fff] to [bus 09] add_size 1000
[Sun Oct 11 11:47:10 2021] pci 0000:00:15.7: bridge window [io 0x1000-0x0fff] to [bus 0a] add_size 1000
[Sun Oct 11 11:47:10 2021] pci 0000:00:16.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 0b] add_size 200000 add_align 100000
[Sun Oct 11 11:47:10 2021] pci 0000:00:16.3: bridge window [io 0x1000-0x0fff] to [bus 0e] add_size 1000
[Sun Oct 11 11:47:10 2021] pci 0000:00:16.4: bridge window [io 0x1000-0x0fff] to [bus 0f] add_size 1000
[Sun Oct 11 11:47:10 2021] pci 0000:00:16.5: bridge window [io 0x1000-0x0fff] to [bus 10] add_size 1000
[Sun Oct 11 11:47:10 2021] pci 0000:00:16.6: bridge window [io 0x1000-0x0fff] to [bus 11] add_size 1000
[Sun Oct 11 11:47:10 2021] pci 0000:00:16.7: bridge window [io 0x1000-0x0fff] to [bus 12] add_size 1000
[Sun Oct 11 11:47:10 2021] pci 0000:00:17.3: bridge window [io 0x1000-0x0fff] to [bus 16] add_size 1000
[Sun Oct 11 11:47:10 2021] pci 0000:00:15.3: res[13]=[io 0x1000-0x0fff] res_to_dev_res add_size 1000 min_align 1000
[Sun Oct 11 11:47:10 2021] pci 0000:00:15.3: res[13]=[io 0x1000-0x1fff] res_to_dev_res add_size 1000 min_align 1000
[Sun Oct 11 11:47:10 2021] pci 0000:00:15.4: res[13]=[io 0x1000-0x0fff] res_to_dev_res add_size 1000 min_align 1000
[Sun Oct 11 11:47:10 2021] pci 0000:00:15.4: res[13]=[io 0x1000-0x1fff] res_to_dev_res add_size 1000 min_align 1000
[Sun Oct 11 11:47:10 2021] pci 0000:00:15.5: res[13]=[io 0x1000-0x0fff] res_to_dev_res add_size 1000 min_align 1000
[Sun Oct 11 11:47:10 2021] pci 0000:00:15.5: res[13]=[io 0x1000-0x1fff] res_to_dev_res add_size 1000 min_align 1000
[Sun Oct 11 11:47:10 2021] pci 0000:00:15.6: res[13]=[io 0x1000-0x0fff] res_to_dev_res add_size 1000 min_align 1000
[Sun Oct 11 11:47:10 2021] pci 0000:00:15.6: res[13]=[io 0x1000-0x1fff] res_to_dev_res add_size 1000 min_align 1000
[Sun Oct 11 11:47:10 2021] pci 0000:00:15.7: res[13]=[io 0x1000-0x0fff] res_to_dev_res add_size 1000 min_align 1000
[Sun Oct 11 11:47:10 2021] pci 0000:00:15.7: res[13]=[io 0x1000-0x1fff] res_to_dev_res add_size 1000 min_align 1000
[Sun Oct 11 11:47:10 2021] pci 0000:00:16.3: res[13]=[io 0x1000-0x0fff] res_to_dev_res add_size 1000 min_align 1000
[Sun Oct 11 11:47:10 2021] pci 0000:00:16.3: res[13]=[io 0x1000-0x1fff] res_to_dev_res add_size 1000 min_align 1000
[Sun Oct 11 11:47:10 2021] pci 0000:00:16.4: res[13]=[io 0x1000-0x0fff] res_to_dev_res add_size 1000 min_align 1000
[Sun Oct 11 11:47:10 2021] pci 0000:00:16.4: res[13]=[io 0x1000-0x1fff] res_to_dev_res add_size 1000 min_align 1000
[Sun Oct 11 11:47:10 2021] pci 0000:00:16.5: res[13]=[io 0x1000-0x0fff] res_to_dev_res add_size 1000 min_align 1000
[Sun Oct 11 11:47:10 2021] pci 0000:00:16.5: res[13]=[io 0x1000-0x1fff] res_to_dev_res add_size 1000 min_align 1000
[Sun Oct 11 11:47:10 2021] pci 0000:00:15.3: BAR 13: no space for [io size 0x1000]
[Sun Oct 11 11:47:10 2021] pci 0000:00:15.3: BAR 13: failed to assign [io size 0x1000]
[Sun Oct 11 11:47:10 2021] pci 0000:00:15.4: BAR 13: no space for [io size 0x1000]
[Sun Oct 11 11:47:10 2021] pci 0000:00:15.4: BAR 13: failed to assign [io size 0x1000]
[Sun Oct 11 11:47:10 2021] pci 0000:00:15.5: BAR 13: no space for [io size 0x1000]
[Sun Oct 11 11:47:10 2021] pci 0000:00:15.5: BAR 13: failed to assign [io size 0x1000]
[Sun Oct 11 11:47:10 2021] pci 0000:00:15.6: BAR 13: no space for [io size 0x1000]
[Sun Oct 11 11:47:10 2021] pci 0000:00:15.6: BAR 13: failed to assign [io size 0x1000]
[Sun Oct 11 11:47:10 2021] pci 0000:00:15.7: BAR 13: no space for [io size 0x1000]
[Sun Oct 11 11:47:10 2021] pci 0000:00:15.7: BAR 13: failed to assign [io size 0x1000]
[Sun Oct 11 11:47:10 2021] pci 0000:00:16.3: BAR 13: no space for [io size 0x1000]
[Sun Oct 11 11:47:10 2021] pci 0000:00:16.3: BAR 13: failed to assign [io size 0x1000]
[Sun Oct 11 11:47:10 2021] pci 0000:00:16.4: BAR 13: no space for [io size 0x1000]
[Sun Oct 11 11:47:10 2021] pci 0000:00:16.4: BAR 13: failed to assign [io size 0x1000]
[Sun Oct 11 11:47:10 2021] pci 0000:00:11.0: bridge window [io 0x2000-0x3fff]
[Sun Oct 11 11:47:10 2021] pci 0000:00:11.0: bridge window [mem 0xfd600000-0xfdffffff]
[Sun Oct 11 11:47:10 2021] pci 0000:00:11.0: bridge window [mem 0xe7b00000-0xe7ffffff 64bit pref]
[Sun Oct 11 11:47:10 2021] pci 0000:03:00.0: BAR 6: assigned [mem 0xfd500000-0xfd50ffff pref]
[Sun Oct 11 11:47:10 2021] pci 0000:00:15.0: PCI bridge to [bus 03]
[Sun Oct 11 11:47:10 2021] pci 0000:00:15.0: bridge window [io 0x4000-0x4fff]
[Sun Oct 11 11:47:10 2021] pci 0000:00:15.0: bridge window [mem 0xfd500000-0xfd5fffff]
[Sun Oct 11 11:47:10 2021] pci 0000:00:15.0: bridge window [mem 0xc0000000-0xc01fffff 64bit pref]
[Sun Oct 11 11:47:10 2021] pci 0000:00:15.1: PCI bridge to [bus 04]
[Sun Oct 11 11:47:10 2021] pci 0000:00:15.1: bridge window [io 0x8000-0x8fff]
[Sun Oct 11 11:47:10 2021] pci 0000:00:15.1: bridge window [mem 0xfd100000-0xfd1fffff]
[Sun Oct 11 11:47:10 2021] pci 0000:00:15.1: bridge window [mem 0xe7800000-0xe78fffff 64bit pref]
[Sun Oct 11 11:47:10 2021] pci 0000:00:15.2: PCI bridge to [bus 05]
[Sun Oct 11 11:47:10 2021] pci 0000:00:15.2: bridge window [io 0xc000-0xcfff]
[Sun Oct 11 11:47:10 2021] pci 0000:00:15.2: bridge window [mem 0xfcd00000-0xfcdfffff]
[Sun Oct 11 11:47:10 2021] pci 0000:00:15.2: bridge window [mem 0xe7400000-0xe74fffff 64bit pref]
[Sun Oct 11 11:47:10 2021] pci 0000:00:15.3: PCI bridge to [bus 06]
[Sun Oct 11 11:47:10 2021] pci 0000:00:15.3: bridge window [mem 0xfc900000-0xfc9fffff]
[Sun Oct 11 11:47:10 2021] pci 0000:00:15.3: bridge window [mem 0xe7000000-0xe70fffff 64bit pref]
[Sun Oct 11 11:47:10 2021] pci 0000:00:15.4: PCI bridge to [bus 07]
[Sun Oct 11 11:47:10 2021] pci 0000:00:15.4: bridge window [mem 0xfc500000-0xfc5fffff]
[Sun Oct 11 11:47:10 2021] pci 0000:00:15.4: bridge window [mem 0xe6c00000-0xe6cfffff 64bit pref]
[Sun Oct 11 11:47:10 2021] pci_bus 0000:00: resource 5 [mem 0x000cc000-0x000cffff window]
[Sun Oct 11 11:47:10 2021] pci_bus 0000:00: resource 6 [mem 0x000d0000-0x000d3fff window]
[Sun Oct 11 11:47:10 2021] pci_bus 0000:00: resource 7 [mem 0x000d4000-0x000d7fff window]
[Sun Oct 11 11:47:10 2021] pci_bus 0000:00: resource 8 [mem 0x000d8000-0x000dbfff window]
[Sun Oct 11 11:47:10 2021] pci_bus 0000:00: resource 9 [mem 0xc0000000-0xfebfffff window]
[Sun Oct 11 11:47:10 2021] pci_bus 0000:00: resource 10 [io 0x0000-0x0cf7 window]
[Sun Oct 11 11:47:10 2021] pci_bus 0000:00: resource 11 [io 0x0d00-0xfeff window]
[Sun Oct 11 11:47:10 2021] pci_bus 0000:02: resource 0 [io 0x2000-0x3fff]
[Sun Oct 11 11:47:10 2021] pci_bus 0000:02: resource 1 [mem 0xfd600000-0xfdffffff]
[Sun Oct 11 11:47:10 2021] pci_bus 0000:02: resource 2 [mem 0xe7b00000-0xe7ffffff 64bit pref]
[Sun Oct 11 11:47:10 2021] pci_bus 0000:02: resource 4 [mem 0x000a0000-0x000bffff window]
[Sun Oct 11 11:47:10 2021] pci_bus 0000:02: resource 5 [mem 0x000cc000-0x000cffff window]
[Sun Oct 11 11:47:10 2021] pci_bus 0000:02: resource 6 [mem 0x000d0000-0x000d3fff window]
[Sun Oct 11 11:47:10 2021] pci_bus 0000:02: resource 7 [mem 0x000d4000-0x000d7fff window]
[Sun Oct 11 11:47:10 2021] pci_bus 0000:02: resource 8 [mem 0x000d8000-0x000dbfff window]
[Sun Oct 11 11:47:10 2021] pci_bus 0000:02: resource 9 [mem 0xc0000000-0xfebfffff window]
[Sun Oct 11 11:47:10 2021] pci_bus 0000:02: resource 10 [io 0x0000-0x0cf7 window]
[Sun Oct 11 11:47:09 2021] ACPI: LAPIC (acpi_id[0x23] lapic_id[0x23] disabled)
[Sun Oct 11 11:47:09 2021] ACPI: LAPIC (acpi_id[0x24] lapic_id[0x24] disabled)
[Sun Oct 11 11:47:09 2021] ACPI: LAPIC (acpi_id[0x25] lapic_id[0x25] disabled)
[Sun Oct 11 11:47:09 2021] ACPI: LAPIC (acpi_id[0x26] lapic_id[0x26] disabled)
[Sun Oct 11 11:47:09 2021] ACPI: LAPIC (acpi_id[0x27] lapic_id[0x27] disabled)
[Sun Oct 11 11:47:09 2021] ACPI: LAPIC (acpi_id[0x28] lapic_id[0x28] disabled)
[Sun Oct 11 11:47:09 2021] ACPI: LAPIC (acpi_id[0x29] lapic_id[0x29] disabled)
[Sun Oct 11 11:47:09 2021] ACPI: LAPIC (acpi_id[0x2a] lapic_id[0x2a] disabled)
[Sun Oct 11 11:47:09 2021] ACPI: LAPIC (acpi_id[0x2b] lapic_id[0x2b] disabled)
[Sun Oct 11 11:47:09 2021] ACPI: LAPIC (acpi_id[0x2c] lapic_id[0x2c] disabled)
[Sun Oct 11 11:47:09 2021] ACPI: LAPIC (acpi_id[0x2d] lapic_id[0x2d] disabled)
[Sun Oct 11 11:47:09 2021] ACPI: LAPIC (acpi_id[0x2e] lapic_id[0x2e] disabled)
[Sun Oct 11 11:47:09 2021] ACPI: LAPIC (acpi_id[0x2f] lapic_id[0x2f] disabled)
[Sun Oct 11 11:47:09 2021] ACPI: LAPIC (acpi_id[0x30] lapic_id[0x30] disabled)
[Sun Oct 11 11:47:09 2021] ACPI: LAPIC (acpi_id[0x31] lapic_id[0x31] disabled)
[Sun Oct 11 11:47:09 2021] ACPI: LAPIC (acpi_id[0x32] lapic_id[0x32] disabled)
[Sun Oct 11 11:47:09 2021] ACPI: LAPIC (acpi_id[0x33] lapic_id[0x33] disabled)
[Sun Oct 11 11:47:09 2021] ACPI: LAPIC (acpi_id[0x34] lapic_id[0x34] disabled)
[Sun Oct 11 11:47:09 2021] ACPI: LAPIC (acpi_id[0x35] lapic_id[0x35] disabled)
[Sun Oct 11 11:47:09 2021] ACPI: LAPIC (acpi_id[0x36] lapic_id[0x36] disabled)
[Sun Oct 11 11:47:09 2021] ACPI: LAPIC (acpi_id[0x37] lapic_id[0x37] disabled)
[Sun Oct 11 11:47:09 2021] ACPI: LAPIC (acpi_id[0x38] lapic_id[0x38] disabled)
[Sun Oct 11 11:47:09 2021] ACPI: LAPIC (acpi_id[0x39] lapic_id[0x39] disabled)
[Sun Oct 11 11:47:09 2021] ACPI: LAPIC (acpi_id[0x3a] lapic_id[0x3a] disabled)
[Sun Oct 11 11:47:09 2021] ACPI: LAPIC (acpi_id[0x3b] lapic_id[0x3b] disabled)
[Sun Oct 11 11:47:09 2021] ACPI: LAPIC (acpi_id[0x3c] lapic_id[0x3c] disabled)
[Sun Oct 11 11:47:09 2021] ACPI: LAPIC (acpi_id[0x3d] lapic_id[0x3d] disabled)
[Sun Oct 11 11:47:09 2021] ACPI: LAPIC (acpi_id[0x3e] lapic_id[0x3e] disabled)
[Sun Oct 11 11:47:09 2021] ACPI: LAPIC (acpi_id[0x3f] lapic_id[0x3f] disabled)
[Sun Oct 11 11:47:09 2021] ACPI: LAPIC (acpi_id[0x40] lapic_id[0x40] disabled)
[Sun Oct 11 11:47:09 2021] ACPI: LAPIC (acpi_id[0x41] lapic_id[0x41] disabled)
[Sun Oct 11 11:47:09 2021] ACPI: LAPIC (acpi_id[0x42] lapic_id[0x42] disabled)
[Sun Oct 11 11:47:09 2021] ACPI: LAPIC (acpi_id[0x43] lapic_id[0x43] disabled)
[Sun Oct 11 11:47:09 2021] ACPI: LAPIC (acpi_id[0x44] lapic_id[0x44] disabled)
[Sun Oct 11 11:47:09 2021] ACPI: LAPIC (acpi_id[0x45] lapic_id[0x45] disabled)
another important part is about the DIMM cards that was replaced on ESX machine , and not sure if our problem related to that
based on the dmesg info , what are the suggestion/s that can be performed in order to solved the CPU load average problem ?
