There is an HP DL380 G8 server with P420i Raid Controller, containing a RAID5 volume. It suddenly became slow, but all physical disks are ok.
[root@localhost ~]# ssacli ctrl all show config detail
Smart Array P420i in Slot 0 (Embedded)
Bus Interface: PCI
Slot: 0
Serial Number: XXXXXXXXXXXX
Cache Serial Number: XXXXXXXXXXXX
RAID 6 (ADG) Status: Enabled
Controller Status: OK
Hardware Revision: B
Firmware Version: 5.42-0
Rebuild Priority: Low
Expand Priority: Medium
Surface Scan Delay: 3 secs
Surface Scan Mode: Idle
Parallel Surface Scan Supported: No
Queue Depth: Automatic
Monitor and Performance Delay: 60 min
Elevator Sort: Enabled
Degraded Performance Optimization: Disabled
Inconsistency Repair Policy: Disabled
Wait for Cache Room: Disabled
Surface Analysis Inconsistency Notification: Disabled
Post Prompt Timeout: 15 secs
Cache Board Present: True
Cache Status: OK
Cache Ratio: 10% Read / 90% Write
Drive Write Cache: Enabled
Total Cache Size: 1.0
Total Cache Memory Available: 0.8
No-Battery Write Cache: Enabled
SSD Caching RAID5 WriteBack Enabled: False
SSD Caching Version: 1
Cache Backup Power Source: Capacitors
Battery/Capacitor Count: 1
Battery/Capacitor Status: OK
SATA NCQ Supported: True
Spare Activation Mode: Activate on physical drive failure (default)
Controller Temperature (C): 72
Cache Module Temperature (C): 41
Capacitor Temperature (C): 33
Number of Ports: 2 Internal only
Encryption: Not Set
Driver Name: hpsa
Driver Version: 3.4.20
Driver Supports SSD Smart Path: True
PCI Address (Domain:Bus:Device.Function): 0000:02:00.0
Port Max Phy Rate Limiting Supported: False
Host Serial Number: XXXXXXXXXXXX
Sanitize Erase Supported: False
Primary Boot Volume: None
Secondary Boot Volume: None
Internal Drive Cage at Port 1I, Box 1, OK
Power Supply Status: Not Redundant
Drive Bays: 4
Port: 1I
Box: 1
Location: Internal
Physical Drives
physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS HDD, 8 TB, OK)
physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS HDD, 8 TB, OK)
physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SAS HDD, 8 TB, OK)
physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SAS HDD, 8 TB, OK)
Internal Drive Cage at Port 2I, Box 1, OK
Power Supply Status: Not Redundant
Drive Bays: 4
Port: 2I
Box: 1
Location: Internal
Physical Drives
physicaldrive 2I:1:5 (port 2I:box 1:bay 5, SAS HDD, 8 TB, OK)
physicaldrive 2I:1:6 (port 2I:box 1:bay 6, SAS HDD, 8 TB, OK)
physicaldrive 2I:1:7 (port 2I:box 1:bay 7, SAS HDD, 8 TB, OK)
physicaldrive 2I:1:8 (port 2I:box 1:bay 8, SAS HDD, 8 TB, OK)
Port Name: 1I
Port ID: 0
Port Connection Number: 0
SAS Address: XXXXXXXXXXXX
Port Location: Internal
Port Name: 2I
Port ID: 1
Port Connection Number: 1
SAS Address: XXXXXXXXXXXX
Port Location: Internal
Array: A
Interface Type: SAS
Unused Space: 0 MB (0.00%)
Used Space: 58.22 TB (100.00%)
Status: OK
Array Type: Data
Smart Path: disable
Logical Drive: 1
Size: 50.94 TB
Fault Tolerance: 5
Heads: 255
Sectors Per Track: 32
Cylinders: 65535
Strip Size: 256 KB
Full Stripe Size: 1792 KB
Status: OK
Unrecoverable Media Errors: None
Caching: Enabled
Parity Initialization Status: Initialization Completed
Unique Identifier: XXXXXXXXXXXX
Disk Name: /dev/sda
Mount Points: /boot 1023 MB Partition Number 2
OS Status: LOCKED
Logical Drive Label: XXXXXXXXXXXX
Drive Type: Data
LD Acceleration Method: Controller Cache
physicaldrive 1I:1:1
Port: 1I
Box: 1
Bay: 1
Status: OK
Drive Type: Data Drive
Interface Type: SAS
Size: 8 TB
Drive exposed to OS: False
Logical/Physical Block Size: 512/4096
Rotational Speed: 7200
Firmware Revision: E004
Serial Number: XXXXXXXXXXXX
WWID: XXXXXXXXXXXX
Model: SEAGATE ST8000NM0075
PHY Count: 2
PHY Transfer Rate: 6.0Gbps, Unknown
Drive Authentication Status: OK
Carrier Application Version: 11
Carrier Bootloader Version: 6
Sanitize Erase Supported: True
Unrestricted Sanitize Supported: True
Shingled Magnetic Recording Support: None
physicaldrive 1I:1:2
Port: 1I
Box: 1
Bay: 2
Status: OK
Drive Type: Data Drive
Interface Type: SAS
Size: 8 TB
Drive exposed to OS: False
Logical/Physical Block Size: 512/4096
Rotational Speed: 7200
Firmware Revision: E003
Serial Number: XXXXXXXXXXXX
WWID: XXXXXXXXXXXX
Model: SEAGATE ST8000NM0075
PHY Count: 2
PHY Transfer Rate: 6.0Gbps, Unknown
Drive Authentication Status: OK
Carrier Application Version: 11
Carrier Bootloader Version: 6
Sanitize Erase Supported: True
Unrestricted Sanitize Supported: True
Shingled Magnetic Recording Support: None
physicaldrive 1I:1:3
Port: 1I
Box: 1
Bay: 3
Status: OK
Drive Type: Data Drive
Interface Type: SAS
Size: 8 TB
Drive exposed to OS: False
Logical/Physical Block Size: 512/4096
Rotational Speed: 7200
Firmware Revision: E003
Serial Number: XXXXXXXXXXXX
WWID: XXXXXXXXXXXX
Model: SEAGATE ST8000NM0075
PHY Count: 2
PHY Transfer Rate: 6.0Gbps, Unknown
Drive Authentication Status: OK
Carrier Application Version: 11
Carrier Bootloader Version: 6
Sanitize Erase Supported: True
Unrestricted Sanitize Supported: True
Shingled Magnetic Recording Support: None
physicaldrive 1I:1:4
Port: 1I
Box: 1
Bay: 4
Status: OK
Drive Type: Data Drive
Interface Type: SAS
Size: 8 TB
Drive exposed to OS: False
Logical/Physical Block Size: 512/4096
Rotational Speed: 7200
Firmware Revision: E003
Serial Number: XXXXXXXXXXXX
WWID: XXXXXXXXXXXX
Model: SEAGATE ST8000NM0075
PHY Count: 2
PHY Transfer Rate: 6.0Gbps, Unknown
Drive Authentication Status: OK
Carrier Application Version: 11
Carrier Bootloader Version: 6
Sanitize Erase Supported: True
Unrestricted Sanitize Supported: True
Shingled Magnetic Recording Support: None
physicaldrive 2I:1:5
Port: 2I
Box: 1
Bay: 5
Status: OK
Drive Type: Data Drive
Interface Type: SAS
Size: 8 TB
Drive exposed to OS: False
Logical/Physical Block Size: 512/4096
Rotational Speed: 7200
Firmware Revision: E003
Serial Number: XXXXXXXXXXXX
WWID: XXXXXXXXXXXX
Model: SEAGATE ST8000NM0075
PHY Count: 2
PHY Transfer Rate: 6.0Gbps, Unknown
Drive Authentication Status: OK
Carrier Application Version: 11
Carrier Bootloader Version: 6
Sanitize Erase Supported: True
Unrestricted Sanitize Supported: True
Shingled Magnetic Recording Support: None
physicaldrive 2I:1:6
Port: 2I
Box: 1
Bay: 6
Status: OK
Drive Type: Data Drive
Interface Type: SAS
Size: 8 TB
Drive exposed to OS: False
Logical/Physical Block Size: 512/4096
Rotational Speed: 7200
Firmware Revision: E003
Serial Number: XXXXXXXXXXXX
WWID: XXXXXXXXXXXX
Model: SEAGATE ST8000NM0075
PHY Count: 2
PHY Transfer Rate: 6.0Gbps, Unknown
Drive Authentication Status: OK
Carrier Application Version: 11
Carrier Bootloader Version: 6
Sanitize Erase Supported: True
Unrestricted Sanitize Supported: True
Shingled Magnetic Recording Support: None
physicaldrive 2I:1:7
Port: 2I
Box: 1
Bay: 7
Status: OK
Drive Type: Data Drive
Interface Type: SAS
Size: 8 TB
Drive exposed to OS: False
Logical/Physical Block Size: 512/4096
Rotational Speed: 7200
Firmware Revision: E003
Serial Number: XXXXXXXXXXXX
WWID: XXXXXXXXXXXX
Model: SEAGATE ST8000NM0075
PHY Count: 2
PHY Transfer Rate: 6.0Gbps, Unknown
Drive Authentication Status: OK
Carrier Application Version: 11
Carrier Bootloader Version: 6
Sanitize Erase Supported: True
Unrestricted Sanitize Supported: True
Shingled Magnetic Recording Support: None
physicaldrive 2I:1:8
Port: 2I
Box: 1
Bay: 8
Status: OK
Drive Type: Data Drive
Interface Type: SAS
Size: 8 TB
Drive exposed to OS: False
Logical/Physical Block Size: 512/4096
Rotational Speed: 7200
Firmware Revision: E004
Serial Number: XXXXXXXXXXXX
WWID: XXXXXXXXXXXX
Model: SEAGATE ST8000NM0075
PHY Count: 2
PHY Transfer Rate: 6.0Gbps, Unknown
Drive Authentication Status: OK
Carrier Application Version: 11
Carrier Bootloader Version: 6
Sanitize Erase Supported: True
Unrestricted Sanitize Supported: True
Shingled Magnetic Recording Support: None
SEP (Vendor ID PMCSIERA, Model SRCv8x6G) 380
Device Number: 380
Firmware Version: RevB
WWID: XXXXXX
Vendor ID: PMCSIERA
Model: SRCv8x6G
All disks look good when I check their SMART info.
[root@localhost ~]# smartctl -a -d cciss,1 /dev/sda
smartctl 7.0 2018-12-30 r4883 [x86_64-linux-3.10.0-957.el7.x86_64] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Vendor: SEAGATE
Product: ST8000NM0075
Revision: E003
Compliance: SPC-4
User Capacity: 8,001,563,222,016 bytes [8.00 TB]
Logical block size: 512 bytes
Physical block size: 4096 bytes
LU is fully provisioned
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Logical Unit id: 0xXXXXXXXXXXXXXXX
Serial number: XXXXXXXXXXXXXXX
Device type: disk
Transport protocol: SAS (SPL-3)
Local Time is: Sat Jan 22 01:58:45 2022 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK
Grown defects during certification <not available>
Total blocks reassigned during format <not available>
Total new blocks reassigned <not available>
Power on minutes since format <not available>
Current Drive Temperature: 41 C
Drive Trip Temperature: 60 C
Manufactured in week 16 of year 2018
Specified cycle count over device lifetime: 10000
Accumulated start-stop cycles: 98
Specified load-unload count over device lifetime: 300000
Accumulated load-unload cycles: 1411
Elements in grown defect list: 0
Vendor (Seagate Cache) information
Blocks sent to initiator = 238399504
Blocks received from initiator = 2085969384
Blocks read from cache and sent to initiator = 3750062329
Number of read and write commands whose size <= segment size = 45023273
Number of read and write commands whose size > segment size = 2781
Vendor (Seagate/Hitachi) factory information
number of hours powered up = 31417.37
number of minutes until next internal SMART test = 7
Error counter log:
Errors Corrected by Total Correction Gigabytes Total
ECC rereads/ errors algorithm processed uncorrected
fast | delayed rewrites corrected invocations [10^9 bytes] errors
read: 2200494275 0 0 2200494275 0 13316.232 0
write: 0 0 0 0 0 1068.584 0
Non-medium error count: 4083
[GLTSD (Global Logging Target Save Disable) set. Enable Save with '-S on']
No Self-tests have been logged
Is there any way to check the IO stats on a physical disk in a hardware raid controller?
How can I find out the root cause of the problem?
