Score:0

VMDK disk became read only & how to avoid such this cases on rhel machines

gb flag

we have Kafka cluster with RHEL 7.6 , all Kafka are VM machines

on one of the Kafka machines , we noticed that sdb disk became read only ( when sda is the OS disk )

 mount | grep sdb
/dev/sdb on /var/data/kafka_DB type ext4 (ro,noatime,data=ordered)

from my point of view its little strange that DISK VMDK became read only ( because its not mechanic disk )

from red-hat I find the following

https://access.redhat.com/solutions/1273213

https://access.redhat.com/solutions/35329

but not sure if above suggestions from redhat are the answer why disk became read only

any others opinions?

from the kernel logs we can see:

[1642397.157193] sd 0:0:2:0: [sdb] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[1642397.157200] sd 0:0:2:0: [sdb] CDB: Write(10) 2a 00 12 c0 01 00 00 00 08 00
[1642397.157214] blk_update_request: I/O error, dev sdb, sector 314573056
[1642397.157242] Buffer I/O error on dev sdb, logical block 39321632, lost async page write
[1642397.157806] sd 0:0:2:0: [sdb] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[1642397.157808] sd 0:0:2:0: [sdb] CDB: Read(10) 28 00 12 c4 03 58 00 00 08 00
[1642397.157810] blk_update_request: I/O error, dev sdb, sector 314835800
[1642397.157843] sd 0:0:2:0: [sdb] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
[1642397.157845] sd 0:0:2:0: [sdb] CDB: Read(10) 28 00 12 c4 0b a0 00 00 08 00
[1642397.157847] blk_update_request: I/O error, dev sdb, sector 314837920
[1642578.412306] sd 0:0:2:0: [sdb] task abort on host 0, ffff8c147c189880
[1642924.513605] sd 0:0:2:0: [sdb] task abort on host 0, ffff8c16a4f01880
[1643034.935334] JBD2: Detected IO errors while flushing file data on sdb-8
[1643035.002651] EXT4-fs error (device sdb): __ext4_new_inode:989: comm pool-6-thread-1: failed to insert inode 8126474: doubly allocated?
[1643036.753397] Aborting journal on device sdb-8.
[1643036.754490] EXT4-fs error (device sdb): ext4_journal_check_start:56: Detected aborted journal
[1643036.754496] EXT4-fs (sdb): Remounting filesystem read-only
[1643226.599854] sd 0:0:2:0: [sdb] task abort on host 0, ffff8c14a4bd3800
[1694249.598258] EXT4-fs (sdb): error count since last fsck: 17
[1694249.598269] EXT4-fs (sdb): initial error at time 1629844995: ext4_find_entry:1312: inode 656236
[1694249.598273] EXT4-fs (sdb): last error at time 1630003886: ext4_journal_check_start:56
[1780756.527074] EXT4-fs (sdb): error count since last fsck: 17
[1780756.527086] EXT4-fs (sdb): initial error at time 1629844995: ext4_find_entry:1312: inode 656236
[1780756.527088] EXT4-fs (sdb): last error at time 1630003886: ext4_journal_check_start:56

what we think to do , is to update the /sys/block/basename /dev/sdb/device/timeout

for example the default value is 180

and we are thinking to set update new value as

echo 3600 > /sys/block/`basename /dev/sda`/device/timeout

we want to know if we are on the right direction with above solution ?

Ginnungagap avatar
gu flag
Have you checked your kernel logs?
King David avatar
gb flag
I update the question
Ginnungagap avatar
gu flag
What was the physical disk load at the time? Do you use a backup tool like Veeam? Is the physical disk exposed via the network to ESXi?
King David avatar
gb flag
we not have access to the physical disk , so I can capture this info , what I want to know if redhat suggestion can help us?
King David avatar
gb flag
we involved the ESX admin and he said that he not see any issues from the physical disk on data store
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.