Score:1

HDFS + how to disable the "du -sk" verifcation on data node disks

gb flag

We are using HDP cluster with 182 data node machines:

HDP version - 2.6.4 Ambari version 2.6.1

We note the following behavior on the data nodes machines (its happens on all data-node machines and on all disks).

When we perform the command as above example:

ps -eo s,user,cmd | grep ^[RD]
D hdfs     du -sk /grid/sdj/hadoop/hdfs/data/current/BP-1018134753-10.3.6.170-1530088122990
D hdfs     du -sk /grid/sdm/hadoop/hdfs/data/current/BP-1018134753-10.3.6.170-1530088122990
R root     ps -eo s,user,cmd

Note - each disk in the data node is 5.4 T Bytes.

We can see that HDFS is running the "du -sk" on the data node disks

We don't like this, because the meaning of that is consuming high load CPU Avrg and sometimes even bad performance.

We are understand that HDFS need to run the "du -sk" in order to verify the disks space, but on other hand its cost - high CPU load avrg and sometimes even poor performance.

Is it possible to tell HDFS in some way to disable this verification?

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.