we have Hadoop cluster , and we noticed that hdfs fsck gives the following results
..............Status: HEALTHY
Total size: 58934628832 B
Total dirs: 127002
Total files: 354214
Total symlinks: 0 (Files currently being written: 8)
Total blocks (validated): 349827 (avg. block size 168467 B) (Total open file blocks (not validated): 1)
Minimally replicated blocks: 349827 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 101699 (29.071226 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 2.6958897
Corrupt blocks: 0
Missing replicas: 106386 (10.137011 %)
Number of data-nodes: 5
Number of racks: 1
we can see Missing replicas: 106386
so we performed the following steps but without success
hadoop fsck / -move
and
hadoop fsck / -delete
and
hadoop fsck / -files -blocks -locations
and also we try to fix under replica
su - <$hdfs_user>
bash-4.1$ hdfs fsck / | grep 'Under replicated' | awk -F':' '{print $1}' >> /tmp/under_replicated_files
-bash-4.1$ for hdfsfile in `cat /tmp/under_replicated_files`; do echo "Fixing $hdfsfile :" ; hadoop fs -setrep 3 $hdfsfile; done
so all above steps are not solved the problem with missing replica
any other idea how to continue from this stage?
