One of HDFS cluster’s hdfs dfsadmin -report reports:
Under replicated blocks: 139016
Blocks with corrupt replicas: 9
Missing blocks: 0
The “Under replicated blocks” can be re-replicated automatically after some time.
How to handle the missing blocks and blocks with corrupt replicas in HDFS?
Understanding these blocks
Table of Contents
A block is “with corrupt replicas” in HDFS if it has at least one corrupt replica along with at least one live replica. As such, a block having corrupt replicas does not indicate unavailable data, but they do indicate an increased chance that data may become unavailable.
If none of a block’s replicas are live, the block is called a missing block by HDFS, not a block with corrupt replicas.
Potential causes
Here are lists of potential causes and actions that you may take to handle the missing or corrupted blocks:
- HDFS automatically fixes corrupt blocks in the background. A failure of this may indicate a problem with the underlying storage or filesystem of a DataNode. Use the HDFS fsck command to identify which files contain corrupt blocks.
- Some DataNodes are down and the replicas that are missing blocks are only on those DataNodes.
- The corrupt/missing blocks are from files with a replication factor of 1. New replicas cannot be created because the only replica of the block is missing.
Possible remedies
Some suggestion as by HDP
- For critical data, use a replication factor of 3
- Bring up the failed DataNodes with missing or corrupt blocks.
- Identify the files associated with the missing or corrupt blocks by running the Hadoop fsck command.
- Delete the corrupt files and recover them from backup, if it exists.
Reference: HDP doc and Cloudera doc.
Hi. I believe there’s a mistake in the description of corrupt blocks: “A block is called corrupt by HDFS if it has at least one corrupt replica along.” I think that all replicas must be corrupted to be marked as corrupt rather than at least one.
JIRA HDFS-7281 explains the following:
1. A block is missing if and only if all DNs of its expected replicas are dead.
2. A block is corrupted if and only if all its available replicas are corrupted. So if a block has 3 replicas; one of the DN is dead, the other two replicas are corrupted; it will be marked as corrupted.
Hi Jim, yes, the description is not accurate. I fixed it to make it clearer – “A block is “with corrupt replicas” in HDFS if it has at least one corrupt replica along with at least one live replica.”
It refers to the count from the HDFS report.