How does HDFS ensure data integrity and prevent corruption during large scale data processing?

All answers to this question.

HDFS is designed for "Write Once, Read Many," and its checksumming is the primary line of defense against bit rot on commodity hardware disks.

   Answered 0024-05-19 by James Taylor

• Exactly, James! And it's important to note that the default replication factor of 3 is what truly provides the safety net for the NameNode to recover those corrupted blocks without data loss.

     Commented 2024-05-21 by Sarah Jenkins

HDFS uses a robust checksum-based verification system to maintain data integrity. When a client writes data, it calculates a checksum for every 512 bytes and stores it alongside the data. During a read operation, the client recalculates the checksum and compares it with the stored version to detect any discrepancies. Additionally, each DataNode runs a background service called the "DataBlockScanner." This scanner periodically verifies all blocks stored on the node against their checksums. If a corruption is detected, the NameNode is notified to trigger a new replica from a healthy source.

   Answered 2024-05-14 by Linda Thompson

That makes sense for the background process, but what happens if the corruption is found right when a client is trying to read the file? Does the client just get an error, or is there a failover?

   Answered 2024-05-16 by Robert Miller

• If a client detects a checksum error during a read, it immediately reports the bad block to the NameNode. The NameNode then marks that specific replica as "corrupt" and directs the client to a different DataNode that holds a healthy replica of that block. Simultaneously, the NameNode schedules the creation of a new, healthy replica to maintain the required replication factor, ensuring the system stays self-healing.

     Commented 2024-05-17 by David Wilson