Understanding the difference between reliability, durability, and availability is critical when designing or operating distributed storage systems like HDFS, Ceph, or cloud object storage. These terms are often confused, but they address distinct concerns. Durability Durability answers: If the system fails completely, will my data survive? Durability is about persistence to non-volatile storage. Data is…
The replication factor determines how many copies of each data block HDFS maintains across your cluster. The default is 3, which provides adequate fault tolerance for most deployments by surviving both node and rack failures simultaneously. Setting Cluster-Wide Default Replication To set the default replication factor for all new files, add the dfs.replication property to…
Hadoop uses three primary configuration files to define YARN, HDFS, and MapReduce behavior: HDFS: hdfs-site.xml YARN: yarn-site.xml MapReduce: mapred-site.xml These files live in $HADOOP_HOME/etc/hadoop/ and override the built-in defaults when present. Finding Official Default Values Apache publishes default configuration documentation for each release. For current versions: Hadoop 3.4.x (Latest) HDFS defaults: https://hadoop.apache.org/docs/r3.4.0/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml YARN defaults: https://hadoop.apache.org/docs/r3.4.0/hadoop-yarn/hadoop-yarn-common/yarn-default.xml…
SQL layers abstract traditional relational query interfaces over distributed NoSQL and big data systems, enabling applications to leverage SQL’s familiarity while benefiting from horizontal scalability. Here are the key approaches and systems that have emerged in this space. Distributed SQL on Key-Value Stores Phoenix (Apache Phoenix) provides a SQL interface over HBase. It compiles SQL…
The Secondary NameNode periodically merges the fsimage and edits log files to keep the edits log manageable and consolidate metadata. When this checkpointing fails, you’ll see errors like: ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in doCheckpoint java.io.IOException: Inconsistent checkpoint fields. This typically indicates a mismatch between the NameNode and Secondary NameNode metadata states, often from unclean shutdowns or…
Hadoop 2.x reached end-of-life in 2016. This guide covers setup for learning purposes and legacy system maintenance only. For production deployments, use Hadoop 3.x or later, which includes performance improvements, better YARN scheduling, HDFS erasure coding, and improved security. Cloud-managed options like AWS EMR, Google Dataproc, and Azure HDInsight eliminate most operational overhead. This guide…
MapReduce is a foundational distributed computing model, but it’s no longer the default choice for new projects. Hadoop’s MapReduce (via YARN in Hadoop 2.x+) remains production-stable and useful for specific scenarios, while Apache Spark, Flink, and cloud-native services handle most modern workloads more efficiently. When MapReduce Still Makes Sense Use MapReduce if you’re: Maintaining existing…
Modern datacenters rely on distributed storage and processing systems designed to handle massive datasets across clusters of commodity hardware. Understanding these systems is essential for anyone working with infrastructure at scale. Here’s an overview of the major architectures and implementations you’ll encounter. Storage Systems Google File System (GFS) and successors GFS established the foundation for…
Hadoop 1.x reached end-of-life in 2014 and is no longer maintained. This post documents a deprecated architecture for historical reference only. For new deployments, use Hadoop 3.x+ with YARN, which offers significant improvements in resource management, multi-tenancy, and reliability. See the Apache Hadoop documentation for current versions. For managed services, consider AWS EMR, Google Dataproc,…
Running benchmarks after a Hadoop installation helps validate that your cluster works correctly and gives you baseline performance metrics. The Sort benchmark is a standard test that exercises the MapReduce execution engine and HDFS, making it useful for stress-testing both the distributed storage layer and computation framework. What the Sort Benchmark Does The Sort benchmark…
Archived Content (2010): This post describes a research project from 2010. The tools and versions mentioned (Hadoop 0.20, MapReduce Streaming) are historically significant but have been largely superseded by modern distributed build systems like Bazel Remote Execution and cloud-native CI/CD pipelines. mrcc – A Distributed C Compiler System on MapReduce Original Project Date: January 2010…