When working with Hadoop Distributed File System (HDFS), files containing spaces in their names require special handling during the hdfs dfs -put command. Without proper escaping or quoting, the shell interprets spaces as delimiters, splitting the filename into multiple arguments. Direct Upload with Quoting The simplest approach is to wrap the filename in quotes: hdfs…
Understanding the difference between reliability, durability, and availability is critical when designing or operating distributed storage systems like HDFS, Ceph, or cloud object storage. These terms are often confused, but they address distinct concerns. Durability Durability answers: If the system fails completely, will my data survive? Durability is about persistence to non-volatile storage. Data is…
SQL layers abstract traditional relational query interfaces over distributed NoSQL and big data systems, enabling applications to leverage SQL’s familiarity while benefiting from horizontal scalability. Here are the key approaches and systems that have emerged in this space. Distributed SQL on Key-Value Stores Phoenix (Apache Phoenix) provides a SQL interface over HBase. It compiles SQL…
Hadoop 2.x reached end-of-life in 2016. This guide covers setup for learning purposes and legacy system maintenance only. For production deployments, use Hadoop 3.x or later, which includes performance improvements, better YARN scheduling, HDFS erasure coding, and improved security. Cloud-managed options like AWS EMR, Google Dataproc, and Azure HDInsight eliminate most operational overhead. This guide…
Choosing the right cloud storage service requires understanding how each platform handles data consistency. This post covers the current consistency guarantees from the three major providers and how they affect your application design. Amazon S3 S3 is a key-based object store built for Internet-scale workloads. The consistency model is now straightforward across all operations. S3…
Andrew Fikes, Principal Engineer at Google, presented on storage architecture and the fundamental challenges that emerge when operating at scale. The presentation from the 2010 Faculty Summit remains relevant for understanding how distributed storage systems must be designed and the trade-offs inherent in large-scale deployments. Core Storage Challenges at Scale Google’s storage systems had to…
Modern datacenters rely on distributed storage and processing systems designed to handle massive datasets across clusters of commodity hardware. Understanding these systems is essential for anyone working with infrastructure at scale. Here’s an overview of the major architectures and implementations you’ll encounter. Storage Systems Google File System (GFS) and successors GFS established the foundation for…
Colossus is Google’s successor to the Google File System (GFS), designed to handle the massive scale of data storage across Google’s infrastructure. While GFS was documented in detail in the 2003 SOSP paper, Colossus information remains sparse in academic literature—much of what we know comes from talks and interviews with Google engineers. Colossus serves as…