TeraSort is Hadoop’s standard benchmark for measuring distributed sorting performance across clusters. It consists of three components: TeraGen generates random input data, TeraSort performs the distributed sort, and TeraValidate confirms the output is globally sorted. Modern Hadoop deployments typically run on Hadoop 3.x or higher, with improvements in YARN resource management, better performance tuning options,…
Hadoop 1.x reached end-of-life in 2014 and is no longer maintained. This post documents a deprecated architecture for historical reference only. For new deployments, use Hadoop 3.x+ with YARN, which offers significant improvements in resource management, multi-tenancy, and reliability. See the Apache Hadoop documentation for current versions. For managed services, consider AWS EMR, Google Dataproc,…
Gitosis provides a lightweight way to manage Git repositories and user access on a self-hosted server. This post covers user management, repository configuration, and access control. For initial server setup, refer to your Gitosis installation documentation. Adding a New User Each user needs an SSH key pair for authentication. The user generates their key locally:…
Gitosis is unmaintained and shouldn’t be used for new deployments. Use Gitolite or Forgejo instead. This guide covers Gitolite, which provides fine-grained access control, is actively maintained, and handles multiple repositories under a single system user. Prerequisites A server running Linux (Debian, Ubuntu, CentOS, or similar) SSH access to the server with sudo privileges A…
Hadoop daemons communicate over specific TCP ports that you’ll need to know for cluster setup, firewall configuration, and application development. These ports are configurable but come with sensible defaults in Hadoop 3.x. HDFS Ports Service Port Configuration Property Purpose Namenode HTTP UI 9870 dfs.namenode.http-address Web UI and metadata operations Namenode HTTPS UI 9871 dfs.namenode.https-address Secure…
Running benchmarks after a Hadoop installation helps validate that your cluster works correctly and gives you baseline performance metrics. The Sort benchmark is a standard test that exercises the MapReduce execution engine and HDFS, making it useful for stress-testing both the distributed storage layer and computation framework. What the Sort Benchmark Does The Sort benchmark…
Hadoop 3.x remains the current stable version with YARN improvements, HDFS federation, and erasure coding. Hadoop 1.x and 2.x are no longer maintained. For new deployments, evaluate whether Hadoop fits your needs or if managed services like AWS EMR, Google Dataproc, or Azure HDInsight better suit your infrastructure. This post covers common configuration mistakes and…
Hadoop’s default configuration runs in standalone (local) mode as a single JVM process — no HDFS daemons, no YARN, no distributed overhead. This mode is useful for development, testing MapReduce jobs, and debugging before moving to a clustered deployment. For new deployments, evaluate whether you need Hadoop at all. Consider cloud-native alternatives like AWS EMR,…