Object-oriented design, design patterns, system design, and software architecture principles. Explore Programming Academy.
Hadoop launched in 2006 as a game-changer for processing massive datasets across distributed clusters. The core idea remains sound: bring computation to the data rather than moving petabytes across networks. But the landscape has shifted dramatically, and understanding where Hadoop fits today matters more than treating it as the default choice. Why Big Data Processing…
Use the -d flag to run a container in the background and return terminal control immediately: docker run -d –name my-service my-image The container starts as a daemon process. Verify it’s running and check output: docker ps docker logs my-service Essential Configuration for Production Daemons A single -d flag isn’t enough for real workloads. Here’s…
In C, the most straightforward way to store a homogeneous collection of objects is an array. An array is a contiguous block of memory holding elements of identical type, with the first element at the lowest address and subsequent elements following immediately after. Arrays provide O(1) random access to any element and are efficient for…
Sometimes you need an older version of Hadoop — for compatibility with existing jobs, to reproduce bugs, or to match a specific cluster environment. The Apache Hadoop project maintains an archive of all past releases. From the Apache Archive All Hadoop releases are available from the Apache Software Foundation’s distribution archive: https://archive.apache.org/dist/hadoop/common/ Browse the directory…
The bottom panel in GNOME Classic displays the window list via the “Window List” extension. If you don’t use it, there are several ways to remove it depending on your approach. Method 1: Context menu removal (Recommended) The simplest approach if your GNOME version supports it: Hover over a blank area of the bottom panel…
You can retrieve detailed metadata about an S3 object using the aws s3api head-object command. This is useful when you need to check object properties, sizes, modification times, or custom metadata without downloading the entire file. Basic Usage The simplest form queries an object’s metadata: aws s3api head-object –bucket test-hkust –key dir2/fileupload/fb0c6353-a90c-4522-9355-7cd16cf756ff.file.txt This returns output…
The YARN resource scheduler determines how cluster resources are allocated to jobs. By default, Hadoop uses the Capacity Scheduler, but you can switch to an alternative like the Fair Scheduler or configure different scheduling policies. Identifying Your Current Scheduler Check which scheduler is currently active by examining your configuration: grep -A 2 “yarn.resourcemanager.scheduler.class” $HADOOP_HOME/etc/hadoop/yarn-site.xml The…
Amazon S3 has a maximum object size of 5TB per object. This limit applies uniformly across all storage classes and regions — there are no exceptions or workarounds. Understanding the 5TB Ceiling The 5TB limit is a hard constraint at the API level. You cannot upload, copy, or restore an object larger than this. While…
Hadoop uses three primary configuration files to define YARN, HDFS, and MapReduce behavior: HDFS: hdfs-site.xml YARN: yarn-site.xml MapReduce: mapred-site.xml These files live in $HADOOP_HOME/etc/hadoop/ and override the built-in defaults when present. Finding Official Default Values Apache publishes default configuration documentation for each release. For current versions: Hadoop 3.4.x (Latest) HDFS defaults: https://hadoop.apache.org/docs/r3.4.0/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml YARN defaults: https://hadoop.apache.org/docs/r3.4.0/hadoop-yarn/hadoop-yarn-common/yarn-default.xml…
When compiling MapReduce jobs against a Hadoop installation, you need to include the correct classpath to resolve Hadoop dependencies. The yarn classpath command handles this automatically. Getting the classpath Run this command to output the full classpath: yarn classpath If yarn isn’t in your $PATH, use the full path: $HADOOP_HOME/bin/yarn classpath Replace $HADOOP_HOME with your…
SQL layers abstract traditional relational query interfaces over distributed NoSQL and big data systems, enabling applications to leverage SQL’s familiarity while benefiting from horizontal scalability. Here are the key approaches and systems that have emerged in this space. Distributed SQL on Key-Value Stores Phoenix (Apache Phoenix) provides a SQL interface over HBase. It compiles SQL…
If you’re working with Mingle Forum and need to remove canonical links from forum pages while preserving them elsewhere, you’ll need to locate the specific code in your theme’s class file. Finding the Code Reference The line numbers (like -1260) in a diff refer to specific locations within the file being modified. In this case,…
The HP DesignJet 500 and similar wide-format plotters handle posters, technical drawings, and large documents well—but they require careful configuration. This guide covers printing a 24″ × 36″ PowerPoint poster on a 36″ roll, though the process generalizes to other sizes. Understanding Your Hardware Constraints The DesignJet 500 accepts roll media up to 36″ wide….
This post analyzes I/O performance between loopback and blktap backends for Xen file-backed virtual block devices (VBDs), based on testing from 2013. While Xen remains in production use (AWS EC2, Citrix Hypervisor), most Linux deployments now default to KVM. For modern virtualization needs, consider KVM/libvirt or container solutions like Docker and Kubernetes. This post is…
TeraSort is Hadoop’s standard benchmark for measuring distributed sorting performance across clusters. It consists of three components: TeraGen generates random input data, TeraSort performs the distributed sort, and TeraValidate confirms the output is globally sorted. Modern Hadoop deployments typically run on Hadoop 3.x or higher, with improvements in YARN resource management, better performance tuning options,…
Hadoop daemons communicate over specific TCP ports that you’ll need to know for cluster setup, firewall configuration, and application development. These ports are configurable but come with sensible defaults in Hadoop 3.x. HDFS Ports Service Port Configuration Property Purpose Namenode HTTP UI 9870 dfs.namenode.http-address Web UI and metadata operations Namenode HTTPS UI 9871 dfs.namenode.https-address Secure…
Hadoop 3.x remains the current stable version with YARN improvements, HDFS federation, and erasure coding. Hadoop 1.x and 2.x are no longer maintained. For new deployments, evaluate whether Hadoop fits your needs or if managed services like AWS EMR, Google Dataproc, or Azure HDInsight better suit your infrastructure. This post covers common configuration mistakes and…
Hadoop’s default configuration runs in standalone (local) mode as a single JVM process — no HDFS daemons, no YARN, no distributed overhead. This mode is useful for development, testing MapReduce jobs, and debugging before moving to a clustered deployment. For new deployments, evaluate whether you need Hadoop at all. Consider cloud-native alternatives like AWS EMR,…