SysTutorials

Setting Up a Hadoop 2.x Cluster for Development and Legacy Systems

· updated April 11, 2026

Hadoop 2.x reached end-of-life in 2016. This guide covers setup for learning purposes and legacy system maintenance only. For production deployments, use Hadoop 3.x or later, which includes performance improvements, better YARN scheduling, HDFS erasure coding, and improved security. Cloud-managed options like AWS EMR, Google Dataproc, and Azure HDInsight eliminate most operational overhead.

This guide walks through setting up a three-node cluster (hofstadter, snell, biot) with hofstadter as the NameNode/ResourceManager. Hadoop 2.x introduced YARN (Yet Another Resource Negotiator), decoupling resource management from MapReduce job scheduling — a significant architectural change from 1.x.

Prerequisites

Hostname and Network Configuration

Hadoop uses hostnames for node discovery. Edit /etc/hosts on every node:

10.0.3.29   hofstadter
10.0.3.30   snell
10.0.3.31   biot

Verify connectivity:

ping -c 1 hofstadter
ping -c 1 snell
ping -c 1 biot

For clusters with 10+ nodes, use DNS instead of /etc/hosts to reduce maintenance burden.

SSH Key-Based Authentication

The Hadoop user on the NameNode must SSH to all DataNodes without password prompts. Generate Ed25519 keys as the hadoop user on the NameNode:

su - hadoop
ssh-keygen -t ed25519 -N "" -f ~/.ssh/id_ed25519

Distribute the public key to all nodes:

for node in snell biot; do
  ssh-copy-id -i ~/.ssh/id_ed25519.pub hadoop@$node
done

Test passwordless login:

ssh hadoop@snell hostname
ssh hadoop@biot hostname

Both commands should return the node’s hostname without prompting for a password.

Java Installation

Hadoop requires a JDK, not just a JRE. Install via your package manager or manually:

# Using package manager (recommended)
apt-get install openjdk-11-jdk  # Debian/Ubuntu
# or
dnf install java-11-openjdk-devel  # RHEL/CentOS/Fedora

For manual installation, download from the OpenJDK repository and extract to /usr/java/:

mkdir -p /usr/java
cd /usr/java
# Download and extract JDK tarball
ln -s jdk-21.x.x default

Add to ~/.bashrc for the hadoop user on all nodes:

export JAVA_HOME=/usr/java/default
export PATH=$JAVA_HOME/bin:$PATH

Source and verify:

source ~/.bashrc
java -version

Hadoop Installation

Download Hadoop 2.10.1 (final 2.x release) from the Apache archive:

cd /home/hadoop
wget https://archive.apache.org/dist/hadoop/common/hadoop-2.10.1/hadoop-2.10.1.tar.gz
tar xzf hadoop-2.10.1.tar.gz
ln -s hadoop-2.10.1 hadoop

Add to ~/.bashrc for the hadoop user on all nodes:

export HADOOP_COMMON_HOME=$HOME/hadoop
export HADOOP_MAPRED_HOME=$HADOOP_COMMON_HOME
export HADOOP_HDFS_HOME=$HADOOP_COMMON_HOME
export YARN_HOME=$HADOOP_COMMON_HOME
export PATH=$PATH:$HADOOP_COMMON_HOME/bin:$HADOOP_COMMON_HOME/sbin

Reload and verify:

source ~/.bashrc
hadoop version

Configuration

All configuration files live in $HADOOP_COMMON_HOME/etc/hadoop/. Edit .xml files directly with a text editor — do not rely on shell scripts to modify them, as manual edits may be lost.

core-site.xml

Set the NameNode address. Add this inside the <configuration> tags:

<property>
  <name>fs.defaultFS</name>
  <value>hdfs://hofstadter/</value>
  <description>NameNode URI</description>
</property>

hdfs-site.xml

Configure HDFS replication and storage directories:

<property>
  <name>dfs.replication</name>
  <value>3</value>
  <description>Number of block replicas</description>
</property>
<property>
  <name>dfs.namenode.name.dir</name>
  <value>file:///home/hadoop/hdfs/namenode</value>
  <description>NameNode metadata storage</description>
</property>
<property>
  <name>dfs.datanode.data.dir</name>
  <value>file:///home/hadoop/hdfs/datanode</value>
  <description>DataNode block storage</description>
</property>

Create these directories on all nodes:

mkdir -p /home/hadoop/hdfs/namenode /home/hadoop/hdfs/datanode
chown hadoop:hadoop /home/hadoop/hdfs

Set dfs.replication to match your number of DataNodes or fewer. A three-node cluster can use replication factor 3; smaller clusters should lower this value.

yarn-site.xml

Configure YARN resource management:

<property>
  <name>yarn.resourcemanager.hostname</name>
  <value>hofstadter</value>
  <description>ResourceManager host</description>
</property>
<property>
  <name>yarn.nodemanager.aux-services</name>
  <value>mapreduce_shuffle</value>
  <description>Shuffle service for MapReduce</description>
</property>
<property>
  <name>yarn.nodemanager.resource.memory-mb</name>
  <value>4096</value>
  <description>Memory available per NodeManager in MB</description>
</property>
<property>
  <name>yarn.scheduler.minimum-allocation-mb</name>
  <value>512</value>
  <description>Minimum memory per container in MB</description>
</property>
<property>
  <name>yarn.scheduler.maximum-allocation-mb</name>
  <value>4096</value>
  <description>Maximum memory per container in MB</description>
</property>

Adjust memory values based on actual available RAM. A common formula: (total_ram – 2GB_system_overhead) = yarn.nodemanager.resource.memory-mb. If a node has 8GB RAM, set this to 6144 MB.

mapred-site.xml

Copy the template and configure MapReduce to use YARN:

cp $HADOOP_COMMON_HOME/etc/hadoop/mapred-site.xml.template \
   $HADOOP_COMMON_HOME/etc/hadoop/mapred-site.xml

Add to mapred-site.xml:

<property>
  <name>mapreduce.framework.name</name>
  <value>yarn</value>
  <description>Use YARN for MapReduce</description>
</property>
<property>
  <name>mapreduce.jobhistory.address</name>
  <value>hofstadter:10020</value>
  <description>Job history server address</description>
</property>
<property>
  <name>mapreduce.jobhistory.webapp.address</name>
  <value>hofstadter:19888</value>
  <description>Job history web UI</description>
</property>
<property>
  <name>mapreduce.map.memory.mb</name>
  <value>512</value>
  <description>Memory per map task</description>
</property>
<property>
  <name>mapreduce.reduce.memory.mb</name>
  <value>512</value>
  <description>Memory per reduce task</description>
</property>

These map/reduce memory settings should not exceed yarn.scheduler.maximum-allocation-mb.

workers (formerly slaves)

Edit etc/hadoop/workers and list all DataNode hostnames:

hofstadter
snell
biot

The NameNode/ResourceManager can also run DataNode/NodeManager services (as shown here), though production deployments often dedicate the NameNode to metadata management only.

Distribute Configuration to All Nodes

Use rsync to copy Hadoop configuration to all DataNodes:

cd /home/hadoop
for node in snell biot; do
  echo "Syncing to $node..."
  rsync -avxz --delete hadoop/ hadoop@$node:/home/hadoop/hadoop/
done

Verify on each node:

ssh hadoop@snell ls /home/hadoop/hadoop/etc/hadoop/core-site.xml
ssh hadoop@biot ls /home/hadoop/hadoop/etc/hadoop/core-site.xml

Format the NameNode

On the NameNode only, initialize HDFS metadata:

hdfs namenode -format

This creates the initial fsimage. Do this only once. Re-formatting destroys all HDFS data and cannot be undone.

Start the Cluster

Option 1: Individual Service Control

On the NameNode:

hdfs --daemon start namenode

On each DataNode:

hdfs --daemon start datanode

On the ResourceManager (usually the NameNode):

yarn --daemon start resourcemanager

On each NodeManager:

yarn --daemon start nodemanager

The --daemon flag runs the service in the background. Older Hadoop versions used start-dfs.sh and start-yarn.sh scripts, which still work but are less flexible.

Option 2: Use Cluster Start Scripts

From the NameNode, start HDFS across all nodes listed in workers:

start-dfs.sh

This command uses SSH to launch NameNode and DataNode services. Similarly, start YARN:

start-yarn.sh

Run these scripts from the NameNode to ensure proper SSH communication.

Verify Cluster Status

Check HDFS health:

hdfs dfsadmin -report

Sample output:

Configured Capacity: 335,854,977,024 (312.59 GB)
Present Capacity: 335,854,977,024 (312.59 GB)
DFS Remaining: 335,854,977,024 (312.59 GB)
DFS Used: 0 (0 B)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0

Live datanodes (3):
Name: 10.0.3.30:50010 (snell)
Hostname: snell
...

Check YARN node status:

yarn node -list

Sample output:

Total Nodes:3
         Node-Id            Node-State Node-Http-Address   Number-of-Running-Containers
hofstadter:43469           RUNNING    hofstadter:8042                              0
snell:57039                RUNNING    snell:8042                                    0
biot:52834                 RUNNING    biot:8042                                     0

All nodes should report RUNNING state.

Web UIs

Access cluster dashboards via browser:

Run a Test Job

Create HDFS home directory:

hadoop fs -mkdir -p /user/hadoop

Upload test data:

hadoop fs -put $HADOOP_COMMON_HOME/etc/hadoop test-input

Run the built-in grep example:

hadoop jar $HADOOP_COMMON_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar \
  grep test-input test-output 'dfs[a-z.]*'

Monitor the job at http://hofstadter:8088/. Once complete, check results:

hadoop fs -ls test-output/
hadoop fs -cat test-output/part-r-00000

The output will list all lines from the input containing the regex pattern.

Stop the Cluster

Option 1: Individual Service Control

On each node, stop services in reverse order:

# On each DataNode
hdfs --daemon stop datanode

# On NameNode
hdfs --daemon stop namenode

# On each NodeManager
yarn --daemon stop nodemanager

# On ResourceManager
yarn --daemon stop resourcemanager

Option 2: Use Cluster Stop Scripts

From the NameNode:

stop-dfs.sh
stop-yarn.sh

These scripts use SSH to stop services across all nodes in workers.

Debugging and Troubleshooting

Check logs on any node:

ls $HADOOP_COMMON_HOME/logs/
tail -f $HADOOP_COMMON_HOME/logs/hadoop-hadoop-namenode-hofstadter.log
tail -f $HADOOP_COMMON_HOME/logs/hadoop-hadoop-datanode-snell.log

Connection Refused / SSH Failures

Port Conflicts

Check if ports are in use:

netstat -tlnp | grep :9000
lsof -i :8088

Java Not Found

Blocks Under-Replicated

NameNode Safemode

On startup, HDFS enters safemode while blocks are inventoried. Check status:

hdfs dfsadmin -safemode get

Force exit (only if blocks are verified):

hdfs dfsadmin -safemode leave

Full Cluster Reset

To reset from scratch:

# Stop all services
stop-dfs.sh
stop-yarn.sh

# Clean data on NameNode
rm -rf /home/hadoop/hdfs /home/hadoop/hadoop/logs

# Clean data on all DataNodes
for node in snell biot; do
  ssh hadoop@$node 'rm -rf /home/hadoop/hdfs /home/hadoop/hadoop/logs'
done

# Re-format and restart
hdfs namenode -format
start-dfs.sh
start-yarn.sh

Configuration Reference

Parameter File Purpose Example
fs.defaultFS core-site.xml NameNode address hdfs://hofstadter/
dfs.replication hdfs-site.xml Block replication factor 3
`d      
← All posts