How to set the number of mappers and reducers of Hadoop in command line?

How to set the number of mappers and reducers of Hadoop in command line?

Number of mappers and reducers can be set like (5 mappers, 2 reducers):

-D mapred.map.tasks=5 -D mapred.reduce.tasks=2

in the command line.

In the code, one can configure JobConf variables.

job.setNumMapTasks(5); // 5 mappers
job.setNumReduceTasks(2); // 2 reducers

Note that on Hadoop 2 (YARN), the mapred.map.tasks and mapred.reduce.tasks are deprecated and are replaced by other variables:

mapred.map.tasks     -->	mapreduce.job.maps
mapred.reduce.tasks  -->	mapreduce.job.reduces

Using map reduce.job.maps on command line does not work. Is there a particular syntax to use?


You can add the options to the command like

bin/hadoop jar -Dmapreduce.job.maps=5 yourapp.jar ...

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *