How to set the number of mappers and reducers of Hadoop in command line?

How to set the number of mappers and reducers of Hadoop in command line?

asked Jul 23, 2014 by Eric Z Ma (44,280 points)

1 Answer

 
Best answer

Number of mappers and reducers can be set like (5 mappers, 2 reducers):

-D mapred.map.tasks=5 -D mapred.reduce.tasks=2

in the command line.

In the code, one can configure JobConf variables.

job.setNumMapTasks(5); // 5 mappers
job.setNumReduceTasks(2); // 2 reducers

Note that on Hadoop 2 (YARN), the mapred.map.tasks and mapred.reduce.tasks are deprecated and are replaced by other variables:

mapred.map.tasks     -->	mapreduce.job.maps
mapred.reduce.tasks  -->	mapreduce.job.reduces
answered Jul 23, 2014 by Eric Z Ma (44,280 points)
edited Jul 23, 2014 by Eric Z Ma

Using map reduce.job.maps on command line does not work. Is there a particular syntax to use?

commented Feb 22, 2016 by Maria Carmela Vitell (100 points)

You can add the options to the command like

bin/hadoop jar -Dmapreduce.job.maps=5 yourapp.jar ...
commented Apr 7, 2016 by Eric Z Ma (44,280 points)

Please log in or register to answer this question.

Copyright © SysTutorials. User contributions licensed under cc-wiki with attribution required.
Hosted on Dreamhost

...