How to set the number of mappers and reducers of Hadoop in command line?
Number of mappers and reducers can be set like (5 mappers, 2 reducers):
-D mapred.map.tasks=5 -D mapred.reduce.tasks=2
in the command line.
In the code, one can configure JobConf variables.
job.setNumMapTasks(5); // 5 mappers
job.setNumReduceTasks(2); // 2 reducers
Note that on Hadoop 2 (YARN), the mapred.map.tasks
and mapred.reduce.tasks
are deprecated and are replaced by other variables:
mapred.map.tasks --> mapreduce.job.maps
mapred.reduce.tasks --> mapreduce.job.reduces
Using map reduce.job.maps on command line does not work. Is there a particular syntax to use?
You can add the options to the command like
bin/hadoop jar -Dmapreduce.job.maps=5 yourapp.jar ...