How to set the number of mappers and reducers of Hadoop in command line?

How to set the number of mappers and reducers of Hadoop in command line?

Number of mappers and reducers can be set like (5 mappers, 2 reducers):

-D mapred.map.tasks=5 -D mapred.reduce.tasks=2

in the command line.

In the code, one can configure JobConf variables.

job.setNumMapTasks(5); // 5 mappers
job.setNumReduceTasks(2); // 2 reducers

Note that on Hadoop 2 (YARN), the mapred.map.tasks and mapred.reduce.tasks are deprecated and are replaced by other variables:

mapred.map.tasks     -->	mapreduce.job.maps
mapred.reduce.tasks  -->	mapreduce.job.reduces
Answered by Eric Z Ma.

Using map reduce.job.maps on command line does not work. Is there a particular syntax to use?


You can add the options to the command like

bin/hadoop jar -Dmapreduce.job.maps=5 yourapp.jar ...

Eric Z Ma

Eric is a father and systems guy. Eric is interested in building high-performance and scalable distributed systems and related technologies. The views or opinions expressed here are solely Eric's own and do not necessarily represent those of any third parties.

Leave a Reply

Your email address will not be published. Required fields are marked *