Making Hadoop Java process heap larger?

ByEric Ma

In Hadoop 2.5.0, I use ‘ps -aux’ and find the Java process has options:

-Xmx1000m

However, my nodes have 32GB memory.

How to make Hadoop Java process heap larger?

In yarn-env.sh, you can find:

# For setting YARN specific HEAP sizes please use this
# Parameter and set appropriately
# YARN_HEAPSIZE=1000

In hadoop-env.sh, you can find:

# The maximum amount of heap to use, in MB. Default is 1000.
#export HADOOP_HEAPSIZE=
#export HADOOP_NAMENODE_INIT_HEAPSIZE=""

So, uncomment them and change them to the required values you want. Restart Hadoop and it will go.

Read more:

Similar Posts

  • Cache at Facebook

    ByQ A

    About caching system at Facebook. According to: https://www.facebook.com/notes/facebook-engineering/monitoring-cache-with-claspin/10151076705703920 Facebook has two major cache systems: Memcache, which is a simple lookaside cache with most of its smarts in the client, and TAO, a caching graph database that does its own queries to MySQL. The NSDI’13 paper introduces more about Memcache: https://www.usenix.org/conference/nsdi13/scaling-memcache-facebook The USENIX ATC’13 paper introduces…

  • How to redirect HTTP to HTTPS in Apache with mod_rewrite?

    ByEric Ma

    I’d like to force using of https on one of my site. How to redirect HTTP to HTTPS in Apache with mod_rewrite? You can put these lines to the .htaccess file in the directory from which you would like to redirect HTTPS to HTTP: RewriteEngine On RewriteCond %{HTTPS} off RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} Read more: How…

  • How to repair a MySQL table?

    ByQ A

    After a server crash and restarting, MyBB reports a SQL Error as follows: MyBB SQL Error MyBB has experienced an internal SQL error and cannot continue. SQL Error: 145 – Table ‘./mybb/mybb_sessions’ is marked as crashed and should be repaired Query: SELECT * FROM mybb_sessions WHERE sid=’40021925bd0494ea31…’ AND ip=’x.x.x.x’ LIMIT 1 The dababase is MySQL….

  • Managing LVM

    ByQ A

    Any tutorials or tips on how to manage LVM? LVM provides a flexible and easy way to management storage disks on Linux. It also provides a set of tools for management. Here are some tutorials and references for managing LVM: LVM man pages: https://www.systutorials.com/docs/linux/man/8-lvm/#lbAL Manage Disk Volume in Linux: http://www.hongkedavid.com/blog/disk_volume.html LVM with Xen: https://www.systutorials.com/b/linux/tag/lvm/ Remove missing phyiscal volumes…

  • |

    git push error

    ByWeiwei Jia

    $ git push error: The requested URL returned error: 403 Forbidden while accessing https://github.com/HarryWei/hummer.git/info/refs fatal: HTTP request failed Replace (or add) url=ssh://git@github.com/HarryWei/hummer.git with url=https://git@github.com/HarryWei/hummer.git under “[remote “origin”]” section in ~/.gitconfig file. Reference:http://stackoverflow.com/questions/7438313/pushing-to-git-returning-error-code-403-fatal-http-request-failed Read more: Git push error under CENTOS 6.7 Push and Pull data from Restful Service in Asp.net MVC Application using Angular JS Fixing…

  • |

    How to improve video rendering quality in MPlayer

    ByQ A

    MPlayer has lots options for video rendering and filtering. Any suggestions on good MPlayer options that improve video rendering quality nicely? My ~/.mplayer/config is as follows with 2 profiles: [fast] vf=eq2 [default] vf=hqdn3d vo=gl:yuv=3:lscale=5:cscale=5 ao=pulse The default one gives good video quality by using the hqdn3d video filter while the CPU usage is high. If…

Leave a Reply

Your email address will not be published. Required fields are marked *