Three Methods of Executing Commands on Many Nodes in Parallel via SSH on Linux

It is common to execute commands on many nodes/hosts via SSH for managing a cluster of Linux servers. On Linux, there are many choices for this task. Generally, to run commands on many nodes, there are two modes: serial mode and parallel mode. In serial mode, the command is executed on the node one by one. In parallel mode, the command is executed on many nodes together. The serial mode is easy to reason about with and debug while the parallel mode is usually much faster.

In this post, I will introduce 3 methods of executing commands on many nodes on Linux: using Bash, clustershell and pdsh. The methods are introduced by an example: execute the command hostname on nodes “lnode31 lnode6 cluster1-1 cluster1-2 … cluster1-8” as user “root”.

Execute commands using Bash

Run commands in serial order (one by one) using Bash over SSH

for h in lnode31 lnode6 cluster1-{1..8} ; do
    ssh root@$h hostname
done

Run commands in parallel using Bash over SSH

for h in lnode31 lnode6 cluster1-{1..8} ; do
    ssh root@$h hostname &
done
wait

Pros: Bash is almost always available on Linux nodes. You can do certain checking logic after each ssh invoking.

Cons: The length of the command is a little bit long.

Execute commands using clustershell

clustershell/clush is a program for executing commands in parallel on a cluster. clush can also gather the commands’ results. If you haven’t installed it on the managing node, you can install the package clustershell (on Fedora Linux).

Run commands in parallel using clustershell over SSH

$ clush -l root -w lnode31,lnode6,cluster1-[1-8] hostname

Pros: clush is designed for parallel execution. clush can also execute commands interactively.

Cons: You will need to install the software on the managing node.

Execute commands using pdsh

pdsh is a variant of the rsh command while pdsh can run multiple remote commands in parallel. pdsh can also run in interactive mode. If you haven’t installed it on the managing node, you need to install the package pdsh abd pdsh-rcmd-ssh (on Fedora Linux) first.

Run commands in parallel using pdsh over SSH

$ pdsh -R ssh -l root -w lnode31,lnode6,cluster1-[1-8] hostname

For more usage of pdsh, check the pdsh manual page.

Pros and Cons: similar to those of clush. In addition, pdsh support other rcmd modules other than ssh such as rsh and exec.

These 3 methods should help managing a cluster of Linux nodes easier. Having other favorite tools to execute ssh commands? Share it with us by commenting.

Eric Ma

Eric is a systems guy. Eric is interested in building high-performance and scalable distributed systems and related technologies. The views or opinions expressed here are solely Eric's own and do not necessarily represent those of any third parties.

3 comments:

  1. But, what if you want to remotely run many more commands, if statements, while loops, etc., and make it all readable?

    1. The bash method can works well for this. On one node, you can run a set of bash code from STDIN like

      echo 'for i in {1..8}; do echo $i; done;' | ssh node1 'bash -s'
      

      Here, the `bash -s` executes command from STDIN which is passed through the SSH tunnel.

Leave a Reply

Your email address will not be published. Required fields are marked *