It is common to execute commands on many nodes/hosts via SSH for managing a cluster of Linux servers. On Linux, there are many choices for this task. Generally, to run commands on many nodes, there are two modes: serial mode and parallel mode. In serial mode, the command is executed on the node one by one. In parallel mode, the command is executed on many nodes together. The serial mode is easy to reason about with and debug while the parallel mode is usually much faster.

In this post, I will introduce 3 methods of executing commands on many nodes on Linux: using Bash, clustershell and pdsh. The methods are introduced by an example: execute the command hostname on nodes “lnode31 lnode6 cluster1-1 cluster1-2 … cluster1-8” as user “root”.

Execute commands using Bash

Run commands in serial order (one by one) using Bash over SSH

for h in lnode31 lnode6 cluster1-{1..8} ; do
    ssh root@$h hostname
done

Run commands in parallel using Bash over SSH

for h in lnode31 lnode6 cluster1-{1..8} ; do
    ssh root@$h hostname &
done
wait

Pros: Bash is almost always available on Linux nodes. You can do certain checking logic after each ssh invoking.

Cons: The length of the command is a little bit long.

Execute commands using clustershell

clustershell/clush is a program for executing commands in parallel on a cluster. clush can also gather the commands’ results. If you haven’t installed it on the managing node, you can install the package clustershell (on Fedora Linux).

Run commands in parallel using clustershell over SSH

$ clush -l root -w lnode31,lnode6,cluster1-[1-8] hostname

Pros: clush is designed for parallel execution. clush can also execute commands interactively.

Cons: You will need to install the software on the managing node.

Execute commands using pdsh

pdsh is a variant of the rsh command while pdsh can run multiple remote commands in parallel. pdsh can also run in interactive mode. If you haven’t installed it on the managing node, you need to install the package pdsh abd pdsh-rcmd-ssh (on Fedora Linux) first.

Run commands in parallel using pdsh over SSH

$ pdsh -R ssh -l root -w lnode31,lnode6,cluster1-[1-8] hostname

For more usage of pdsh, check the pdsh manual page.

Pros and Cons: similar to those of clush. In addition, pdsh support other rcmd modules other than ssh such as rsh and exec.

These 3 methods should help managing a cluster of Linux nodes easier. Having other favorite tools to execute ssh commands? Share it with us by commenting.