Best way to connect to hundreds of nodes and grep log files

Shell Programming and Scripting
1

Hi, What will be the best way to connect (ssh) to hundreds of nodes and grep log files parallely from shell. Using for loop seems to be sequential. Are there any shell built in construct which could be used to achieve this? Is the sub shell any good here?

2

There is GNU parallel. It will run a series of commands for you at the same time. I do not recommend hundreds of ssh sessions at once. Keep it down to a reasonable limit.

GNU Parallel - GNU Project - Free Software Foundation

Modern shells also support a way to achieve this. bash example:

cnt=0
while read nodename
do
   cnt=$(( $cnt + 1 ))
   ssh $nodename 'grep "warning" *log' > $nodename.log &
   [  $(( $cnt % 10 )) -eq  0 ] && wait
done < listofnodes.txt

This runs ten operations of the command at one time.

1 Like