Multiple SSH connections at same time

sdosanjh · September 29, 2022, 10:13pm

Hi
I am trying to speed up the script by making SSH connections to different machines at same time and get output instead of making connection to system one at a time.

I tried following but for loop does not serve the purpose.

tried using ssh -f but that goes to background and loop through one at time for systems in HOST variable

HOST="eagle snake ladder"

downProcess()
{
        for process in `cat /tmp/abc |grep -i $HOSTNAME|awk '{print $1}'`
        do
                echo "Maint command run on $process on $HOSTNAME"
                ssh - f $HOSTNAME "echo "i am in $HOSTNAME";" | tee >> $LOG
                sleep 1
        done
}

for HOSTNAME in $HOST
do
echo "============ Started bouncing processes on $HOSTNAME at `date` ==========\n" >> $LOG
echo "Keeping the process in to Maintenance and Disabling processes on $HOSTNAME ...\n" >> $LOG
downProcess;
done

munkeHoller · September 30, 2022, 12:16am

hi, a few things to note.

run any/all scripts through the shellcheck utility before posting - and address any relevant issues it raises. if not installed install or search for the online version.

I think your ssh line has at least 1 syntax error '- f' should read -f unless i'm mistaken ( ie no space between the - and the f ).

ssh - f $HOSTNAME "echo "i am in $HOSTNAME";" | tee >> $LOG

here's a cleaned up attempt , read/digest/test/check/double-check. I've added a bunch more logging,once you are happy the script works to requirements feel free to remove/add/amend as you feel necessary..

you will need to

uncomment the ssh command line
its assumed no password is being prompted for ...
amend the awk command line to match your input file.
I've used a simple 2 field file (embedded in the script)
- to be removed

I've presumed the script will be using the bash shell ....

#!/bin/bash

#
# /tmp/abc - used in this example
# 
cat << EOF > /tmp/abc
1234 eagle
2345 ladder
3456 eagle
4567 ladder
5678 snake
7890 xWing
6789 ladder
7890 snake
7890 rogue1
9876 ladder
8765 snake
EOF

downProcess()
{
        local thisHOST="${1}"
        local logfile="${2}"
        for process in $( awk -vhost="${thisHOST}" '$2 == host { print $1}' /tmp/abc )
        do
                printf "Maint command run on pid i%s on host %s\n"  "${process}" "${thisHOST}" | tee -a "${logfile}"
                #ssh -f "${thisHOST}" 'echo "USER:$(whoami) logged on $(hostname) its $(date)";' | tee -a "${logfile}"
                sleep 1
        done
}

HOSTS="eagle snake ladder"

LOGFILE="${PWD}/$$.log"

printf "%s START of processing\n" "$(date)" | tee "${LOGFILE}"
for host in ${HOSTS}
do
        printf "\nPROCESSING %s\n" "${host}" | tee -a "${LOGFILE}"
        printf "============ Started bouncing processes on %s at %s ==========\n" "${host}" "$(date)" | tee -a "${LOGFILE}"
        printf "Keeping the process in to Maintenance and Disabling processes on %s\n" "${host}" | tee -a "${LOGFILE}"

        downProcess "${host}" "${LOGFILE}" &

done
printf "\n%s  END of processing\n" "$(date)" | tee -a "${LOGFILE}"

cjt007 · September 30, 2022, 7:10am

I would recommend that you think about generating logins via a Script that basically creates multiple screens and each screen can then establish a SSH connection based on its name

It is dangerous but it might make you more able to handle - say a simple
upgrade because most of the work has already been done - just getting
the ssh to open a connection from the multiple screens that you created.
With the proviso - I got caught - never ever let someone harass you
or distract you .

So once you have the screens up you can run the upgrade.

I had this setup.

NOTE doing upgrades is fun but make sure you down load the needed files PRIOR to the upgrade.

Sometimes you may have to move them to another slightly different spot so that upgrade process can find them

Good Luck
Helpful

sdosanjh · September 30, 2022, 8:17pm

Hi
Perhaps I was not clear in my requirement. Sorry about that.

I have 3 hosts
HOST="eagle snake ladder"
And, I am on some jumpbox where all these 3 hosts can be accessible using SSH
say Jumpbox1

Now, I need to make 3 simultaneous connections to $HOST
that is
jumpbox1> ssh eagle
jumpbox1> ssh snake
jumpbox1> ssh ladder

Then execute the same commands on these 3 different hosts and come back to jumpbox1

that is why I was using the function downProcess()
and was thinking of calling that function in for loop
But for is sequential. so I am stuck here on how to fork multiple SSH from jumper to hosts

sdosanjh · September 30, 2022, 8:20pm

munkeHoller:

printf "%s START of processing\n" "$(date)" | tee "${LOGFILE}"
for host in ${HOSTS}
do
        printf "\nPROCESSING %s\n" "${host}" | tee -a "${LOGFILE}"
        printf "============ Started bouncing processes on %s at %s ==========\n" "${host}" "$(date)" | tee -a "${LOGFILE}"
        printf "Keeping the process in to Maintenance and Disabling processes on %s\n" "${host}" | tee -a "${LOGFILE}"

        downProcess "${host}" "${LOGFILE}" &

done

Thank you @munkeHoller
i can try this to call the function in background

printf "%s START of processing\n" "$(date)" | tee "${LOGFILE}"
for host in ${HOSTS}
do
printf "\nPROCESSING %s\n" "${host}" | tee -a "${LOGFILE}"
printf "============ Started bouncing processes on %s at %s ==========\n" "${host}" "$(date)" | tee -a "${LOGFILE}"
printf "Keeping the process in to Maintenance and Disabling processes on %s\n" "${host}" | tee -a "${LOGFILE}"

downProcess "${host}" "${LOGFILE}" &

done

munkeHoller · September 30, 2022, 10:23pm

yes, the line downProcess "${host}" "${LOGFILE}" & executes the commands in a subshell to allow asynchronous processing.

did it work , if not, show issues.

tks

sdosanjh · October 3, 2022, 8:50pm

munkeHoller:

#!/bin/bash

#
# /tmp/abc - used in this example
# 
cat << EOF > /tmp/abc
1234 eagle
2345 ladder
3456 eagle
4567 ladder
5678 snake
7890 xWing
6789 ladder
7890 snake
7890 rogue1
9876 ladder
8765 snake
EOF

downProcess()
{
        local thisHOST="${1}"
        local logfile="${2}"
        for process in $( awk -vhost="${thisHOST}" '$2 == host { print $1}' /tmp/abc )
        do
                printf "Maint command run on pid i%s on host %s\n"  "${process}" "${thisHOST}" | tee -a "${logfile}"
                #ssh -f "${thisHOST}" 'echo "USER:$(whoami) logged on $(hostname) its $(date)";' | tee -a "${logfile}"
                sleep 1
        done
}

HOSTS="eagle snake ladder"

LOGFILE="${PWD}/$.log"

printf "%s START of processing\n" "$(date)" | tee "${LOGFILE}"
for host in ${HOSTS}
do
        printf "\nPROCESSING %s\n" "${host}" | tee -a "${LOGFILE}"
        printf "============ Started bouncing processes on %s at %s ==========\n" "${host}" "$(date)" | tee -a "${LOGFILE}"
        printf "Keeping the process in to Maintenance and Disabling processes on %s\n" "${host}" | tee -a "${LOGFILE}"

        downProcess "${host}" "${LOGFILE}" &

done
printf "\n%s  END of processing\n" "$(date)" | tee -a "${LOGFILE}"

I can see the downProcess function being called in script but the print statement does not print anything from downProcess.
does this mean the function is not being called properly?

here are the details

/tmp/processlist.config contains

process1 snake XXX
process2 snake XXX
process3 snake XXX
process4 snake XXX
process1 eagle XXX
process2 eagle XXX
process3 eagle XXX
process14 eagle XXX

Script details

#!/bin/bash


if [ `whoami` != "dev" ]; then
   echo "\n!!! You need to login as dev user to run this script.!!!\n";
   exit 2;
fi

LOGNAME=`basename $0`                      # Name of the Script
LOGFILE="/tmp/$LOGNAME.log"


# Removing logs older than 5 days

find /tmp/$LOGNAME.log.* -mtime +5 -exec rm {} \;

mv $LOGFILE $LOGFILE.`date +%Y%m%d%H%M`

cat /dev/null >> $LOGFILE
chmod 666 $LOGFILE

ARBHOST="eagle snake"

downProcess()
{
        local thisHOST="${1}"
        local logfile="${2}"
        for process in $( awk -vhost="${thisHOST}" '$2 == host { print $1}' /tmp/processlist.config )
        do
                printf "Maint command run on pid i%s on host %s\n"  "${process}" "${thisHOST}" | tee -a "${logfile}"
                #ssh -f "${thisHOST}" 'echo "USER:$(whoami) logged on $(hostname) its $(date)";' | tee -a "${logfile}"
                sleep 1
        done
}



printf "%s START of processing\n" "$(date)" | tee "${LOGFILE}"
for host in ${ARBHOST}
do
        printf "\nPROCESSING %s\n" "${host}" | tee -a "${LOGFILE}"
        printf "============ Started bouncing processes on %s at %s ==========\n" "${host}" "$(date)" | tee -a "${LOGFILE}"
        printf "Keeping the process in to Maintenance and Disabling processes on %s\n" "${host}" | tee -a "${LOGFILE}"

        downProcess "${host}" "${LOGFILE}" &

done
printf "\n%s  END of processing\n" "$(date)" | tee -a "${LOGFILE}"

OUTPUT

jumpbox1:/tmp> ./test_bounce
Mon Oct  3 20:17:32 GMT 2022 START of processing

PROCESSING eagle
============ Started bouncing processes on eagle at Mon Oct  3 20:17:32 GMT 2022 ==========
Keeping the process in to Maintenance and Disabling processes on eagle

PROCESSING snake
============ Started bouncing processes on snake at Mon Oct  3 20:17:32 GMT 2022 ==========
Keeping the process in to Maintenance and Disabling processes on snake

Mon Oct  3 20:17:32 GMT 2022  END of processing

I don't see this is being printed; so does that means function is not being called?

printf "Maint command run on pid i%s on host %s\n" "${process}" "${thisHOST}" | tee -a "${logfile}"

munkeHoller · October 3, 2022, 10:43pm

I added a printf statement into the downProcess function
added a timestamp and hostname against the printf inside the for process loop , makes it easier to identify log messages as they could be intermingled.
updated code below

downProcess()
{
	printf "\n\ndownProcess invoked\n\n"
	startedAT="$(date +%Y-%m-%d-%H-%M.%S)"
        local thisHOST="${1}"
        local logfile="${2}"
        for process in $( awk -vhost="${thisHOST}" '$2 == host { print $1}' /tmp/processlist.config )
        do
                printf "%s-%s-:Maint command run on pid %s on host %s\n" "${thisHOST}" "${startedAT}" "${process}" "${thisHOST}" | tee -a "${logfile}"
                ssh -f "${thisHOST}" 'echo "USER:$(whoami) logged on $(hostname) its $(date)";' | tee -a "${logfile}"
                sleep 1
        done
}

here's what I get when I run your code ( I named the script users.latest)

./users.latest
Mon  3 Oct 23:12:26 BST 2022 START of processing

PROCESSING eagle
============ Started bouncing processes on eagle at Mon  3 Oct 23:12:26 BST 2022 ==========
Keeping the process in to Maintenance and Disabling processes on eagle

downProcess invoked

PROCESSING snake
============ Started bouncing processes on snake at Mon  3 Oct 23:12:26 BST 2022 ==========
Keeping the process in to Maintenance and Disabling processes on snake

downProcess invoked

eagle-2022-10-03-23-12.26-:Maint command run on pid process1 on host eagle

Mon  3 Oct 23:12:26 BST 2022  END of processing
snake-2022-10-03-23-12.26-:Maint command run on pid process1 on host snake
ssh: Could not resolve hostname eagle: No address associated with hostname
ssh: Could not resolve hostname snake: No address associated with hostname
eagle-2022-10-03-23-12.26-:Maint command run on pid process2 on host eagle
snake-2022-10-03-23-12.26-:Maint command run on pid process2 on host snake
ssh: Could not resolve hostname eagle: No address associated with hostname
ssh: Could not resolve hostname snake: No address associated with hostname
eagle-2022-10-03-23-12.26-:Maint command run on pid process3 on host eagle
snake-2022-10-03-23-12.26-:Maint command run on pid process3 on host snake
ssh: Could not resolve hostname snake: No address associated with hostname
ssh: Could not resolve hostname eagle: No address associated with hostname
snake-2022-10-03-23-12.26-:Maint command run on pid process4 on host snake
eagle-2022-10-03-23-12.26-:Maint command run on pid process14 on host eagle
ssh: Could not resolve hostname snake: No address associated with hostname
ssh: Could not resolve hostname eagle: No address associated with hostname


#########################
# now, lets see what's in the logfile
#

cat /tmp/users.latest.log
Mon  3 Oct 23:12:26 BST 2022 START of processing

PROCESSING eagle
============ Started bouncing processes on eagle at Mon  3 Oct 23:12:26 BST 2022 ==========
Keeping the process in to Maintenance and Disabling processes on eagle

PROCESSING snake
============ Started bouncing processes on snake at Mon  3 Oct 23:12:26 BST 2022 ==========
Keeping the process in to Maintenance and Disabling processes on snake
eagle-2022-10-03-23-12.26-:Maint command run on pid process1 on host eagle

Mon  3 Oct 23:12:26 BST 2022  END of processing
snake-2022-10-03-23-12.26-:Maint command run on pid process1 on host snake
eagle-2022-10-03-23-12.26-:Maint command run on pid process2 on host eagle
snake-2022-10-03-23-12.26-:Maint command run on pid process2 on host snake
eagle-2022-10-03-23-12.26-:Maint command run on pid process3 on host eagle
snake-2022-10-03-23-12.26-:Maint command run on pid process3 on host snake
snake-2022-10-03-23-12.26-:Maint command run on pid process4 on host snake
eagle-2022-10-03-23-12.26-:Maint command run on pid process14 on host eagle

sdosanjh · October 4, 2022, 9:20pm

Thank you for helping me here, but i am not getting same results as yours.

here is my output using same down process function

Jumpbox1> ./test_bounce
Tue Oct  4 20:53:57 GMT 2022 START of processing

PROCESSING eagle
============ Started bouncing processes on eagle at Tue Oct  4 20:53:57 GMT 2022 ==========
Keeping the process in to Maintenance and Disabling processes on eagle


downProcess invoked


PROCESSING snake
============ Started bouncing processes on snake at Tue Oct  4 20:53:57 GMT 2022 ==========
Keeping the process in to Maintenance and Disabling processes on snake


downProcess invoked


Tue Oct  4 20:53:57 GMT 2022  END of processing

The print statement in for loop is not being executed

CODE

downProcess()
{
        printf "\n\ndownProcess invoked\n\n"
        startedAT="$(date +%Y-%m-%d-%H-%M.%S)"
        local thisHOST="${1}"
        local logfile="${2}"
        for process in $( awk -vhost="${thisHOST}" '$2 == host { print $1}' /tmp/processlist.config )
        do
                printf "%s-%s-:Maint command run on pid %s on host %s\n" "${thisHOST}" "${startedAT}" "${process}" "${thisHOST}" | tee -a "${logfile}"
                ssh "${thisHOST}" 'echo "USER:$(whoami) logged on $(hostname) its $(date)";' | tee -a "${logfile}"
                sleep 1
        done
}


printf "%s START of processing\n" "$(date)" | tee "${LOGFILE}"
for host in ${ARBHOST}
do
        printf "\nPROCESSING %s\n" "${host}" | tee -a "${LOGFILE}"
        printf "============ Started bouncing processes on %s at %s ==========\n" "${host}" "$(date)" | tee -a "${LOGFILE}"
        printf "Keeping the process in to Maintenance and Disabling processes on %s\n" "${host}" | tee -a "${LOGFILE}"

        downProcess "${host}" "${LOGFILE}" &

done
printf "\n%s  END of processing\n" "$(date)" | tee -a "${LOGFILE}"

Just tried manually passing values to variables in the function
the for loop returns nothing in my case because awk returns nothing in my code.

jumpbox1:/tmp> thisHOST="eagle"
jumpbox1:/tmp> awk -vhost="${thisHOST}" '$2 == host { print $1}' /tmp/processlist.config
jumpbox1:/tmp>

jumpbox1:/tmp> cat /tmp/processlist.config
process1 snake XXX
process2 snake XXX
process3 snake XXX
process4 snake XXX
process1 eagle XXX
process2 eagle XXX
process3 eagle XXX
process14 eagle XXX

The simple awk works fine

awk '/'${thisHOST}'/ {print $1}'  /tmp/processlist.config

MadeInGermany · October 4, 2022, 10:09pm

Which of the following works for you?
Least precise, search in the whole line
awk '/'${thisHOST}'/ {print $1}' ...
More precise, search in field 2 only
awk -v host="${thisHOST}" '$2 ~ host { print $1}' ...
Most precise, the whole field 2 must match
awk -v host="${thisHOST}" '$2 == host { print $1}' ...

munkeHoller · October 4, 2022, 11:00pm

that line
awk '/'${thisHOST}'/ {print $1}' /tmp/processlist.config is not what the for loop operates on in the downProcess function - you need to test against the code that is being executed not something else.

What does running the below show ?

thisHOST=eagle
echo $( awk -vhost="${thisHOST}" '$2 == host { print $1}' /tmp/processlist.config)

for me it outputs the following

process1 process2 process3 process14

if you get something different then perhaps your field separator is not a tab or space or the file is not standard ascii text (I also tried with file in 'dos' format - (ascii with CRLF terminator and it works)), if the file is UTF-16 (nothing returned if the file is in this format) (or some other character set then I don't see it working).
.... what do the following produce

file /tmp/processlist.config

cat -A /tmp/processlist.config

sdosanjh · October 5, 2022, 8:33pm

The file type is ascii text
but does not get the same output as yours

jumpbox1:/tmp> thisHOST=eagle
jumpbox1:/tmp> echo $( awk -vhost="${thisHOST}" '$2 == host { print $1}' /tmp/processlist.config )

jumpbox1:/tmp>

jumpbox1:/tmp> file /tmp/processlist.config
/tmp/processlist.config:     ascii text

munkeHoller · October 5, 2022, 8:57pm

hmm, if your code works for me but not for you then theres some gremlin somewhere at play.

you said this worked
awk '/'${thisHOST}'/ {print $1}' /tmp/processlist.config

.... (but showed no output !!), please run against the file and copy/paste all output - including the command line.

additionally,
cat the file please.
try with a file with only a single entry
try with a command you have tested and returns what you expect (share that please)
you can test the function in isolation , just set variables to suit then invoke .

vgersh99 · October 5, 2022, 9:26pm

provide the output of cat -vet /tmp/processlist.config (using markdown code tags).
also try - notice the trailing space after -v:

thisHOST=eagle
awk -v host="${thisHOST}" '$2 == host { print $1}' /tmp/processlist.config

munkeHoller · October 5, 2022, 10:08pm

you haven't shown the output from

cat -A /tmp/processlist.config

as requested, please do that

sdosanjh · October 5, 2022, 11:25pm

That option does not work might be because it is SunOS 5.11

jumpbox1:/tmp> cat -A /tmp/processlist.config
cat: illegal option -- A
usage: cat [ -usvtebn ] [-|file] ...

jumpbox1:/tmp> uname -a
SunOS jumpbox1 5.11 11.4.44.113.4 i86pc i386 i86pc non-global-zone

Work for me is only the least precise one:

jumpbox1:/tmp> thisHOST=eagle
jumpbox1:/tmp> awk '/'${thisHOST}'/ {print $1}' /tmp/processlist.config
process1
process2
process3
process14
jumpbox1:/tmp>

Moderate and fully precise returns nothing

jumpbox1:/tmp> thisHOST=eagle
jumpbox1:/tmp> echo $( awk -vhost="${thisHOST}" '$2 == host { print $1}' /tmp/processlist.config )

jumpbox1:/tmp> echo $( awk -vhost="${thisHOST}" '$2 ~ host { print $1}' /tmp/processlist.config )
awk: syntax error near line 1
awk: bailing out near line 1

jumpbox1:/tmp> echo $( awk -v host="${thisHOST}" '$2 ~ host { print $1}' /tmp/processlist.config )
awk: syntax error near line 1
awk: bailing out near line 1

cat -vet output

jumpbox1:/tmp> cat -vet /tmp/processlist.config
process1 snake XXX$
process2 snake XXX$
process3 snake XXX$
process4 snake XXX$
process1 eagle XXX$
process2 eagle XXX$
process3 eagle XXX$
process14 eagle XXX$
jumpbox1:/tmp>

sdosanjh · October 5, 2022, 11:30pm

sdosanjh:

jumpbox1:/tmp> cat -vet /tmp/processlist.config
process1 snake XXX$
process2 snake XXX$
process3 snake XXX$
process4 snake XXX$
process1 eagle XXX$
process2 eagle XXX$
process3 eagle XXX$
process14 eagle XXX$
jumpbox1:/tmp>

Output of cat -vet

jumpbox1:/tmp> cat -vet /tmp/processlist.config
process1 snake XXX$
process2 snake XXX$
process3 snake XXX$
process4 snake XXX$
process1 eagle XXX$
process2 eagle XXX$
process3 eagle XXX$
process14 eagle XXX$
jumpbox1:/tmp>

munkeHoller · October 5, 2022, 11:35pm

logic dictates that if the line with the 'least' precision works and does what you need to get done then use that.

vgersh99 · October 6, 2022, 12:11am

Well... You should have mentioned this...
Use nawk instead of awk.
( and space after -v).

P.S. this thread is in Solaris forum. I guess our collective eyesight isn't as sharp as it used to be...

munkeHoller · October 6, 2022, 7:43pm

@sdosanjh , is this matter now concluded ?