Assistance to connect to servers via ssh once and collect various commands into separate variables

Hello,

We use as bash script to connect to servers listed in an .csv file to issue commands against these servers and retrieve data to be saved in a .csv file. The data we want to collect is saved in variables. We issue an ssh command for each variable we want to capture. I'm thinking this is inefficient and slow and perhaps there is a better way to do this by connecting to the server through ssh once and issuing all the commands we need and still being able to save the commands in individual variables for our output file so I can decide in the output file to place the new commands we've captured as well as list some data from the input file into our final report.

Would someone be able to point me in the right direction on how I can connect once to a server via ssh, issue multiple commands to that server yet save the results from each command to their individual variable?

Here is what our script looks like:

#!/bin/bash

SOURCE_FILE=/home/user/scripts/input.csv
if test -f "$SOURCE_FILE"; then
    echo "$SOURCE_FILE exists"
fi

REPORT_FILE="/home/user/scripts/Report_$(date +"%F_%T").csv"

echo $REPORT_FILE

UNREACHABLE="255"
INVALID_PASSWORD="5"
NO_HOME_DIRECTORY="2"
PASSWORD_EXPIRED="1"
FLAVOUR=""
HOSTNAME=""

touch $REPORT_FILE

       echo "Input Hostname,\
Input IP,\
SSH Access,\
Tag,\
OS Type,\
Hostname,\
IP Adress,\
Uptime,\
ACTIVE KERNEL,\
INACTIVE KERNELS"  >> $REPORT_FILE

exec < $SOURCE_FILE || exit 1
read header # read (and ignore) the first line
while IFS=, read -r InputHost\
InputIP\
Tag\
UserAgentTag\
ResourceProvisioner\
OSType\
OSVersion\
OSDetail\
Location\
PowerState\
; do
        sshpass -p '********' ssh -qno StrictHostKeyChecking=no -o ConnectTimeout=1 userlogin@$InputIP 'ls -l /home/userlogin >/dev/null 2>&1' > /dev/null 2>&1
       
        status="$(echo $?)"
       
        echo "processing $InputHost"
       
        FLAVOUR=$(sshpass -p '********' ssh -qno StrictHostKeyChecking=no -o ConnectTimeout=1 userlogin@$InputIP 'cat /etc/redhat-release 2>/dev/null || lsb_release -a 2>/dev/null | grep Description | cut -f2')
       
        HOSTNAME=$(sshpass -p '********' ssh -qno StrictHostKeyChecking=no -o ConnectTimeout=1 userlogin@$InputIP 'hostname')

        IPADDRESS=$(sshpass -p '********' ssh -qno StrictHostKeyChecking=no -o ConnectTimeout=1 userlogin@$InputIP 'hostname -I')
       
        ACTIVE_KERNEL=$(sshpass -p '********' ssh -qno StrictHostKeyChecking=no -o ConnectTimeout=1 userlogin@$InputIP 'uname -r')

        INACTIVE_KERNEL=$(sshpass -p '********' ssh -qno StrictHostKeyChecking=no -o ConnectTimeout=1 userlogin@$InputIP 'rpm -qa | grep '^kernel-[0-9]' |grep -vE `uname -r`')
       
        UPTIME=$(sshpass -p '********' ssh -qno StrictHostKeyChecking=no -o ConnectTimeout=1 userlogin@$InputIP 'uptime | cut -d "," -f 1')


        if [ "$status" == "$UNREACHABLE" ]
        then
                echo "$InputHost,\
                $InputIP,\
                cannot connect,\
                $Tag,\
                $FLAVOUR,\
                $HOSTNAME,\
                $IPADDRESS,\
                $UPTIME,\
                "$ACTIVE_KERNEL,\
                $INACTIVE_KERNEL >> $REPORT_FILE
        elif [ "$status" == "$INVALID_PASSWORD" ]
        then
                echo "$InputHost,\
                $InputIP,\
                invalid account access,\
                $Tag,\
                $FLAVOUR,\
                $HOSTNAME,\
                $IPADDRESS,\
                $UPTIME,\
                "$ACTIVE_KERNEL,\
                $INACTIVE_KERNEL >> $REPORT_FILE
        elif [ "$status" == "$NO_HOME_DIRECTORY" ]
        then
                echo "$InputHost,\
                $Tag,\
                $FLAVOUR,\
                $HOSTNAME,\
                $IPADDRESS,\
                $UPTIME,\
                "$ACTIVE_KERNEL,\
                $INACTIVE_KERNEL >> $REPORT_FILE

        elif [ "$status" == "$PASSWORD_EXPIRED" ]
        then
                echo "$InputHost,\
                $InputIP,\
                password expired,\
                $Tag,\
                $FLAVOUR,\
                $HOSTNAME,\
                $IPADDRESS,\
                $UPTIME,\
                "$ACTIVE_KERNEL,\
                $INACTIVE_KERNEL >> $REPORT_FILE
        else
             echo "$InputHost,\
                $InputIP,\
                successful ssh access,\
                $Tag,\
                $FLAVOUR,\
                $HOSTNAME,\
                $IPADDRESS,\
                $UPTIME,\
                "$ACTIVE_KERNEL,\
                $INACTIVE_KERNEL >> $REPORT_FILE

        fi
done < $SOURCE_FILE

Thank you.

If it was me....

I would write a script on the server side to read all the vars you are interested in and store the vars in a JSON file where the filename contains the timestamp, or put the timestamp in a JSON var as well..

Then I would have a remote call to get the JSON file and process the JSON file locally.

Thank you @Neo for your reply and suggestion.

I do see the benefit in doing this on each server. From my usecase we won't have this server script on every newly built server and I only scan a certain list of servers based on another report which I then use as my input file. For my usecase I will need to continue with using the script as we've currently written it to reach out to each server, collect data into variables and concatenate this collected data along with other variables from my input file.

I'm hoping there is a way to issue one ssh connection to each server and issue various command line instructions like I do in the script I've provided and collect each result in their own variable.

Thanks in advance for any help anyone can provide me.

You could write a script (local or remote) and execute that, or use "here documents" like

IFS=$'\t' read FLAVOUR HOSTNAME IPADDRESS ACTIVE_KERNEL INACTIVE_KERNEL UPTIME <<- EOFREAD
        $(ssh -q user@targethost <<- EOFSSH | tr $'\n' $'\t'
        cat /etc/redhat-release 2>/dev/null || lsb_release -a 2>/dev/null | grep Description | cut -f2
        hostname
        hostname -I
        uname -r
        rpm -qa | grep '^kernel-[0-9]' |grep -vE `uname -r`
        uptime | cut -d "," -f1
        EOFSSH
        )
 EOFREAD

Try this to improve readability:

case "$status" in
        $UNREACHABLE)           OUT="cannot connect";;
        $INVALID_PASSWORD)      OUT="invalid account access";;
        $NO_HOME_DIRECTORY)     OUT="";;
        $PASSWORD_EXPIRED)      OUT="password expired";;
        $SUCCESS)               OUT="successful ssh access";;
esac
echo "$InputHost,\
      $InputIP,\
      $OUT,\
      $Tag,\
      $FLAVOUR,\
      $HOSTNAME,\
      $IPADDRESS,\
      $UPTIME,\
      $ACTIVE_KERNEL,\
      $INACTIVE_KERNEL" 

(or use a shell array)

You duplicate the stdin redirection by exec < $SOURCE_FILE before the while loop and then redirecting the loop's stdin again.

Thank you @RudiC for the reply! Very much appreciate you taking the time to review my post. I will also look into using the case statement to clean up the code readability!

For this code suggestion, I would like to clarify a few things.

Could I still use sshpass when connecting and could I use the $InputIP from the input file? So would your code suggestion look like the following:

IFS=$'\t' read FLAVOUR HOSTNAME IPADDRESS ACTIVE_KERNEL INACTIVE_KERNEL UPTIME <<- EOFREAD
        $(sshpass -p '*******' ssh -qno StrictHostKeyChecking=no -o ConnectTimeout=1 root@$InputIP <<- EOFSSH | tr $'\n' $'\t'

Secondly...each command instruction, I would like to assign a variable to each. Would I be able to do that as follows based on your code suggestion:

        FLAVOUR=$(cat /etc/redhat-release 2>/dev/null || lsb_release -a 2>/dev/null | grep Description | cut -f2)
        HOSTNAME=$(hostname)
        IPADDRESS=$(hostname -I)
        ACTIVE_KERNEL=$(uname -r)
        INACTIVE_KERNEL=$(rpm -qa | grep '^kernel-[0-9]' |grep -vE `uname -r`)
        UPTIME=$(uptime | cut -d "," -f1)

The reason I would like to assign variables to each result is so that I may place these into the final report where necessary.

Thanks!

Not sure I understand your questions / problems.
Of course, you can adapt the "command substitution" $(...) to fit your needs - in fact, you are required to do so as I only gave a proof of concept.
And, the read VAR1 VAR2 ... assigns each individual command line's output to its respective individual variable - did you try that? The way you propose can't be used as those variables used it the one single ssh session would be defined on the remote system and not expandable (not even existent) on your local system.

Hello RudiC,

I'm finally getting around to testing the code snippets you've provided. Here is the code I've added to my script as per the suggestion. The code is allowing me to connect to the list of servers from the $SOURCE_FILE, but I'm not capturing the vars from the first line of the 'IFS=$'\t' read' line. There is also no progress on which Input ip I'm connecting too when I'm in this read loop so I can't see if it's working.

My script ended after running through about 50 servers with the following error - id: cannot find name for group ID 1000.

Here is how I added the code to my script. I don't think I've added it correctly. Any pointers on what I did wrong?

exec < $SOURCE_FILE || exit 1
read header # read (and ignore) the first line
while IFS=, read -r InputHost\
 InputIP\
 MaintenanceWindowTag\
 UserAgentTag\
 ResourceProvisioner\
 OSType\
 OSVersion\
 OSDetail\
 Location\
 PowerState\
 ; do

IFS=$'\t' read FLAVOUR HOSTNAME IPADDRESS ACTIVE_KERNEL INACTIVE_KERNEL UPTIME <<- EOFREAD
        $(sshpass -p 'xxxxxxx' ssh -qno StrictHostKeyChecking=no -o ConnectTimeout=1 user@$InputIP <<- EOFSSH | tr $'\n' $'\t'
        cat /etc/redhat-release 2>/dev/null || lsb_release -a 2>/dev/null | grep Description | cut -f2
        hostname
        hostname -I
        uname -r
        rpm -qa | grep '^kernel-[0-9]' |grep -vE `uname -r`
        uptime | cut -d "," -f1
EOFSSH
        )
EOFREAD

       if [ "$status" == "$UNREACHABLE" ]
        then
                echo "$InputHost,\
                $InputIP,\
                cannot connect,\
                $Tag,\
                $FLAVOUR,\
                $HOSTNAME,\
                $IPADDRESS,\
                $UPTIME,\
                "$ACTIVE_KERNEL,\
                $INACTIVE_KERNEL >> $REPORT_FILE
        elif [ "$status" == "$INVALID_PASSWORD" ]
        then
                echo "$InputHost,\
                $InputIP,\
                invalid account access,\
                $Tag,\
                $FLAVOUR,\
                $HOSTNAME,\
                $IPADDRESS,\
                $UPTIME,\
                "$ACTIVE_KERNEL,\
                $INACTIVE_KERNEL >> $REPORT_FILE
        elif [ "$status" == "$NO_HOME_DIRECTORY" ]
        then
                echo "$InputHost,\
                $Tag,\
                $FLAVOUR,\
                $HOSTNAME,\
                $IPADDRESS,\
                $UPTIME,\
                "$ACTIVE_KERNEL,\
                $INACTIVE_KERNEL >> $REPORT_FILE

        elif [ "$status" == "$PASSWORD_EXPIRED" ]
        then
                echo "$InputHost,\
                $InputIP,\
                password expired,\
                $Tag,\
                $FLAVOUR,\
                $HOSTNAME,\
                $IPADDRESS,\
                $UPTIME,\
                "$ACTIVE_KERNEL,\
                $INACTIVE_KERNEL >> $REPORT_FILE
        else
             echo "$InputHost,\
                $InputIP,\
                successful ssh access,\
                $Tag,\
                $FLAVOUR,\
                $HOSTNAME,\
                $IPADDRESS,\
                $UPTIME,\
                "$ACTIVE_KERNEL,\
                $INACTIVE_KERNEL >> $REPORT_FILE

        fi
done < $SOURCE_FILE

Attention:
ssh -qn is ssh -q -n where -n inhibits reading from stdin - good for ssh -q -n remotehost remotecommand . But it must be ssh -q remotehost < file and ssh -q remotehost << heredoc and ssh -q remotehost <<< herestring because here you want it to read from stdin.

Also I suspect that in a normal here-document the backtick expression `uname -r` is already locally evaluated, so the local result will be passed to the remote host.
A workaround is <<-"EOFSSH" where the quotes should suppress the immediate evaluation of backtick-expressions and all kinds of $-expressions in the following here-document.

Thanks, MadeInGermany, for pointing that ssh -n option out. Still, the commands are passed as such (no backtics `` !) to the remote server, so uname -r will be executed remotely.

The script was tested successfully as given in post #4, admittedly on one single server only. The respective variables were correctly filled with the remote info. If something does not work as expected, start over and test stepwise:

• log into the respective remote server(s) and interactively run the commands as given. They should give a one line output each. Multiline output will make the approach fail.

• run the entire ssh command on one server and with one remote command, then with multiple commands, reading results into local variables

• test remote access on all servers, with one command, and then with multiple. If one server (the 50th?) fails, repeat former steps on this one. The error may be due to a remote inconsistency.

• include thorough debuggung / error handling between command (blocks).

I quickly verified the code reading multiple remote servers from a SOURCE_FILE, and it works. echo the remote IP before running the ssh command.

Hello RudiC,

I've continued to work on this script using your suggestions and I'm very happy to report that your script suggestion has worked for me. I did as you suggested and tried your code on one VM and I was successful producing the variables in the HERE DOCUMENT with one connection to each server. This should greatly improve the speed and efficiency of my script.

Thanks very much to you and all who contributed to this thread! Very much appreciate your patience in explaining how to make this code work.