ksh scripting SSH to Compare File Sizes

Shell Programming and Scripting
1

Hello,
I currently have very little experience with Shell scripting and trying to create a script for the purpose of collecting the size of a couple sizes on 4 different Hosts. The Idea is to collected the information from the files in which the script is kicked off on, store the values into Variables, SSH to the next Host and repeat for the remaining 2. With Variables Collected I would like to compare the sizes and output that they matched.

This is what I have come up with so far but not the most familiar with constructs and especially the SSH Process.

#!/bin/ksh

FILENAME1=/root/directory1/files/file1.fil
FILE1SIZEa=$(stat -c%s "$FILENAME1")

FILENAME2=/root/directory2/files/file2.fil
FILE2SIZEa=$(stat -c%s "$FILENAME2")

ssh user@server2 |

FILENAME3=/root/directory1/files/file1.fil
FILE1SIZEb=$(stat -c%s "$FILENAME3")

FILENAME4=/root/directory2/files/file2.fil
FILE2SIZEb=$(stat -c%s "$FILENAME4")

ssh user@server3 |

FILENAME5=/root/directory1/files/file1.fil
FILE1SIZEc=$(stat -c%s "$FILENAME5")

FILENAME6=/root/directory2/files/file2.fil
FILE2SIZEc=$(stat -c%s "$FILENAME6")

ssh user@server4 |

FILENAME7=/root/directory1/files/file1.fil
FILE1SIZEd=$(stat -c%s "$FILENAME7")

FILENAME8=/root/directory2/files/file2.fil
FILE2SIZEd=$(stat -c%s "$FILENAME8")


if && [[ "$FILE1SIZEa" =~ ^($FILE1SIZEb|$FILE1SIZEc|$FILE1SIZEd)$ ]]; then
	print "FILE1 Sizes Match"
else
	print "Something is Wrong with FILE1"
fi

if && [[ "$FILE2SIZEa" =~ ^($FILE2SIZEb|$FILE2SIZEc|$FILE2SIZEd)$ ]]; then
	print "FILE2 Sizes Match"
else
	print "Something is Wrong with FILE2"
fi

If someone could help get this functioning, or possibly has suggestions for more efficient scripting with less repetition It would be greatly appreciated.

Thanks for your consideration.

2

You should build a function that do what you want.

Then loop over the servers with a "while" or a "for" instruction and call that function.

You should also choose whether you want to hold this server list into a config file or a variable.

3

I appreciate the advice, I was kind of thinking the same thing myself while looking at it that I needed some sort of for loop to iterate through the servers executing the same command while storing file locations in variables and simply comparing each server to the 1st. I don't really have any idea of how i could apply a for loop to ssh multiple servers that were in an Array for instance. Though I did rework the code a bit and came up with the following solution, though you couldn't be as specific in your output you could easily spot an error.

#!/bin/ksh


FILE1=/root/directory1/files/file1.fil
FILE2=/root/directory2/files/file2.fil
FILESZ1==$(stat -c%s "$FILE1")
FILESZ2==$(stat -c%s "$FILE2")

filecheck() {
CHKFILE1=$(stat -c%s "$FILE1")
CHKFILE2=$(stat -c%s "$FILE2")
if [[ $FILESZ1 = $CHKFILE1 ]]; then
	print "file sizes match"
else
	print "file sizes do not match"
fi
if [[ $FILESZ2 = $CHKFILE3 ]]; then
	print "file sizes match"
else
	print "file sizes do not match"
fi
}

ssh user@server2 "$(typeset -f filecheck); filecheck;
ssh user@server3 "$(typeset -f filecheck); filecheck;
ssh user@server4 "$(typeset -f filecheck); filecheck;

Would the following code do the trick or am I missing something?
Thanks for the suggestions.

4

You will have to define the function on each node separately, and the local variables like FILESZ1 won't be available there.

How about creating a server file like

$ cat svctl
user@server2
user@server3
user@server4

to which you could even add your local server entry (user@server1) as the first line provided it has an sshd service running, and then, using shell arrays for the file sizes

$ FILE1=/root/directory1/files/file1.fil
$ FILE2=/root/directory2/files/file2.fil 
$ while read SV; do { read FILESZ1[++CNT]; read FILESZ2[CNT]; } <<< $(ssh -n $SV "stat -c%s $FILE1; stat -c%s $FILE2"); done <  svctl

leaving you with two file size arrays of four elements each to work upon locally. Admittedly, this has been tested on bash only, as I don't have a ksh machine here, but corrections - if necessary at all - should be minor.

1 Like
5

Thanks for the feedback and option, Though being very new this that code is a bit over my head so I'm currently Dissecting it right now to try and get a better understanding as to what exactly is going on piece by piece.

So if i have this correct for the 1st part you could create a file like SV.fil in the same directory with code such as:

$ cat svctl
user@server1
user@server2
user@server3
user@server4

I get that the cat command is simply stating "Read the following Contents" what is the " svctl " all about?

Next it appears you are starting a while loop that's reading that SV.fil that would have to be in the same directory in the loop it appears you have created 2 Arrays "read FILESZ1[++CNT] read FILESZ2[CNT] ", which i'm guessing the "++" increments it, though why is that not in " FILESZ2[CNT] " ? Did I not understand that portion correctly?

The rest of it look self explanatory iterating through the servers collecting the file size of each file adding them to each array (yet then there's that svctl again?)

Once this line has been executed I then have 2 Arrays in which I can simply use functions to compare all values and give outputs :slight_smile:

Sorry I'm new and this scripting, and appreciate you taking the time :slight_smile:

6

We need a text file containing the login information of the servers to visit. Choose a file name to taste, call it SV.fil , or, as I did, svctl (~ "server control"). No "extension" (remainder from MSDOS times) needed. As I can see, you already added your local host for ease of operation.

Yes, cat copies its stdin (or the files given as arguments) to stdout - that was just for showing readers the contents of that file.

In the while loop, we're reading the server info line by line into the SV variable, then execute a "compound command" consisting of two read s to populate two arrays' elements indexed by the CNT variable which in turn is pre-incremented once per loop, so both arrays are indexed by the same value. stdin of this compound command is redirected from a "here string", consisting of a "command substitution" $(...) that provides the stdout of the entire ssh command including the two stat commands executed on the remore node for the two target files.

With four servers in the control file, we should end up with eight file sizes, four for file1 and four for file2, in arrays FILESZ1 and FILESZ2 , indexed 1 ... 4 (given CNT was unset or reset to 0 upfront). You now can work on those array elements, as you did before, like

if [ ${FILESZ1[1]} -eq ${FILESZ1[2]} ]; then

. You could do it e.g. in a for loop 2 ... 4 to improve efficiency and readability.

1 Like
7

Another approach using ksh. I setup two arrays of the FILES to check and the HOSTS.

Then as and example, I print each file name and it's size on host numbers 0,1 and 2.

#!/bin/ksh

set -A FILES /root/directory1/files/file1.fil /root/directory2/files/file2.fil
set -A HOSTS user@server2 user@server3

i=0
while [[ $i -lt ${#HOSTS[@]} ]]
do
   set -A FS$i $(ssh ${HOSTS} "stat -c%s ${FILES[@]}")
   ((i++))
done

i=0
while [[ $i -lt ${#FILES[@]} ]]
do
  echo ${FILES} ${HOSTS[0]}=${FS0} ${HOSTS[1]}=${FS1} ${HOSTS[2]}=${FS2}
  ((i++))
done
2 Likes
8

I like that consequent application of arrays; that might simplify my approach and even offer simple extensions...

9

Thanks for the other approach, I really appreciate you both taking the time to give an example, I am going to play around with both of these and see what I can come up with :slight_smile:

10

So messing around with both Methods I have come to find out that "Stat" does not work with the Version of AIX I'm on but Instead I must use "wc"
So I have improvised on the line

set -A FS$i $(ssh ${HOSTS} "wc -c ${FILES[@]} | awk '{print $1}'")

I have tested the line starting with ssh and it works just fine but when running the Script i get the following error

"i++: 0403-053 Expression is not complete; more tokens expected

" I'm almost positive that line is the issue as it goes to ssh , pulls the server disclaimer but continues no further. I have tried putting the command in Double Quotations as Well as the

'{print $1}'

in double apostrophes, not sure what I'm still missing here?

11

Does your ksh version understand the (( ... )) "Arithmetic Expansion"?
Did you check for the istat command?
Did you assess using ls -l 's size field (one inode access) against counting every single byte in a file?

12 
set -A FS$i $(ssh ${HOSTS} "wc -c ${FILES[@]} | awk '{print \$1}'")
13 
i=0 while [[ $i -lt ${#HOSTS[@]} ]]; do    set -A FS$i $(ssh ${HOSTS} "stat -c%s ${FILES[@]}")    ((i++)) done

Is this not the problem for ksh? I will have to give that a shot.

set -A FS$i $(ssh ${HOSTS} "wc -c ${FILES[@]} | awk '{print \$1}'")

Whats The Significance of the Back Slash here?

as for the Version of Ksh, I'm pretty sure it predates the paleolithic era, as for the the istat (inode) command I have not tried that, looked at it previously but couldn't find any sort of modifiers or description on how to extract just the file size from IBM's site so wasn't positive that was an option. and for assessing the "ls -l" are you referring to something like this?

$ set -A FS$i $(ssh ${HOSTS} "ls -lah ${FILES[@]} | awk '{print $5}'")
14

Same problem as before. Without escaping, $5 will evaluate before ssh happens.

set -A FS$i $(ssh ${HOSTS} "ls -lah ${FILES[@]} | awk '{print \$5}'")
15

To understand which version you are on you might eventually consider some really farfetched idea, like telling us this information. Issue

oslevel -s

and post the output.

I think the problem is with the missing spaces. As far as i know ksh needs spaces around the double brackets. Either this or you use ksh88 and this does not understand every infix notation. Try one of those instead (notice the additional spaces!):

(( i++ ))
(( i += 1 ))

I don't think so. Which version exactly you have would depend on the version of AIX (see above) but in general AIX uses a ksh88 as /usr/bin/ksh and as default shell and has a ksh93 as /usr/bin/ksh93 .

istat is the correct method of getting inode information. According to this link which not only provides a thorough explanation of what it does but also a sample output it will look like:

istat /usr/bin/ksh
Inode 10360 on device 10/6    File
Protection: r-xr-xr-x
Owner: 2(bin)     Group: 2(bin)
Link count: 2     Length 372298 bytes
   
Last updated:  Wed May 13 14:08:13 1992
Last modified: Wed May 13 13:57:00 1992
Last accessed: Sun Jan 31 15:49:23 1993

therefore, to display the filesize, issue:

istat /path/to/file | sed -n '/Length/ {;s/.*Length //;s/ bytes.*//p;}'

I hope this helps.

bakunin

1 Like
16

Thanks for taking the Time in explaining that for me. I tried a simple command that told me the version was 3, but I don't believe that is specific enough so I will give that a shot when I'm back at the terminal next week, I will also look into iNode and the Arithmetic options you gave me and see what I can gather.

Thank you for the Explanation :slight_smile:

1 Like
17

I don't know (and, in fact, can't even start to imagine) which simple command that might have been: the Korn shell comes in two "versions" (actually rather "language standards", like "FORTRAN77", than versions): "ksh88" basically means compatible to the version which was released 1988, eventually minus bugs in the original corrected since then and "ksh93" (the same but for the enhanced version released 1993).

You get the real version information for both of them in the following way: display the content of the system-variable ${.sh.version} . For my system i get:

# print - ${.sh.version}
Version AJM 93u+ 2012-08-01

This means it is a ksh93 and the version is "u+" from 2012, as ksh93 versions are "numbered" by characters: a, b, c, ... If i remember correctly the latest version of ksh93 is "w", so mine is pretty up to date.

I hope this helps.

bakunin

18 
 uname -v