Hi all, I am brand new to these forums and I am a brand new UNIX Administartor. Don't know any scripting yet :wall:, and would like to learn as my boss is slowly letting me understand everything about being a Sys/*Nix Admin. He created this script which I am trying to replicate because I lost it to the dreaded /tmp folder. It was like 20 lines long. It seemed crazy easy to him . Oh well!
On to the problem...
I have a generated Directory Sync Report list of files that need to be synced across all servers/hosts (HP-UX ones). They are all supposed to be properly updated from the server which we push all the changes from, to all the other mirrors!
I have to use MD5 and SSH and SCP to do this. I have a script that generates hostnames/server-names (hostname.sh) which I am to use. My script should read the output of the file list (I think it was using SCP to get the files over from the other servers, and check the MD5 hash of each file, use SSH to check the same files across the other servers!
I REALIZE THIS MAY SEEM LIKE A HOMEWORK QUESTION BUT BELIEVE ME IT IS WORK RELATED.
AGAIN, I NEVER MADE A SCRIPT BEFORE!
I just need to be pointed in the right direction so that I can learn how to script myself these little things which come so easily to more experienced people!
md5 creates a distinct hash string based on the actual data in a file, ssh lets you run md5 on remote nodes, and scp copies files to remote nodes.
Assume the output of md5 looks like:
MD5 (t.lis) = ac0395498ed01862baf46065b2b92b16
The hash is the string at the end so it is the fourth field, fields defined by spaces is the default.
Assume that /path/to/master is on the server that you want as the master. And has the files to be "synced".
node1 node2 node3 are the names of remote servers/mirrors.
for fname in /path/to/master/*
do
hash=$(md5 $fname | awk '{print $4}' ) # get hash for each master file
bname=$(basename $fname) # just the file name, no path
for box in node1 node2 node3 # check each remote mirror
do
t=$(ssh myusername@$box "md5 /path/to/file/$bname" )
test_hash=$(echo "$t" | awk '{print $4}')
if [ "$hash" != "$test_hash ] ; then # files are not identical
scp $fname myusername@$box:/path/to/file/$bname
ssh /path/to/file/$bname "chmod 775 /path/to/file/$bname"
fi
done
done
Ask your boss about the chmod statement. You may not need it.
Thank you very much for your quick reply. This seems to be something I can shoot off of. Thank you for an introduction.
I think though that the script should not really base itself on a "master" server. I want to check all the servers at once, and find out the latest, most important files, whether they are on the main server or not. Do you understand what I mean.
I think I can work off of your example and see if I can work something out. But if you have any other suggestions I would love to hear them!
My boss had his script starting with declaring something and then he used the Vi file with the /pathname/files which differed for that declaration in a string:
Something to the effect of how you proposed:
for box in node1 node2 node3 # check each remote mirror
do
t=$(ssh myusername@$box "md5 /path/to/file/$bname" )
test_hash=$(echo "$t" | awk '{print $4}')
if [ "$hash" != "$test_hash ] ; then # files are not identical
scp $fname myusername@$box:/path/to/file/$bname
ssh /path/to/file/$bname "chmod 775 /path/to/file/$bname"
fi
I think he declares the "t" beforehand, and then assigns the values from one of the hostname sripts!
Ask your boss about the chmod statement. You may not need it.
I don't think I need to change permissions on the files, he didn't use that in his script!
Thank you so much for all your help!
Ivan
P.S. Hey while I have you, how can I read his scripts that he made. They are executable and just say hostname, and host. If I open them with Vi they just read "executable" on the bottom! Weird! I'm trying to understand his approach to scripting, he's amazing!
You should check out man rsync ("rsync"), it is very versatile, works over ssh connections and can sync up files based on checksum, last-modified time and more.
Right I did see everyone talking about that program. I will have to look into that. I just wanted to also learn scripting to aid me in my job. But yeah that program seems like a very good solution!
Thanks for that, and yet still I have questions, I apologize.
I think originally he had /usr/bin declared...I'm getting errors about "md5" function!
Here is what I have so far!
hosts=`cat /home/izivanov/iz_hosts` #where iz_hosts is a txt file list of all hosts/servers
for fname in /home/izivanov/iz2 #where iz2 is a file with all the files with pathnames
do
hash=$(md5 $fname | awk '{print $4}' )
bname=$(basename $fname)
for box in ${hosts}
do
t=$(ssh ${box} "md5 fname /opt/dba/scp/$bname")
test_hash=$(echo "$t" | awk '{print $4}')
if [ "hash" != "$test_hash"] ; then
echo Files Are Not Identical
fi
done
done
And this is what I get in return:
:/home/izivanov# sh dirsync.sh
sh: md5: not found.
dirsync.sh[10]: test: Specify a parameter with this command.
sh: md5: not found.
dirsync.sh[10]: test: Specify a parameter with this command.
sh: md5: not found.
dirsync.sh[10]: test: Specify a parameter with this command.
sh: md5: not found.
dirsync.sh[10]: test: Specify a parameter with this command.
sh: md5: not found.
dirsync.sh[10]: test: Specify a parameter with this command.
sh: md5: not found.
dirsync.sh[10]: test: Specify a parameter with this command.
sh: md5: not found.
dirsync.sh[10]: test: Specify a parameter with this command.
I think its not recognising my md5 command. BTW I am working in SecureCRT connecting to HP-UX servers. If I didn't already mention that!
Your second try, 'for fname in /home/izivanov/iz2' is even weirder, since for doesn't read files.
Both should be done as while read X ; do stuff ; done < filename
while read fname
do
hash=$(/usr/bin/md5 "${fname}" | awk '{print $4}' )
bname=$(basename $fname)
while read box
do
# what is 'md5 fname' supposed to do? is there really a file named 'fname' over there?
# t=$(ssh ${box} "/usr/bin/md5 fname /opt/dba/scp/$bname"| awk '{print $4}')
t=$(ssh ${box} "/usr/bin/md5 /opt/dba/scp/$bname"| awk '{print $4 }')
[ "${hash}" = "${t}" ] || echo Files Are Not Identical
done < /home/izivanov/iz_hosts
done < /home/izivanov/iz2
Depending on your shell there may be more efficient ways to get the fourth parameter than awk '{print $4}' but we don't know what your shell is yet.
Thank you. I'm , hence the very bad programming mistakes. I never programmed in my life. But now have to learn it!
fname is supposed to be a file name which is taken out of the file iz2 (the one in /home/izivanov/iz2)
The file just lists the 5 files to be changed with their path names like so:
/opt/dba/scp/so-and-so
/opt/dba/scp/so-and-so2
/opt/dba/scp/so-and-so3
etc.
---------- Post updated at 05:13 PM ---------- Previous update was at 04:47 PM ----------
After putting this in my script i still get the same errors about MD5. Don't know what ails it!
My shell is:
:/home/izivanov# echo $SHELL
/sbin/sh
Don't know if that helps!
Here is the script:
(root):/home/izivanov# vi dirsync.sh
"dirsync.sh" 12 lines, 312 characters
while read fname
do
hash=$(/usr/bin/md5 ${fname} | awk '{print $4}' )
bname=$(basename $fname)
while read box
do
t=$(ssh ${box} "/usr/bin/md5 /opt/dba/scp/$bname" | awk '{print $4}')
[ "${hash}" = "${t}" ] || echo Files are not Identical!!!
done < /home/izivanov/iz_hosts
done < /home/izivanov/iz2
Here is the output!
(root):/home/izivanov# sh dirsync.sh
dirsync.sh[3]: /usr/bin/md5: not found.
sh: /usr/bin/md5: not found.
dirsync.sh[3]: /usr/bin/md5: not found.
sh: /usr/bin/md5: not found.
dirsync.sh[3]: /usr/bin/md5: not found.
sh: /usr/bin/md5: not found.
dirsync.sh[3]: /usr/bin/md5: not found.
sh: /usr/bin/md5: not found.
rtidsva(root):/home/izivanov#
/usr/local/bin/ is a weird place for it to be. That's not usually in your PATH, either. It might have been custom-built and installed on just that server... You'd better make sure all the servers in question actually have it.
If they do, there you go, /usr/local/bin/md5.
If they don't, hmm.... how big are these files you're checking?
The ssh command ends when I've colored it red. Everything else is local. (to put a pipe in the foreign host you'd have to backslash it, like \|, to prevent your shell from handling it itself.)
I got some pointers but am still not getting anywhere. The stuff thats commented out are suggestions to use!
#!/usr/bin/sh
set -A HOSTS `/usr/local/bin/hosts` `hostname`
while read fname
do
hash=$(/usr/local/bin/md5 ${fname} | awk '{print $4}' )
bname=$(basename $fname)
#first=0
#for box in `/usr/local/bin/hosts` `hostname`; do
#for box in ${HOSTS[@]}; do
while read box
do
#if [ $first eq 0 ]; then
# <do stuff here>
# $first=1
#else
# <more stuff>
#fi
t=$(ssh ${box} "/usr/local/bin/md5 /opt/dba/scp/$bname" | awk '{print $4}')
[ "${hash}" != "${t}" ] | echo Files are not Identical!!!
done < /home/izivanov/iz_hosts
done < /home/izivanov/iz2
So I used that:
#!/usr/bin/sh
set -A HOSTS `/usr/local/bin/hosts` `hostname`
while read fname; do
hash=$(/usr/local/bin/md5 ${fname} | awk '{print $4}' )
bname=$(basename $fname)
first=0
for box in `/usr/local/bin/hosts` `hostname`; do
for box in ${HOSTS[@]}; do
while read box; do
if [ $first eq 0 ]; then
t=$(ssh ${box} "/usr/local/bin/md5 /opt/dba/scp/$bname" | awk '{print $4}')
$first=1
else
[ "${hash}" != "${t}" ] | echo Files are not Identical!!!
fi
done
done
Now I'm getting a syntax error on line 11, saying that 'for' is not matched!
It means what it says, the 'for' is unmatched. What usually matches for? done; you never closed that second 'for' loop.
If you have more than a small amount of data, backticks are a bad idea, since shell variables have limits on many platforms -- try and cram more than a few K into them and the end will get chopped off. See useless use of backticks.
#!/usr/bin/sh
while read fname; do
bname=$(basename $fname)
first=0
{
hostname
/usr/local/bin/hosts
} | while read box; do
t=$(ssh ${box} "/usr/local/bin/md5 /opt/dba/scp/$bname" | awk '{print $4}')
if [ $first -eq 0 ]; then
hash=$t
first=1
else
[ "${hash}" == "${t}" ] || echo Files are not Identical!!!
fi
done
done < /home/izivanov/iz
Things to note:
Get rid of the 3 different methods to loop thru host list and just use 1. I picked the while read box (with input from hostname and /usr/local/bin/hosts)
With test the numeric equals operator is -eq not eq
On first loop just assign hash with value from t, on 2nd and subsequent loops check t match hash
Don't put $ in front of a variable when assigning it $first=1 will not work
When testing hash use == with || or != with && ( single | is a pipe and would not have worked at all)
You forgot to direct your filelist into the while read fname loop
. Oh well!
It's a very bad programming habit...
, hence the very bad programming mistakes. I never programmed in my life. But now have to learn it!