script to get file size

Shell Programming and Scripting
1

Hi All,
I have some requirement. i dont know if we can write some shell script to fulfill the requirement. Here is my requirement...
I have some files under /var/opt/abc/xyz like below.
-rw-r--r-- 1 root root 789222 Aug 14 11:03 big.sh
-rw-r--r-- 1 root root 789222 Aug 14 11:03 big.txt
-rw-r--r-- 1 root root 789222 Aug 14 11:03 big.txt
-rw-r--r-- 1 root root 789222 Aug 14 11:03 big.sh
-rw-r--r-- 1 root root 789222 Aug 14 11:03 big.txt
-rw-r--r-- 1 root root 789222 Aug 14 11:03 big.txt (This is basically to get the file size)

In those files, there are some files are .txt files. I need to get find out those .txt files and output shoud like below...
-rw-r--r-- 1 root root 789222 Aug 14 11:03 big.txt
-rw-r--r-- 1 root root 789342 Aug 14 11:03 big.txt
-rw-r--r-- 1 root root 7823252 Aug 14 11:03 big.txt

And also i need to do this on several servers. I dont know if we can write a script and put it on one server and run it from there to get the final output. output should be like this....
servername: abc
-rw-r--r-- 1 root root 789222 Aug 14 11:03 big.txt
-rw-r--r-- 1 root root 789342 Aug 14 11:03 big.txt
-rw-r--r-- 1 root root 7823252 Aug 14 11:03 big.txt
servername: xyz
-rw-r--r-- 1 root root 789222 Aug 14 11:03 big.txt
-rw-r--r-- 1 root root 789342 Aug 14 11:03 big.txt
-rw-r--r-- 1 root root 7823252 Aug 14 11:03 big.txt
servername: mnc
-rw-r--r-- 1 root root 789222 Aug 14 11:03 big.txt
-rw-r--r-- 1 root root 789342 Aug 14 11:03 big.txt
-rw-r--r-- 1 root root 7823252 Aug 14 11:03 big.txt
anybody help me out to write this script. Thanks in advance.

2

You want to ls -l *.txt in several servers? Is that it? Do the servers have ssh?

3

Also it would be nice to know the OS. If it is AIX there would be dsh (Distributed Shell) to simultaneously execute a command on several servers, but other OSes might have different means of achieving that or none at all.

So: please help us to help you and tell us some details.

I hope this helps.

bakunin

4

Thanks for your reply guys...

I got some SLES and some RHEL servers. I dont know whether these servers has ssh. How can we check that.
Also to do ls -l *.txt, we have to go to particular file path then only we can run this command to get the desired output. Instead of going to particular path, can we run this command from Home directory or somewhere else with out doing sudo su - (non root user). Thanks.

5

Issue "ssh". If you see "unknown command" or something such it is not installed, if you get a (ssh-)prompt it is.

ls -l /some/path/*txt

This will list all files "*txt" in the directory "/some/path".

I hope this helps.

bakunin

6

If it's OpenSSH I don't think you get an interactive prompt, you just get a brief usage message. The big question is how you connect to those other servers now, if you have to walk to the console of each one to type commands then that's not easily scriptable, but telnet or rsh or ssh can be scripted (ssh more easily, and of course, it's the only secure option).

7

In reality you are looking for sshd on remote server and ssh on local.
On remote from CLI check if sshd is running using the command:

pgrep sshd
or..
ps -aux | grep 'shd'
or..
ssh localhost    #if sshd run on default port 22
8

Thanks for your reply...

ls -l /some/path/*txt

This worked. I put it in file and saved it in .sh format and ran it and it gave me the output how I want to. thats great. Then I checked ssh. it seems to be installed on my machine. I tried with ssh, it gave me something. so ssh installed. Then how can I use ssh to run the same .sh file to get the same information from other servers.(I mean as I given output in 1st thread). Thanks

9 
for host in server1 server2 golem elemental shibboleth; do
  ssh $host 'ls -l /some/path/*.txt'
done

... assuming your servers have names like server1, server2, golem, elemental, and shibboleth. (Sorry, not very imaginative.)

10

hmm,

I just tried with this code

for host in server1 server2; do
ls -l /some/path/.txt (since it is the same server where I'm running the script)
ssh $ 111.111.111.111 'ls -l /some/path/.txt' (I also tried with server name)
done

when i ran this, I got the first server information,
-rw------- 1 root root 32768 Aug 23 14:58 /some/pathcert.txt
-rw------- 1 root root 3238 Aug 23 14:58 /some/path/cert.txt
ssh: $: Temporary failure in name resolution
-rw------- 1 root root 32768 Aug 23 14:58 /some/pathcert.txt
-rw------- 1 root root 3238 Aug 23 14:58 /some/path/cert.txt
ssh: $: Temporary failure in name resolution

I also tried this....

for host in 111.111.111.111; do
#ls -l /some/path/.txt (since it is the same server where I'm running the script)
ssh $ 111.111.111.111 'ls -l /some/path/.txt' (I also tried with server name)
done

I got the same error...

Thanks for your reply

11

This is how to do it on one remote host:

ssh 111.111.111.111 'ls -l /some/path/*.txt'

The argument $host (including the dollar sign) is a single token which expands to server1 on the first iteration of the for loop, to server2 on the next iteration, etc. A lone dollar sign is not meaningful or useful in this context.

So if your remote hosts are called 111.111.111.111 and 222.222.222.222 that would give you

for host in 111.111.111.111 222.222.222.222; do
  ssh $host 'ls -l /some/path/*.txt'
done

The results you report are from the command running locally.

12

great that works. I ran the below code

for host in 111.111.111.111 222.222.222.222; do
ssh $host 'ls -l /some/path/*.txt'
done

it prompted me to do some steps...I did all those and finally i got the output as i want to. But when I ran it for second and even third time, its prompting me for password for both servers like below...

abc:/home # ./shell2.sh
Password:
-rw------- 1 root root 32768 Aug 23 15:22 /some/path/cert.db
-rw------- 1 root root 8192 Aug 23 15:22 /some/path/crl.db
-rw-rw-rw- 1 root root 40960 Aug 16 10:12 /some/path/dx32847.db
-rw-rw-rw- 1 root root 40960 Aug 23 20:39 /some/path/dx32858.db
-rw------- 1 root root 294912 Aug 23 15:25 /some/path/nds.db
Password:
-rw------- 1 root root 8192 Aug 23 22:06 /some/path/crl.db
-rw------- 1 root root 724992 Aug 23 14:19 /some/path/nds.db
abc:/home #

can we do something to not prompt for password again and again when we run the script.

Thanks once agian....

13

That's (also) a frequent topic in these forums; google for "passwordless ssh" or "public key".

14

My bad, i have misunderstood the question. How about the following script function:

# ------------------------------------------------------------------------------
# f_CheckConnectivity                          checking connectivity for a host
# ------------------------------------------------------------------------------
# Author.....: Wolf Machowitsch
# last update: 2007 01 18    by: Wolf Machowitsch
# ------------------------------------------------------------------------------
# Revision Log:
# - 0.99   2007 01 18   Original Creation
#                       -
#
# ------------------------------------------------------------------------------
# Usage:
#     f_CheckConnectivity  char Hostname [ char user ] 
#     checks the connectivity for the host given in $1 optionally using a
#     username given in $2. If no user name is given the current user is
#     assumed.
#
#     Example:  f_CheckConnectivity $host    # checks if $host can be worked on
#                                            # using the current user (usually
#                                            # this will be root)
#
# Prerequisites:
# -   to use this function, the FPATH variable must be set
#
# ------------------------------------------------------------------------------
# Documentation:
#     f_CheckConnectivity() checks in a successive manner. First an (IP)-ping
#     is issued. If this is successful a connection test is done by issuing
#     a command via ssh. f_CheckConnectivity() does NOT try to correct any
#     errors, merely stating them. 
#
#     Parameters: char Host
#
#     returns:    0: connectivity test passed
#                 1: no ssh connection
#                 2: no IP connection
#                 3: parameter/other/internal error
#
# ------------------------------------------------------------------------------
# known bugs:
#
#     none
# ------------------------------------------------------------------------------
# .....................(C) 2007 Wolf Machowitsch ...............................
# ------------------------------------------------------------------------------

f_CheckConnectivity ()
{

$chFullDebug
                                                 # internal variables
typeset -i iRetVal=0                             # return value (see docu)
typeset    chHost="$1"                           # hostname
typeset    chUser="$2"                           # optional username

if [ -z "$1" ] ; then                            # check for prereqs
     iRetVal=3
elif [ -z "$chUser" ] ; then
     chUser="$(who am i | cut -d' ' -f1)"
fi

if [ $iRetVal -eq 0 ] ; then
     if [ $(ping -c1 $chHost 1>/dev/null 2>&1 ; print - $?) -gt 0 ] ; then
	  iRetVal=2
     fi
fi
if [ $iRetVal -eq 0 ] ; then
     if [ $(ssh -o 'BatchMode = yes' ${chUser}@${chHost} date 1>/dev/null 2>&1;\
	    print - $?
	   ) -gt 0 ] ; then
	  iRetVal=1
     else
	  iRetVal=0
     fi
fi

return $iRetVal

}
# --- EOF f_CheckConnectivity

I hope this helps.

bakunin

15

thats fine I'll google it. I have some question here, with same code we were using, can we add something to do this..

1, I need to get the .txt files which are greater than 10mb in size from all servers.(i dont need every .txt file)
2, can we point output to file?
3, if we can point to the file, can we send this file to my mail?

basically this is my final requirement.

Thanks once again.

16 
for host in foo bar baz quux; do
  ssh $host 'ls -l /some/path/*.txt' | awk '$5 > 10000000 { print "'"$host"': $0 }'
done  >file.out

test -s file.out && mailx -s "Files bigger than 10M" you@example.com <file.out

You could try to avoid using a temporary file, but just add "rm file.out" at the end if you don't want to keep it.

17

Thanks for your reply,

I'm getting the output, but I can't tell which file belongs to which server. Can we get output like this..

servername: abc
-rw-r--r-- 1 root root 789222 Aug 14 11:03 big.txt
-rw-r--r-- 1 root root 789342 Aug 14 11:03 big.txt
-rw-r--r-- 1 root root 7823252 Aug 14 11:03 big.txt
servername: xyz
-rw-r--r-- 1 root root 789222 Aug 14 11:03 big.txt
-rw-r--r-- 1 root root 789342 Aug 14 11:03 big.txt
-rw-r--r-- 1 root root 7823252 Aug 14 11:03 big.txt
servername: mnc
-rw-r--r-- 1 root root 789222 Aug 14 11:03 big.txt
-rw-r--r-- 1 root root 789342 Aug 14 11:03 big.txt
-rw-r--r-- 1 root root 7823252 Aug 14 11:03 big.txt

Thanks

18

The awk snippet adds a host: prefix to each line precisely for this reason. I had it the way you describe it at first, but having the information on each line makes the processing somewhat simpler to code. You can post-process it to "lift up" the host before the lines it describes, of course. Are you not seeing the host name at the beginning of every line of ls output?

(Maybe you have an earlier version of the script; I edited it a couple of times after initially posting it. Sorry for the mess.)

19

No I dont see the server name at the begining of ls output. its giving all together and I dont see any server name.

Thanks

20

And the script above is the one you are running, with the $host with the complicated quoting in the awk part? Actually the quoting isn't even right, there's a double-quote missing after the colon. I've edited it into the quoted script above.

You could add echo servername: $host before the ssh but then the logic for finding out whether or not an email should be sent needs to be changed, too; and there will be a servername: header for all servers, regardless of whether they had any files larger than 10M or not. Maybe that would be acceptable, though.