Read file and skip the line starting with #

Hi all,
I'm new in unix. Need some help here.

I have a file called server.cfg which contains the servers name, if I don't want to run on that server, I'll put a "#" infront it.

username1@hostname.com
username2@hostname.com
#username3@hostname.com
#username4@hostname.com
username1@hostname.com

and I need to retrieved them to run a script to copy file to the servers.
here is my script.

#!/usr/bin/sh
file2put=$1
if [ -f $file2copy ]
then
 echo "File found, preparing to transfer... \n"
 
  grep -v '^#' | while read line 
  do
   ssh $line < checkdirectory.sh #another script to check the existing of directory
  scp $file2put $line:~/folder_name/$file2copy
 done < server.cfg
 exit
 else
  echo "File does not exist. Exiting..."
  exit 
fi
 

The problem is that it won't skip the server that start with a "#". I had search the similar thread such as [how to skip the line starting with "#"] etc, and I try to use the 'grep' function but it won't solve my problem.

Fyi,
my operating system : SunSolaris

Anyone can help on this? Sorry for my poor of English.

Thanks in advanced.

I think this will skip the servers with the # in front:

#!/usr/bin/sh
file2put=$1
if [ -f $file2copy ]; then
  echo "File found, preparing to transfer... \n"
  grep -v '^#' server.cfg > valid_servers
  while read line; do
    ssh $line < checkdirectory.sh #another script to check the existing of directory
    scp $file2put $line:~/folder_name/$file2copy
  done < valid_servers
  exit
else
  echo "File does not exist. Exiting..."
  exit 
fi
1 Like

Hi hanson44,

Thanks for your reply.
It really skip the line with #..
Really thanks! :")

---------- Post updated at 03:06 PM ---------- Previous update was at 02:55 PM ----------

Hi,

After I execute, it did skip the # infront
but it run one more time after reading the last server in the file.
how can I stop it? :confused:

Try simplifying some. Comment out the ssh and scp lines that do the work, and add the diagnostic echo line, to see what is going on:

#!/usr/bin/sh
file2put=$1
if [ -f $file2copy ]; then
  echo "File found, preparing to transfer... \n"
  grep -v '^#' server.cfg > valid_servers
  while read line; do
    echo line = $line
    # ssh $line < checkdirectory.sh #another script to check the existing of directory
    # scp $file2put $server:~/folder_name/$file2copy
  done < valid_servers
  exit
else
  echo "File does not exist. Exiting..."
  exit 
fi

If that does not help, put "set -x" on the second line of the script, and trace the diagnostic messages.

this the output,
it just simply run one more time without bring anything from the file.
and assume the username2@hostname.com is the last line of the file.

 
file2put=mesg.txt
dirname=
+ [ -f mesg.txt ]
+ echo File found, preparing to transfer... \n
File found, preparing to transfer...
+ grep -v ^# share.cfg
+ read line
+ echo line = username1@hostname.com
line = username1@hostname.com
+ read line
+ echo line = username2@hostname.com
line = username2@hostname.com
+ read line
+ echo line =
line =
+ read line
+ exit

---------- Post updated at 03:55 PM ---------- Previous update was at 03:30 PM ----------

Hi,
problem solved.
My server.cfg end with a blank line.
that's why it copy the blank line to the valid_servers too..

I'd remove the blank line and it works now.
Thanks again :slight_smile:

Great. You figured it out!

The other thing needed sometimes, you probably already know, is to put "sleep 1" at key points to slow the script down if it's scrolling by too fast to keep track of what's going on.

1 Like

Hi hanson44,

Thanks for the remind :slight_smile:

Hey!!!

Do you really want to create a physical file "valid_servers"? Instead, we can just redirect the output of grep to while loop.

egrep -v '^#|^$' file1|while read server_name
do
    echo "$server_name"
    # File transfer goes here
done

Cheers,
Ranga:)

Hi Ranga,
Thanks for your reply,
I had tried this before I post my problem here, too bad it doesn't skip any # in front of the line. :frowning:

It's good of you to point that out. I already knew that. I was just trying to present a simple solution.

Normally, I prefer to put the input to the while loop at the end, because it seems cleaner and easier to read. And making a separate file makes the intent more clear. It's a way of self-documentation.

Your suggestion related to the blank line is an improvement.

Hey!!!

This will work and look at the highlighted part. before that you have redirected the file to while loop instead of grep.

egrep -v '^#|^$' server.cfg|while read server_name
do
echo "$server_name"
# File transfer goes here
done

Cheers,
Ranga:)

Yes. I did try the same thing before.
Still the same. :o

Please make a try with this now and then post us.

Cheers,
Ranga:)

---------- Post updated at 04:15 AM ---------- Previous update was at 04:13 AM ----------

Yeah you are right. If the file size is biggy then, you should be rethink of your self documentation!!!:slight_smile:

Cheers,
Ranga:)

No, you did not. Your original attempt is not the same.

What you did:

grep -v '^#' | while read line; do ... done < server.cfg

What rangarasan suggested:

grep -v '^#' server.cfg | while read line; do ... done

There is a big difference. Pipe redirections occur before other redirection operations.

rangarasan is telling grep to read the file. Your original solution has grep reading something else (whatever standard input descriptor it inherited, which could be a teriminal or some other file).

You redirected the file directly to the while-read loop, bypassing grep. Your grep is doing nothing. Even if there is something for it to read on standard input (we can't tell from your script what stdin is, since it depends on how the script is called), there is nowhere for the grep data to go. Why? Because after the pipe connects the standard output of grep to the standard input of the while-read loop, the standard input of the while-read loop is overridden by the redirection operation immediately following the while-loop. So, even if grep does write to the pipe, there is nothing on the other end to read the data.

On a different note, there is no need for grep. Within the while-loop, a case statement can do the job:

case $line in
    \#*) continue ;;
esac

Or, if the shell supports substring parameter expansion:

[ "${line:0:1}" = '#' ] && continue

Regards,
Alister

Hi,

hmm.. I think u guys misunderstanding..
I'm trying not only that way on my post..
I post before I search for solution..
and my post is the latest tried. so I decide to post that as my problem..
thanks for your solution..Alister.
I also did tried that, but still didn't work for me..
It prompt "bad substitution" in my error message. :frowning:

Hi all,

I have another problem..
I do some changes on my code

My code is something like this..

#!/usr/bin/sh
file2copy=$1
foldername=$2 
grep -v '^#' servers.cfg > valid_servers
while read line
 do 
  echo $line
  if ssh $line "if [ ! -d $foldername ]; then mkdir $foldername; fi"
  then echo " folder exists"
  else echo " error"
  fi
 
  if scp -r $file2copy $line:~/$foldername 
  then echo "File/Folder $file2copy fully copied ..."
  else echo "Error: $file2copy not copied ..."
  fi
done < valid_servers
exit 

For example, valid_servers has 8 servers inside..
when I run the script, it only pass 4 servers.
But
if I "#" the ssh part, and let it run only the scp part (assume folder exist in all servers),
it doesn't have any problem, and successful to copy files into all the 8 servers.

Anyone know what happen in this problem? :confused:

FYI, I had keep trying the egrep command too..
maybe the space_bar or what make the script won't run,
and now it works with egrep too..
thanks Alister and Ranga. :b:

Regards,
beezy :o

ssh's standard input is the same descriptor that the while loop is reading. ssh is probably consuming some of the data. Try -n or explicitly overriding its stdin with /dev/null.

Regards,
Alister

Hi Alister,
thanks for your reply. You mean ssh -n $line..... ?

Thanks .. It works now. :o