i am stuck with this strange problem..... maybe you can help.
i have one master_file which has two column username and id_number separated by , somewhat like :
cat master_file :
sample,1234567
javacode,4567891
companion,23456719
adamsandler,1237681
tomcruise,56328910
bradpitt,901236781
sample,782109432
tomcruise,89210321
adamsandler,90812314
bradpitt,78325610
..............
.....and so on
now there is a directory named sample that consist of multiple directories whose name is same as that of first column of "master_file" like :
ls -l sample
javacode
companion
sample
bradpitt
tomcruise
joseadams
adamsandler
...... and so on
and each of these directory contains two files namely primery.txt, secondry.txt somewhat like :
ls -l /sample/javacode/
primery.txt
secondry.txt
now my problem is to pick record from master_file search the corresponding directory in sample and append the id_number (second column from master_file) to primery.txt and secondry.txt in a way that primery.txt should contain atmost five entries and the rest will get append to secondry.txt
I came up with this script but its not working... also i know its not one of the standard form of scripting, maybe you can help me with a better solution
#!/bin/sh
while read line
do
user_name=$(echo $line | cut -d, -f1)
id_number=$(echo $line | cut -d, -f2)
main_directory=$(find /usr/myproject/sample -type d -name $user_name)
lines_in_popular=$(wc -l $main_directory/popular.txt | awk '{print $1 ;}')
if [ $lines_in_popular -lt 5 ] ; then
echo $id_number >> $main_directory/primery.txt ;
else
echo $id_number >> $main_directory/secondry.txt ;
fi
done < /usr/myproject/content_to_add/master_file
if "find" returns more than one file then "wc" will return more than one line which will break the -lt test. Try adding " | tail -1" after the find command inside the parenthesis.
I suppose your master file has unique lines and doesn not contain doubles like with "adamsandler" from your example. Is it so?
Your script has several rather weak spots. Lets go over them one by one:
The first thing is you do not explicitly state a shell to be used. This is not an error, but why take chances? Always state in the first line your intended shell with a "shebang": "#! /path/to/your/shell". I will use "/bin/ksh" in my examples, but change that to whatever you really want to use.
You first read in a line, then spend several commands to split this line. You can save an awful lot of execution time by using shell variable expansion instead of "cut":
user_name="${line%%,*}"
id_number="${line##*,}"
but even faster and saving even more would be to let the shell itself do the splitting by redefining the IFS so that word splitting is done implicitly:
while IFS=',' read user_name id_number ; do
....
done < /path/to/master.file
Another point is:
main_directory=$(find /usr/myproject/sample -type d -name $user_name)
The output of "find" would be several lines if "/usr/myproject/sample/$user_name" would contain a subdirectory or several subdirectories) with the username. Consider the following directory structure:
Furthermore you do not take any precautions against the directory missing at all. Instead of finding a directory you already know to be there you could just construct its name and then test against it (see "-d" option of "test"):
#! /bin/ksh
while IFS=',' read user_name id_number ; do
main_dir="/usr/myproject/sample/$user_name"
if [ -d "$main_dir" ] ; then
print - "The directory $main_dir exists."
else
print - "something went wrong, $main_dir does not exist."
fi
...
done < /path/to/master.file
No, my master file does contain duplicate username but i dont think it should make any difference since we are trying to execute script for each and every individual line.. please correct me if i am wrong
Thanks i will take care of it in future
I am not very good at shell variable expansion could you suggest me some references to read
My directory structure does not have multiple directory but yes there was no precaution in case if directory does not exist, i am gonna adopt your method from now on
Yes that's a typo and signifies the id_number, same with popular.txt and notpopular.txt too
I am gonna try and change the script as per your suggestion could you please also help for the error that ai am getting while testing the number of lines? i.e
First off, could you please use standard colour for your normal text, instead of red? If you use the proper tags ("quote", "code", like you did) it is easy to diffrentiate between question and answer and other colours may make it hard to read when usign different colour schemes. Thank you.
You are right. I just wanted to get a better picture of what can be expected as input for the script.
Just read the man page for ksh/bash (in this regard they work the same). You will find it under "variable expansion" or "parameter expansion" and it is basically a set of string-related functions with which you can extract sub-strings from variables. Example: i wrote in my first answer
user_name="${line%%,*}"
id_number="${line##*,}"
This means: "$user_name" is loaded with the contents of "$line" up to the first "," - the same as your "cut -d',' -f1" - and "id_number" is loaded with "$line" from the first "," to the end. This will do the same as your "echo ... | cut" but without the necessity to start an external program. The difference might not be much in absolute time but calling an external program weighs in by a factor of about 100 compared to an internal shell-function. Do it often enough and you will see a very noticeable difference.
That is probably coming from the file "primery.txt" not being there. You once call the two files in this directory "primery.txt" and "secondry.txt" and the other time you call them "popular.txt" and "notpopular.txt" - what is it gonna be?
Here is the modified script that i come up with your suggestion:
#!/bin/bash
typeset -i lines_in_primery=0
while IFS=',' read name id_number ; do
name_directory=/usr/myproject/content_to_add/config/$name
if [ -d $name_directory ] ; then
lines_in_primery=$(sed -n '$ =' $name_directory/primery.txt)
if [ $lines_in_primery -lt 5 ] ; then
echo $id_number >> $name_directory/primery.txt
else
echo $id_number >> $name_directory/secondry.txt
fi
else
echo directory $name does not exit >>logs ;
fi
done < /usr/myproject/content_to_add/master_file
I have done some initial round of tests and its working fine, will check the complete functionality.
I think the error
was because i was not setting "lines_in_variable" to integer. whats your opinion?
I'm glad it works. Why your error occurred i have no idea. The shell soes not really differentiate between data types like high-level-languages and even if a string variable contains a numeric value it should work:
x="5"
if [ $x -lt 10 ] ; then
echo "it works"
fi
Will work. It would only throw an error when "$x" contains something which can't be interpreted as integer, like "blabla".
The error message suggests that the variable was completely empty, so probably something went wrong when you assigned it a value.
A final suggestion: quote variable contents! For instance, try the following lines:
name="abc"
name_directory=/usr/myproject/content_to_add/config/$name
if [ -d $name_dir ] ; then
echo "It works."
fi
If you create the directory "/usr/myproject/content_to_add/config/abc" this will produce the message as expected. Now create the directory "/usr/myproject/content_to_add/config/ab c" (it is possible if you enclose it in double quotes) and try the above code again, replacing "abc" with "ab c". It will throw an error.
The reason is the shell has to somehow know where a word ends and another starts. Every time it encounters a space char it declares the word it is reading at that moment to be finished. If you type "mv a b" it this leads to "mv", "a" and "b" being interpreted as different words (the first being a command, the two others file names) and this is like one would expect things to work. But suppose a file is named "a b": how would you tell the shell to move that? This is called "word splitting" and it is performed on every line during the parsing process. Enclosing something in double quotes is to protect that part from this word splitting and therefore it is a good idea to write:
name="abc"
# name="ab c"
name_directory="/usr/myproject/content_to_add/config/$name"
if [ -d "$name_dir" ] ; then
echo "It works."
fi
Which you will notice will work even with a file with spaces in its name. Try it out!
