I tried to simulate some example in Shell particularly finding item and matching in another file.
A very simple example:-
Basically I have 2 files, file A and file B.
In file A, I have columns of fields such that:-
aaa 107
bbb 108
ccc 109
In file B, I have columns of fields such that:-
101 2 1
102 3 1
107 2 1
108 3 1
109 2 1
I would like to know, if I would like to extract let say first element of file A and compare with file B elements.
If found, I would like to have the position of element 107 in file B in this case it is on 3rd line. From the elements found in file B, I would like to perform some computation on their fields. And the same goes for other elements in file A which in the end will pipe to another output file.
Currently, I tried to do below:-
#!/bin/sh
cat a.txt | awk '
{
for $2 in a.txt
do
grep -i $2 b.txt
# printf("%d", ); #to print the position of the string found in b.txt
}'
cat b.txt| awk '
{
count[$i]=$2+$3;
}
END{
printf("%d",count[$i]);
}'
done
I could not get it working because of several reason:-
i)rom the grep option -i it match any numbers entries start with let say "1"
rather than the exact number.
ii)I could not figure out how to open two files similarly using AWK so I can extract fields.
Some of the initial complexity can be removed by using command join:
#!/usr/bin/env sh
# @(#) s1 Demonstrate a join, using a field different from 1.
set -o nounset
echo
debug=":"
debug="echo"
## Use local command version for the commands in this demonstration.
echo "(Versions displayed with local utility \"version\")"
version >/dev/null 2>&1 && version bash my-nl join
echo
echo " Input files:"
cat data1
echo
cat data2
echo
echo " Result of join:"
join -1 2 data1 data2
exit 0
You may still wish to use awk for the post-join processing, because that makes it easy to use the resulting fields. See man pages for details ... cheers, drl
However if we are using Join command, the position of the element in the matching file could not be retrived as similar elements are already grouped together.
I assume that the join results would be pipe into a new output file.
So far I have:-
cat a.txt| while read LINE
do
char=`echo "${LINE}"| awk '{print $2}'`
echo $char
cat b.txt| awk '{
#grep -n $char b.txt| tr ":" " "|awk '{print $1}'
done
However, I am stuck in how to retrieve the first field of b.txt.
Next, I am stuck in getting the current line found in b.txt
That is why I asked you to provide the desired results -- to let us see what you need. An example is better than 1000 words, yes?
#!/usr/bin/env sh
# @(#) s2 Demonstrate a join, with file position added.
set -o nounset
echo
debug=":"
debug="echo"
## Use local command version for the commands in this demonstration.
echo "(Versions displayed with local utility \"version\")"
version >/dev/null 2>&1 && version bash my-nl join
echo
echo " Input files:"
cat data1
echo
cat data2
echo
cat -n data2 >data3
cat data3
echo
echo " Result of join:"
join -1 2 -2 2 data1 data3
exit 0
#to get the second field of a.txt
char=`echo "${LINE}"| awk '{print $2}'`
echo $char
#to grep the line position of b.txt which match the element in a.txt
grep -n $char b.txt | tr ":" " "| awk '{print $1}'
done
The output is:-
1
2
1
3
1
2
100
The problem here is i) grep does not want to recoqnize only the first field of b.txt (which you can see for 3 in a.txt there is 1, 2 returned because there is "3" in b.txt for two entries.
ii)And there is problem if I applied grep -m 1 $char b.txt|awk '{print $1}'
for printing the matching element for first field of b.txt. The output would be
1
2
2
3
2
100
Ultimately, what I would like to have the desired otuput is:-
1 Not Found
2 Position 1 3+4
3 Position 2 4+5
100 Not Found
I am still using your original 2 data sets, which I call data1 and data2. The final output consists of a first set of matching lines, with all the fields from both files. The second set is the set of lines from data2 that is unmatched with the text string " Not found" appended.
The line position is given, as you can see, because the file was augmented with the line numbers by "cat -n".
The output from the first join should be easily read by awk, and the fields will be available to you, as usual, in variables "$1", "$2", etc.
If you need the reports to be different, you can change the join options -- see man join.
#!/usr/bin/env sh
# @(#) s3 Demonstrate a join, file position, unmatched lines.
set -o nounset
echo
debug=":"
debug="echo"
## Use local command version for the commands in this demonstration.
echo "(Versions displayed with local utility \"version\")"
version >/dev/null 2>&1 && version bash my-nl join
echo
echo " Input files:"
cat data1
echo
cat data2
echo
cat -n data2 >data3
cat data3
echo
echo " Result of join for matched lines:"
join -j 2 data1 data3
echo
echo " Result of join unmatchable lines from data2"
join -v 2 -j 2 data1 data3 |
sed -e 's/$/ Not found/'
exit 0
Producing:
% ./s3
(Versions displayed with local utility "version")
GNU bash 2.05b.0
my-nl (local) 296
join (coreutils) 5.2.1
Input files:
aaa 107
bbb 108
ccc 109
101 2 1
102 3 1
107 2 1
108 3 1
109 2 1
1 101 2 1
2 102 3 1
3 107 2 1
4 108 3 1
5 109 2 1
Result of join for matched lines:
107 aaa 3 2 1
108 bbb 4 3 1
109 ccc 5 2 1
Result of join unmatchable lines from data2
101 1 2 1 Not found
102 2 3 1 Not found
Now i see this as easier implementation compared to others.
Unfortunately,I just realized that the results expected is not what I expected at the first place. Sorry for that:( I just get confused.
Instead it is supposed to be in a.txt;each element is compared separated to b.txt at one time (b.txt is multiple text file which varies across different directories).
To give a bigger picture, a.txt contain all the elements to compare with different file of b.txt.
Can we still use join in this case?
Example:-
Input files:
a.txt
aaa 107
bbb 108
ccc 109
b.txt
101 2 1
102 3 1
107 2 1
108 3 1
109 2 1
b.txt
101 2 1
102 3 1
107 2 1
108 4 2
109 2 1
b.txt
101 2 1
102 3 1
107 2 1
108 4 2
109 6 1
At one time, each line of a.txt is read starting with "aaa 107" is match with elements in b.txt. Next, second line of a.txt which is "bbb 108" is match with elements in another file of b.txt which can be similar to earlier b.txt.
Ultimately, I would think your idea of join is great where it would be nice if
every element of a.txt could be joined with matching element in different b.txt file.
Desired output would be :-
aaa 107 2 1
bbb 108 4 2
ccc 109 6 1
I am so sorry for the earlier confusion of the intended otuput i would liek to see.Please advise.