# Help in extracting multiple files and taking average at same time

**URL:** <https://community.unix.com/t/help-in-extracting-multiple-files-and-taking-average-at-same-time/209712>\
**Category:** Shell Programming and Scripting\
**Created:** [August 23, 2008, 11:23am UTC](https://community.unix.com/t/help-in-extracting-multiple-files-and-taking-average-at-same-time/209712 "2008-08-23T11:23:26Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![ahjiefreak](https://community.unix.com/letter_avatar/ahjiefreak/32/6_449c533f586939b6e5f246632a46421f.png) [@ahjiefreak](https://community.unix.com/u/ahjiefreak)\
**Post date:** [August 23, 2008, 11:23am UTC](https://community.unix.com/t/help-in-extracting-multiple-files-and-taking-average-at-same-time/209712/1 "2008-08-23T11:23:26Z")

</div>

Hi,

I have 20 files which have respective 50 lines with different values.

I would like to process each line of the 50 lines in these 20 files one at a time and do an average of 3rd field ($3) of these 20 files. This will be output to an output file.

Instead of using join to generate whole bunch of redundant files and then compute the average, im looking any other possible better way to do the above right away.

E.g

apple.txt  
tool1 2.00 4 30.20  
tool2 3.00 5 40.22  
tool3 2.00 6 45.32  
....  
tool50 ...........

orange.txt  
tool1 1.00 2 30.20  
tool2 4.00 3 40.22  
tool3 6.00 4 45.32  
...  
tool50 ...

bar.txt  
tool1 2.10 1 30.20  
tool2 3.04 4 40.22  
tool3 2.02 5 45.32  
...  
tool50 .....

and the remaining 17 files of different names.

The output would be:-  
tool1 (4+2+1+....)/20  
tool2 (5+3+4+...)/20  
tool3 (6+4+5+...)/20  
....  
tool50....

Please advise. THanks.

---

<div class="post-metadata">

**Author:** ![vgersh99](https://community.unix.com/user_avatar/community.unix.com/vgersh99/32/14851_2.png) [@vgersh99](https://community.unix.com/u/vgersh99)\
**Post date:** [August 23, 2008, 12:03pm UTC](https://community.unix.com/t/help-in-extracting-multiple-files-and-taking-average-at-same-time/209712/2 "2008-08-23T12:03:14Z")

</div>

Pls show the proof of the effort.

---

<div class="post-metadata">

**Author:** ![ghostdog74](https://community.unix.com/letter_avatar/ghostdog74/32/6_449c533f586939b6e5f246632a46421f.png) [@ghostdog74](https://community.unix.com/u/ghostdog74)\
**Post date:** [August 23, 2008, 12:03pm UTC](https://community.unix.com/t/help-in-extracting-multiple-files-and-taking-average-at-same-time/209712/3 "2008-08-23T12:03:39Z")

</div>

I see you have been here for a while. How about showing some code on what you did that doesn't work?

---

<div class="post-metadata">

**Author:** ![ahjiefreak](https://community.unix.com/letter_avatar/ahjiefreak/32/6_449c533f586939b6e5f246632a46421f.png) [@ahjiefreak](https://community.unix.com/u/ahjiefreak)\
**Post date:** [August 23, 2008, 6:04pm UTC](https://community.unix.com/t/help-in-extracting-multiple-files-and-taking-average-at-same-time/209712/4 "2008-08-23T18:04:51Z")

</div>

Hi,

My initial thought/idea was:-

!/bin/awk

filename={apple.txt,orange.txt,bar.txt................}

for file in filename;do

while(getline\<"file"); do

val+=$3;  
count++;

done

done  
tada=val/count;

print $1"," tada\>output.txt

That's my first intuition to the above case problem using awk.

---

<div class="post-metadata">

**Author:** ![ghostdog74](https://community.unix.com/letter_avatar/ghostdog74/32/6_449c533f586939b6e5f246632a46421f.png) [@ghostdog74](https://community.unix.com/u/ghostdog74)\
**Post date:** [August 23, 2008, 8:54pm UTC](https://community.unix.com/t/help-in-extracting-multiple-files-and-taking-average-at-same-time/209712/5 "2008-08-23T20:54:48Z")

</div>

Is that just your pseudocode or you are definitely writing an awk script? you seem to be mixing shell and awk syntaxes all over.

---

<div class="post-metadata">

**Author:** ![era](https://community.unix.com/letter_avatar/era/32/6_449c533f586939b6e5f246632a46421f.png) [@era](https://community.unix.com/u/era)\
**Post date:** [August 23, 2008, 9:41pm UTC](https://community.unix.com/t/help-in-extracting-multiple-files-and-taking-average-at-same-time/209712/6 "2008-08-23T21:41:54Z")

</div>

awk offers arrays for precisely this type of problem. Make each item with a count a key of the array, then at END print the results. If all keys occur in all files you can simply divide by the number of input files, otherwise you will need to collect both the sum and the count (divisor) for each key.

---

<div class="post-metadata">

**Author:** ![summer\_cherry](https://community.unix.com/letter_avatar/summer_cherry/32/6_449c533f586939b6e5f246632a46421f.png) [@summer\_cherry](https://community.unix.com/u/summer_cherry)\
**Post date:** [August 25, 2008, 1:43am UTC](https://community.unix.com/t/help-in-extracting-multiple-files-and-taking-average-at-same-time/209712/7 "2008-08-25T01:43:22Z")

</div>

below code should be ok, it assume all file with same naming convention .txt,  
nomatter how many files, it can process them

```nohighlight
sum=`ls -l *.txt | wc -l`
paste -d" " *.txt | nawk -v s="$sum" '{
	for(i=0;i<=s-1;i++)
	{
		t=3+(i*4)
		temp=temp+$t
	}
	print $1"---->"temp/s
	temp=0
}'

```

---

<div class="post-metadata">

**Author:** ![manosubsulo](https://community.unix.com/letter_avatar/manosubsulo/32/6_449c533f586939b6e5f246632a46421f.png) [@manosubsulo](https://community.unix.com/u/manosubsulo)\
**Post date:** [August 25, 2008, 2:47am UTC](https://community.unix.com/t/help-in-extracting-multiple-files-and-taking-average-at-same-time/209712/8 "2008-08-25T02:47:50Z")

</div>

This might help you..

rm -f output.txt  
count=1  
while ["$count" -le "50"]  
do  
grep\_var="tool$count"  
f\_avg=`grep "$grep_var" *.txt | awk '{sum+= $3;} END { print sum/NR}'`  
echo "$grep\_var : $f\_avg" \>\> output.txt  
count=`expr $count + 1`  
done

Note:  
Output will be written in output.txt file

---

<div class="post-metadata">

**Author:** ![era](https://community.unix.com/letter_avatar/era/32/6_449c533f586939b6e5f246632a46421f.png) [@era](https://community.unix.com/u/era)\
**Post date:** [August 26, 2008, 2:55am UTC](https://community.unix.com/t/help-in-extracting-multiple-files-and-taking-average-at-same-time/209712/9 "2008-08-26T02:55:46Z")

</div>

The approach proposed by summer\_cherry is kind of clever, although you should note that it has several assumptions: it assumes that all the files contain all the same tools, in the same order, and that they all have exactly four columns.

Also, if there are many files, the backticks will overflow. Moreover, it's rather excessive to use ls -l when you don't care for the long format, just the number of files. And since you are pasting the files, the number of columns will indicate how many files there are, so you can calculate the sum in awk from that.

```nohighlight
paste -d " " *.txt | nawk '{
  for (i=0; i < $NF; ++i) sum+=$(3+(i*4))
  print $1 " " sum/($NF*4)
}'

```
