How to combine data files using for loop

Shell Programming and Scripting
1

Hi,

I have 5 files basically;namely file1.txt situated each at folder A to E respectively.

I would like to extract out third column from each of these file1.txt from folder A to folder E. Also, I wanted to extract the first and second column which are common. In other words, e.g

A/file1.txt
1 1 10

B/file1.txt
1 1 20

C/file1.txt
1 1 30

D/file1.txt
1 1 40

E/file1.txt
1 1 50

I tried

dirs=(A,B,C,D,E);
files=(file1.txt);

for file in "${files[@]}";

do for dir in "${dirs[@]}";
do awk {'print $3 '} "$dir/$file" >test.txt

    done

done

This only able to print out the 3rd column separately. Ultimately, I wanted to have the below output in an external file (test.txt):-
1 1 10 20 30 40 50

The file1.txt might have more than one line. The above file1.txt data is just an example.

Thanks.

2

Try this:

dirs=(A,B,C,D,E);
files=(file1.txt);

for file in "${files[@]}";
do
    v=$(awk '{print $1,$2}' ${dirs[1]}/$file)
    for dir in "${dirs[@]}";
    do
        v="$v $(awk '{ print $3 }' $dir/$file)"
    done
    echo "$v" >> test.txt
done

I'm presuming the "1 1" on the beginning is the same in all files, and you just obtain it from the file in the first directory.

3

Not nice but works:

awk '{print $1,$2; while (getline) {print $3} }' `find A B C D E -type f -name "file1.txt" -print`| sed -e :a -e '{ N; s/\n/ /g;ta }'

1 1 20 30 40 50
4

Hi Annihilannic.

I still could not get the right output.

It doesnt combine the third element until the last line read from the first directory's file.

I modified to be something like below:-

dirs=(A B C D E); 
files=(file1.txt); 

for file in "${files[@]}"; 
do
     v=$(awk '{print $1,$2}' ${dirs[1]}/$file)
	  
     for dir in "${dirs[@]}"; 
	do 
	    w=$(awk '{print $5 }' $dir/$file)
         
        done
        echo "$v $w" >> ../test.txt
done

But its not working too, Basically i wanted to have 1 1 20(from dir A) 30(from dir B) 40 (from dir C) and 50(from dir D) corresponded with respect to each line.

E.g :-

A/file1.txt
1 1 10
2 2 100
B/file1.txt
1 1 20
2 2 1000

C/file1.txt
1 1 30
2 2 10000

D/file1.txt
1 1 40
2 2 100000

E/file1.txt
1 1 50
2 2 1000000

Output:- 1 1 10 20 30 40 50
2 2 100 1000 10000 100000 1000000

Please advise. Thanks.

5

Try (and adapt) the following script :

awk -v File=file1.txt -v Dirs_list="A,B,C,D,E" '
BEGIN {
   dirs_count = split(Dirs_list, dirs, ",");
   for (i=1; i<=dirs_count; i++)
      files = dirs "/" File;
   while (getline < files[1]) {
      out = $1 OFS $2 OFS $3;
      for (i=2; i<=dirs_count; i++) {
         getline < files;
         out = out OFS $3;
      }
      print out
   }
}
'

Input files:

> for dir in A B C D E
> do
> echo "--- $dir/file1.txt ---"
> cat $dir/file1.txt
> done
--- A/file1.txt ---
1 1 A11
1 2 A12
--- B/file1.txt ---
1 1 B11
1 2 B12
--- C/file1.txt ---
1 1 C11
1 2 C12
--- D/file1.txt ---
1 1 D11
1 2 D12
--- E/file1.txt ---
1 1 E11
1 2 E12
>

Output:

1 1 A11 B11 C11 D11 E11
1 2 A12 B12 C12 D12 E12

Jean-Pierre.

6

Hi Jean,

Thanks for the reply. I havent try the suggested solution but I am thinking how is that possible if i have about 30 files to perform the above operation.

Is that possible the first solution suggested by Annihilannic to be tweaked ?

Please advise. Thanks.

7

Another way :

for dir in A B C D E
do
   echo $dir/file1.txt
done | 
xargs paste |
awk '{out = $1 OFS $2; for (i=3; i<=NF; i+=3) out = out OFS $i; print out}'

Execution (step by step):

> for dir in A B C D E
do
    echo $dir/file1.txt
done
A/file1.txt
B/file1.txt
C/file1.txt
D/file1.txt
E/file1.txt
>

> for dir in A B C D E
do
    echo $dir/file1.txt
done |
xargs echo paste  # echo for debug purpose
paste A/file1.txt B/file1.txt C/file1.txt D/file1.txt E/file1.txt
>

> for dir in A B C D E
do
    echo $dir/file1.txt
done |
xargs paste
1 1 A11 1 1 B11 1 1 C11 1 1 D11 1 1 E11
1 2 A12 1 2 B12 1 2 C12 1 2 D12 1 2 E12
>

> for dir in A B C D E
do
    echo $dir/file1.txt
done |
xargs paste |
awk '{out = $1 OFS $2; for (i=3; i<=NF; i+=3) out = out OFS $i; print out}'
1 1 A11 B11 C11 D11 E11
1 2 A12 B12 C12 D12 E12
>

Jean-Pierre.