Faster than nested while read loops?

UNIX for Beginners Q & A UNIX for Dummies Questions & Answers
1

Hi experts,
I just want to know if there is a better solution to my nested while read loops below:

 
while read line; do
    while read line2; do
        while read line3; do
            echo "$line $line2 $line3"
        done < file3.txt
    done < file2.txt
done < file1.txt > output.txt

file1.txt:
1a
1b

file2.txt:
2a
2b

file3.txt:
3a
3b

output.txt:
1a 2a 3a
1a 2a 3b
1a 2b 3a
1a 2b 3b
1b 2a 3a
1b 2a 3b
1b 2b 3a
1b 2b 3b

I am thinking that a perl or an awk script could do this, which are much faster than while loops.

Thanks!

2

Why not use the join command or maybe paste?

paste file1 file2 file3
3

paste doesn't sound like what he wants as that just combines individual lines.

I don't think perl or awk would necessarily be faster as they are interpreted languages just like shell.

4

The while loops are not your problem; it's the repeated reading of the same file over and over again (file3 = L1*L2, file2 = L1, file1 = 1). Even if the entire file's contents are stored in the operating system's cache, while reading will be much faster than hitting the disk, a switch to kernel mode is still required (this is expensive compared to accessing process memory).

The easiest way to speed this up would be to store each file's lines in an array. While it's not necessary to use perl or awk for that purpose, since modern shells support arrays, it's a typical approach.

join(1) may not be appropriate either, as it requires both a common field (not present in the sample data) and pre-sorted data (which the sample data is, but the real data may not be).

Regards,
Alister

5

The fastest way, arguably, to read a file is process substitutuion in bash, ksh, or zsh.

a=$(< filename)
b=$(< filename2)

But based on his example and output it looks like he is pasting.