any way to speed up calculations in bash script

Shell Programming and Scripting
1

hi i have a script that is taking the difference of multiple columns in a file from a value from a single row..so far i have a loop to do that.. all the data is floating point..fin [i]has the difference between array1 and array2..array1 has 700 x 300= 210000 values and array2 has 700 values..

while [ $j -lt $outer ];do
b=${ARRAY2[j]}
i=0
while [ $i -lt $count1 ];do
a=${ARRAY1[k]}
fin=`echo $a - $b | bc`
printf "%.1f," ${fin} >> "$count".diff
((i++))
((k++))
done 
echo >> "$count".diff
((j++))
done

i am using bc to calculate the floating point subtraction. problem is i have about 600 columns and 300 rows..the script takes a long time to execute.. is there any way i can speed it up?

2

Don't call the external command 21,000 times. Pipe your calculations to a single instance of it.

And I would use awk rather than bc.

while [ $j -lt $outer ];do
  i=0
  while [ $i -lt $count1 ];do
    printf "%f\n" "${ARRAY1[k]} - ${ARRAY2[j]}"
    i=$(( $i + 1 ))
    k=$(( $k + 1 ))
  done 
  echo
  j=$(( $j + 1 ))
done | awk '/^$/ { print; next }
{ printf "%.1f," $1 - $2 }
'  > "$count.diff"

If you are processing the entire arrays:

for a in "${ARRAY1[@]}"
do
  printf "%f $a\n" ${ARRAY2[@]}
  echo
done | awk '/^$/ { print; next }
{ printf "%.1f,", $1 - $2 }
'
3

thanks. i am using bc because the numbers are floating point.. i also tried:

while [ $j -lt $outer ];do
  i=0
  while [ $i -lt $count1 ];do
    printf "%f\n" "${ARRAY1[k]} - ${ARRAY2[j]}"
    i=$(( $i + 1 ))
    k=$(( $k + 1 ))
  done 
  echo
  j=$(( $j + 1 ))
done | awk '/^$/ { print; next }
{ printf "%.1f," $1 - $2 }
'  > "$count.diff"

and i got an error:

./lines1: line 77: printf: -0.5422 - : invalid number
./lines1: line 77: printf: -0.5422 - : invalid number
./lines1: line 77: printf: -0.5422 - : invalid number
./lines1: line 77: printf: -0.5422 - : invalid number
./lines1: line 77: printf: -0.5422 - : invalid number
./lines1: line 77: printf: -0.5422 - : invalid number
./lines1: line 77: printf: -0.5422 - : invalid number
./lines1: line 77: printf: -0.5422 - : invalid number
./lines1: line 77: printf: -0.5422 - : invalid number
./lines1: line 77: printf: -0.5422 - : invalid number

seemed like it was taking the difference but errored out..do i still need bc?

4

printf cannot subtract numbers for you, that's what bc is for.

5 
    printf "%f %f\n" "${ARRAY1[k]}" "${ARRAY2[j]}"
6

that is just going to print the values.. i still need to take the difference.. is there any way i can round the data values in ARRAY1 and ARRAY2? this way i can get rid of the extra precision and speed up my calculations..

7

It prints the values which are piped to awk where the subtraction is performed.

That's done in awk.

Changing the precision will not affect the speed.

8

ok i tried the printf and here is the error i got:

while [ $j -lt $outer ];do
  i=0
  while [ $i -lt $count1 ];do
    printf "%f\n" "${ARRAY1[k]} - ${ARRAY2[j]}"
    i=$(( $i + 1 ))
    k=$(( $k + 1 ))
  done 
  echo
  j=$(( $j + 1 ))
done | awk '/^$/ { print; next }
{ printf "%.1f," $1 - $2 }
'  > "$count.diff"

awk: cmd. line:2: (FILENAME=- FNR=1) fatal: not enough arguments to satisfy form
at string
        `%.1f,-0.006148'
           ^ ran out for this one

its erroring out at that awk printf statement..

9 
printf "%f %f\n" "${ARRAY1[k]}" "${ARRAY2[j]}"
10

ok that worked..one last thing.. i have a newline in my original script as follows:

[code]
#while [ $j -lt $outer ];do
#b=$(printf %.1f ${ARRAY2[j]})
#i=0
#while [ $i -lt $count1 ];do
#a=$(printf %.1f ${ARRAY1[k]})
#fin[i]=`echo $a - $b | bc`
#printf "%.1f," ${fin[i]} >> "$count".diff
#((i++))
#((k++))
#done
#echo >> "$count".diff # i have this newline here
#((j++))
#done

[\code]

how do i get a newline using the awk statement?

11

Replace echo with:

printf "\n\n"

Or modify the awk code:

/^$/ { printf "\n\n"; next }
12

ok great that worked!! thanks a lot for your help..