Loop through variables

I am pretty new to Unix. Trying to pick up some slack while a coworker is out on vacation.

Basically the script is working fine however when I go through the testing phase and have to make mods it is a pita.

Here is an example of what I have

#!/bin/ksh
if [ ! -s "${DATAPATH}/${src_file_1}.csv" ]
then
    echo "${src_file_1}.csv is missing" >> ${file_log}
    fi

That is a small piece. There are several other areas alot more complex. There are 36 csv files that we are loading into a reporting website every quarter. Each variable ties to a file name. That being said we have variable names from src_file_1 through src_file_36. My thought is that I could generate a loop that would change the variable name with each cycle.

Thanks in advance for any insight.

---------- Post updated at 09:24 AM ---------- Previous update was at 07:28 AM ----------

Got it figured out...

for i in ${src_file_1} ${src_file_2} ${src_file_3} ${src_file_4} ${src_file_5} ${src_file_6} \
${src_file_7} ${src_file_8} ${src_file_9} ${src_file_10} ${src_file_11} ${src_file_12} \
${src_file_13} ${src_file_14} ${src_file_15} ${src_file_16} ${src_file_17} ${src_file_18} \
${src_file_19} ${src_file_20} ${src_file_21} ${src_file_22} ${src_file_23} ${src_file_24} \
${src_file_25} ${src_file_26} ${src_file_27} ${src_file_28} ${src_file_29} ${src_file_30} \
${src_file_31} ${src_file_32} ${src_file_33} ${src_file_34} ${src_file_35} ${src_file_36} 
 
do 
if [ ! -s "${DATAPATH}/${i}.csv" ]
then
    echo "${i}.csv is missing" >> ${file_log}
    fi
    
done

That's painful. See if you have "seq" command in your system:

$ 
$ seq 1 10
1
2
3
4
5
6
7
8
9
10
$ 

If you do, then it could be used in a "for" loop:

for i in $(seq 1 36)
do
  echo "src_file_$i"
  # perform the test
done

Or you can always go for the plain old-fashioned "while" loop:

first=1
last=36
while [[ $first -le $last ]]
do
  echo "src_file_$first"
  # perform the test and then increment the iterator variable
  (( first += 1 ))
done

I tested seq and I dont have it so I would have to use the old school method

for i in $(seq 1 36)
do
  echo "src_file_$i"
  # perform the test
done

So in this example which I think will work, but confuses me a bit.

How do i get

echo "src_file_$i"

to work in a test like this

if [ ! -s "${DATAPATH}/${src_file_1}.csv" ]

So how do I go about replacing ${src_file_1} with echo "src_file_$i" for the test. Hope that makes sense.

This would be relevant for either the "for" or "do while" methods

---------- Post updated at 01:04 PM ---------- Previous update was at 10:39 AM ----------

I am not one to sit idle especially when I am trying to figure something out so please look at the following:

Test: eval echo \$${src_file}
the_right_text
test: src_file_name=`eval echo \$${src_file}`
test: echo ${src_file_name}
29199{src_file}

so why when I manually do the eval it comes up with the right text, but when assigning the eval to a variable and then echo it it comes out wierd?

I am unable to reproduce your test in my system. This looks like the output of some shell script. I'll need to look at the script to figure out what's going on.

However, what's happening at the end is that the value of the variable "src_file_name" is the following two values appended to each other in the order shown:
(1) the value of $$
(2) the constant text "{src_file}"

Now, $$ is a built-in shell variable which holds the value of the PID, the process id. You can check the PID of each process in your shell by executing the following command and looking at the value in the 2nd column:

ps -ef

You can also check the value of your current shell PID:

echo $$

And view it and its descendent processes in the process list:

ps -ef | grep <the_number_you_saw_for_$$_above>

As for your problem, you are creating variable names within the loop, and you want to access their values for your file names. There is a construct "${!variable}" for that.

A simple test follows. I create and assign a variable and echo its value:

$ 
$ pointer_to_val=123456
$ echo $pointer_to_val
123456
$ 

Now, I assign the text "pointer_to_val" to another variable and echo it:

$ 
$ val=pointer_to_val
$ echo $val
pointer_to_val
$ 
$ 

So far, everything is as expected.

But in the last echo command above, let's say I want to see the value of pointer_to_val, i.e. I want to see the value 123456 that pointer_to_val points to.

To do that, I'd use ${!variable} construct:

$ 
$ echo ${val}
pointer_to_val
$ 
$ echo ${!val}
123456
$ 

This idea could be incorporated in your script as follows:

$ 
$ cat -n check_for_files_1.sh
     1    #!/usr/bin/ksh
     2    DATAPATH="/my/data/dir"
     3    src_file_1=file_one.txt
     4    src_file_2=file_two.sql
     5    src_file_3=file_three.py
     6    src_file_4=file_four.java
     7    src_file_5=file_five.c
     8    first=1
     9    last=5
    10    while [[ $first -le $last ]]
    11    do
    12      var=src_file_$first
    13      file=${!var}
    14      echo "${DATAPATH}/${file}"
    15      (( first += 1 ))
    16    done
$ 
$ 
$ . check_for_files_1.sh
/my/data/dir/file_one.txt
/my/data/dir/file_two.sql
/my/data/dir/file_three.py
/my/data/dir/file_four.java
/my/data/dir/file_five.c
$ 
$ 
$ 

And if the ${!variable} construct is not supported in your shell, you could use the old-fashioned eval construct:

$ 
$ cat -n check_for_files.sh
     1    #!/usr/bin/ksh
     2    DATAPATH="/my/data/dir"
     3    src_file_1=file_one.txt
     4    src_file_2=file_two.sql
     5    src_file_3=file_three.py
     6    src_file_4=file_four.java
     7    src_file_5=file_five.c
     8    first=1
     9    last=5
    10    while [[ $first -le $last ]]
    11    do
    12      var=src_file_${first}
    13      file=$(eval "echo \$$var")
    14      echo "${DATAPATH}/${file}"
    15      (( first += 1 ))
    16    done
$ 
$ 
$ . check_for_files.sh
/my/data/dir/file_one.txt
/my/data/dir/file_two.sql
/my/data/dir/file_three.py
/my/data/dir/file_four.java
/my/data/dir/file_five.c
$ 
$ 

I ended up getting it working. I wish the exclamation point piece of code would work on our box. Would have saved me hours of searching on the net.

This is wheat I ended up with. Might be a little crude but it works.

i=1
last=36
while [[ $i -le $last ]]
do
src_file="src_file"
ctl="_ctl"
eval src_file_name="\$src_file_$i"
eval ctl_file_name="\$src_file_$i$ctl"
      
    echo "************************************************************************" >> ${cron_log} 
    echo "File Details" >> ${cron_log}
    echo "------------------------------------------------------------------------" >> ${cron_log}
    echo "File Name: ${src_file_name}.csv" >> ${cron_log}
    echo "File Location: ${cron_dir}" >> ${cron_log}
    FILE_LINE_COUNT=`wc -l < ${cron_dir}/${src_file_name}.csv`
    echo "File Line Count:  $FILE_LINE_COUNT"   >> ${cron_log}
    echo " "  >> ${cron_log}
  
    echo "------------------------------------------------------------------------" >> ${cron_log}
    echo "Data Load Details:" >> ${cron_log}
    $ORACLE_HOME/bin/sqlldr ${ORA_DBCU}/${ORA_DBCP} control=${control_dir}/${ctl_file_name}.ctl
    grep -i "logical records read" ${cron_dir}/${ctl_file_name}.log >> ${cron_log}
    grep -i "logical records skipped" ${cron_dir}/${ctl_file_name}.log >> ${cron_log}
    grep -i "successfully loaded" ${cron_dir}/${ctl_file_name}.log >> ${cron_log}tina
    grep -i "not loaded" ${cron_dir}/${ctl_file_name}.log >> ${cron_log}
    echo " " >> ${cron_log}
    err_cnt=`grep "not loaded due to data error"  ${cron_dir}/${ctl_file_name}.log | cut -f3-3 -d" "`
    echo "Records not loaded due to errors: $err_cnt" >> ${cron_log}
    echo " " >> ${cron_log}
    
if ${src_file_name} -eq ${src_file_1}
then
      tot_err_cnt=$err_cnt
else
      tot_err_cnt=$(($tot_err_cnt+$err_cnt))
fi
  i=$(($i+1))
done

That's the correct use of eval. Thanks for posting it!

I'm glad it is working for you, but I'm surprised that the line I marked in red is working. It looks like you intended to execute test with the three operands trying to determine if two filenames were the same. But:

1. you are missing the test or [ and ] that would be required to make it a test command, and
2. when comparing strings, you should use = (or == if you were using [[ and ]] instead of test ) instead of -eq .

The way you have this if statement written, it is executing the 1st filename with the two arguments -eq and the 2nd filename. If that returns a zero exit status, the then branch is executed; otherwise the else branch is executed.

If the intent was to use one branch the 1st time through the loop, why not test for the value of i being 1 instead of comparing filenames? You might consider changing:

if ${src_file_name} -eq ${src_file_1}
then
      tot_err_cnt=$err_cnt
else
      tot_err_cnt=$(($tot_err_cnt+$err_cnt))
fi
  i=$(($i+1))

close to the end of your script to:

    if [ $i -eq 1 ]
    then
        tot_err_cnt=$err_cnt
    else
        ((tot_err_cnt += err_cnt))
    fi
    ((i++))

Your welcome, took me a few hours to find an example I could actually get working with what I was attempting to achieve. Glad it also helpded someone else out.

Original thread where I got the example from:

Thanks for the examples. I am actually surprised I got any of this to work as Unix definately isnt my forte. I am more oracle based. I dont know why I didnt think to us the i variable instead of comparing strings. Probably just tired. Thanks to everyone for the input. I have learned alot this week.

---------- Post updated at 09:05 AM ---------- Previous update was at 08:38 AM ----------

I did change to i variable instead of comparing strings as it made more sense, however for some reason ((i++)) did not work so I reverted that piece back to the way I had it.

# bash and ksh
((i+=1))
# posix standard = works also bash and ksh
i=$(($i + 1 ))