1 script or multiple scripts?? - check files, run jobs

jnanasakti · November 4, 2008, 2:39pm

Question for anyone that might be able to help:

My objective is to eheck if a file (a source file) exists in a directory. If it does then, I'd like to call an application (Informatica ETL file...not necessary to know) to run a program which extracts data and loads it into multiple targets.

There are mutliple sources files to check for, and multiple applications to call, but the structure of the script should be the same for each source/target pair.

Hence would it be better to use 1 script which checks for all the files and runs a specific app depending on whether a specific file exists, or run a simply create 1 script template, and multiple scripts, 1 for each source/target pair?

The question also relates to job scheduling, schedule 1 script (which would be more advanced of a script considering all the looping or such), or schedule mutliple simpler scripts?

Any help is appreciated - novice script writer.

joeyg · November 4, 2008, 2:52pm

one script that executes all commands
one text file with all the parameters and instructions

Then, your script would execute a loop reading all the entries in your text file with all necessary parameters.

At least, that is how I woudl approach your question.

jim_mcnamara · November 4, 2008, 2:56pm

one way:

Write a simple driver script, then use a data file to supply program name(s), data file -
use "progname -p parm1" for programs that require options.

mycontrolfile:

datafile1 script.sh "someprogram1 -p parm1 -q parm2"
datafile2 script1.sh someprogram3 anotherprogram script3.sh

script

#!/bin/ksh
while read datafile prog1 prog2 prog3  prog4  prog5
do
    if [[ ! -s $datafile ]] ; then  # no data file, skip over it?
        continue
    fi
    # run up to 5 programs against the datafile
    for prog in $prog1 $prog2 $prog3 $prog4 $prog5
    do
          "$prog" "$datafile"
    done 
done < mycontrolfile

For example, $prog3 (on first line in the input file) will evaluate to "" if it is empty. So the for loop terminates early. On the second line of datafile $prog5 is "".

jnanasakti · November 4, 2008, 3:42pm

This sounds like a feasible approach, although at this point I'm not quite sure on how to reference a text file which is read through a loop in main script. But it sounds straightforward enough that when I do create the script I should be able to find tons of information calling text files (seems simple enough). Thanks for the info and suggestion, I've only written scripts sparecly in my time

jnanasakti · November 4, 2008, 3:49pm

jim mcnamara:

one way:

Write a simple driver script, then use a data file to supply program name(s), data file -
use "progname -p parm1" for programs that require options.

mycontrolfile:
datafile1 script.sh "someprogram1 -p parm1 -q parm2"
datafile2 script1.sh someprogram3 anotherprogram script3.sh
script
#!/bin/ksh
while read datafile prog1 prog2 prog3  prog4  prog5
do
   if [[ ! -s $datafile ]] ; then  # no data file, skip over it?
   continue
   fi
   # run up to 5 programs against the datafile
   for prog in $prog1 $prog2 $prog3 $prog4 $prog5
   do
   "$prog" "$datafile"
   done 
done < mycontrolfile
For example, $prog3 (on first line in the input file) will evaluate to "" if it is empty. So the for loop terminates early. On the second line of datafile $prog5 is "".

Thanks for the Info and I appreciate the level of detail! Looks like a great approach that I'll have to further understand and utilize when I create the scripts (I'm in design phase right now). Curious, when you say 'read data file' in the script, is that synonomous with 'read mycontrolfile'? Or are those two different things? In addition, if you state 'prog1 prog 2' etc in the script file, how does the reference to 'someprogram1', 'anotherprogram' come into play?

joeyg · November 4, 2008, 4:00pm

Here is a file I want to process

> cat file30
55 65 48 45 48 68 32 68 44 34 88 65
82 63 52 54 51 68 75 0 0 20 10 77
55 77 60 55 22 60 40 25 75 55 45 90
20 80 33 63 0 64 32 22 75 0 43 56
54 54 12 35 48 87 65 12 77 85 0 15

For each instance, there are 12 parameters.
Now my script:

while read x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 x11 x12
  do echo "start process"
  echo "x2 = "$x2
  echo "x9 = "$x9
done <file30

And its execution:

start process
x2 = 65
x9 = 44
start process
x2 = 63
x9 = 0
start process
x2 = 77
x9 = 75
start process
x2 = 80
x9 = 75
start process
x2 = 54
x9 = 77

My example only uses two of the variables I am reading in, but you should get the idea. I have a text file with paramaters, and then read through the file executing commands on my read variables.

jnanasakti · November 5, 2008, 12:10pm

joeyg:

Here is a file I want to process
> cat file30
55 65 48 45 48 68 32 68 44 34 88 65
82 63 52 54 51 68 75 0 0 20 10 77
55 77 60 55 22 60 40 25 75 55 45 90
20 80 33 63 0 64 32 22 75 0 43 56
54 54 12 35 48 87 65 12 77 85 0 15
For each instance, there are 12 parameters.
Now my script:
while read x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 x11 x12
  do echo "start process"
  echo "x2 = "$x2
  echo "x9 = "$x9
done <file30
And its execution:
start process
x2 = 65
x9 = 44
start process
x2 = 63
x9 = 0
start process
x2 = 77
x9 = 75
start process
x2 = 80
x9 = 75
start process
x2 = 54
x9 = 77
My example only uses two of the variables I am reading in, but you should get the idea. I have a text file with paramaters, and then read through the file executing commands on my read variables.

I think I see what you're saying here. But i didn't clarify that I won't be using unix to read the file. All I'm doing is using shell script to see if the file exists on a directory. If it does, the I'll call an external program to execute (extract and load data from) the file.

As such, I'm looking to see how I should handle multiple (source) files in the directory, all called by the same program (different call command parameters per different target structures). Do I make sense?