# Shell script for scheduling

**URL:** <https://community.unix.com/t/shell-script-for-scheduling/151192>\
**Category:** Shell Programming and Scripting\
**Created:** [January 21, 2005, 10:47pm UTC](https://community.unix.com/t/shell-script-for-scheduling/151192 "2005-01-21T22:47:34Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![chanduhyd4u](https://community.unix.com/letter_avatar/chanduhyd4u/32/6_449c533f586939b6e5f246632a46421f.png) [@chanduhyd4u](https://community.unix.com/u/chanduhyd4u)\
**Post date:** [January 21, 2005, 10:47pm UTC](https://community.unix.com/t/shell-script-for-scheduling/151192/1 "2005-01-21T22:47:34Z")

</div>

Hi guys,

I am new guy for unix. I need help writing the following shell script.  
My requirement is:

I have few pl/sql procedures(lets say 10). I want to run some of them (say 5) parallally in the background and once these are completed trigger the next ones one after other.

Could some one give some ideas on how to keep track of the first 5 background processes and how it can be implemented.(some sample script would be appreciated.)

Thanks  
Chandu

---

<div class="post-metadata">

**Author:** ![locustfurnace](https://community.unix.com/letter_avatar/locustfurnace/32/6_449c533f586939b6e5f246632a46421f.png) [@locustfurnace](https://community.unix.com/u/locustfurnace)\
**Post date:** [January 21, 2005, 11:35pm UTC](https://community.unix.com/t/shell-script-for-scheduling/151192/2 "2005-01-21T23:35:28Z")

</div>

Welcome, this seems to be a homework question. You should post what you have worked out already. If this is a homework question, please read the [Unix.com](http://Unix.com) forum's Rules.

> [@http://www.unix.com/showthread.php?t=2971](#):
>
> (6) Do not post classroom or homework problems.

---

<div class="post-metadata">

**Author:** ![google](https://community.unix.com/user_avatar/community.unix.com/google/32/65_2.png) [@google](https://community.unix.com/u/google)\
**Post date:** [January 22, 2005, 1:01am UTC](https://community.unix.com/t/shell-script-for-scheduling/151192/3 "2005-01-22T01:01:24Z")

</div>

This may be a legitimate post. Create a temp table in your database, have each procedure post back to the table. This is how our project tends to do these sorts of things. Table is small and has columns such as: job ID, current status (in progress, success, failure) and a date/time column of the last update.Drop the table if it is not needed after your run.

---

<div class="post-metadata">

**Author:** ![chanduhyd4u](https://community.unix.com/letter_avatar/chanduhyd4u/32/6_449c533f586939b6e5f246632a46421f.png) [@chanduhyd4u](https://community.unix.com/u/chanduhyd4u)\
**Post date:** [January 22, 2005, 1:13am UTC](https://community.unix.com/t/shell-script-for-scheduling/151192/4 "2005-01-22T01:13:18Z")

</div>

This is actually our project requirement. I described here it as a small assignment, but actually we have about 150 procedures and some of them can be run parallally and some of them have dependency. Our objective is to write a shell script to schedule this task. I have no idea about how to start with and request your ideas/suggestions.

Google: could you be more specific. When the first set of procedures triggered in the background how do we monitor for its status and trigger the next dependent job when it is done. I mean do we need to use some kind script which keeps on checking this job status table in specific intervals?

thanks for your suggestions in advance.

---

<div class="post-metadata">

**Author:** ![google](https://community.unix.com/user_avatar/community.unix.com/google/32/65_2.png) [@google](https://community.unix.com/u/google)\
**Post date:** [January 22, 2005, 1:24am UTC](https://community.unix.com/t/shell-script-for-scheduling/151192/5 "2005-01-22T01:24:31Z")

</div>

We have a couple of tables set up:a job dependency table (holds job id and jod dependency ID among other things) and aother is a job master table (table containing all jobs and job ids) and finally a job status table. Shell scripts control the execution of stored procedures. Before a procedure is executed the shell runs a small sql statement to check the job status table to see if the current job is able to run. If the last status is success, and the dependency is also met everything is good. The controlling shell also updates the job status table based upon the status of the PL/SQL it kicked off (in progress, success, failure). this works well for us but we dont have 150 procedures to run in parallel either 😉

---

<div class="post-metadata">

**Author:** ![google](https://community.unix.com/user_avatar/community.unix.com/google/32/65_2.png) [@google](https://community.unix.com/u/google)\
**Post date:** [January 22, 2005, 1:33am UTC](https://community.unix.com/t/shell-script-for-scheduling/151192/6 "2005-01-22T01:33:28Z")

</div>

You may not need this but your first post mentions you are new to UNIX, disregard if you already know this. [Here is a link](http://www.unix.com/showthread.php?s=&threadid=10240&highlight=PL%2FSQL) that shows how you execute a stored procedure from a shell and capture output variables.

---

<div class="post-metadata">

**Author:** ![Perderabo](https://community.unix.com/user_avatar/community.unix.com/perderabo/32/27_2.png) [@Perderabo](https://community.unix.com/u/Perderabo)\
**Post date:** [January 22, 2005, 9:55am UTC](https://community.unix.com/t/shell-script-for-scheduling/151192/7 "2005-01-22T09:55:31Z")

</div>

Once you follow google's tips you will have the individual jobs. Now let's say that you want job1, job2, and job3 to run in parallel followed by job4, job5, and job6 running in parallel. You could write a script:  
#! /usr/bin/ksh  
/path/to/job1 &  
/path/to/job2 &  
/path/to/job3 &  
wait  
/path/to/job4 &  
/path/to/job5 &  
/path/to/job6 &  
wait  
exit 0

The ampersands tell ksh to background the jobs. The "wait" without any parameters says to wait for all background jobs.

---

<div class="post-metadata">

**Author:** ![TurkishCoder](https://community.unix.com/letter_avatar/turkishcoder/32/6_449c533f586939b6e5f246632a46421f.png) [@TurkishCoder](https://community.unix.com/u/TurkishCoder)\
**Post date:** [January 23, 2005, 6:19am UTC](https://community.unix.com/t/shell-script-for-scheduling/151192/8 "2005-01-23T06:19:27Z")

</div>

#! /usr/bin/bash  
/path/to/job1 &  
/path/to/job2 &  
/path/to/job3 &  
wait  
/path/to/job4 &  
/path/to/job5 &  
/path/to/job6 &  
wait  
exit 0

join/to/job 1 % ...  
pico  
wait  
exit 0,1

---

<div class="post-metadata">

**Author:** ![chanduhyd4u](https://community.unix.com/letter_avatar/chanduhyd4u/32/6_449c533f586939b6e5f246632a46421f.png) [@chanduhyd4u](https://community.unix.com/u/chanduhyd4u)\
**Post date:** [January 24, 2005, 1:21pm UTC](https://community.unix.com/t/shell-script-for-scheduling/151192/9 "2005-01-24T13:21:51Z")

</div>

Hi:  
Could you explain what does this extra code does.  
-------  
join/to/job 1 % ...  
pico  
wait  
exit 0,1  
-------  
Thanks,  
Chandu

---

<div class="post-metadata">

**Author:** ![tmarikle](https://community.unix.com/letter_avatar/tmarikle/32/6_449c533f586939b6e5f246632a46421f.png) [@tmarikle](https://community.unix.com/u/tmarikle)\
**Post date:** [January 28, 2005, 5:53pm UTC](https://community.unix.com/t/shell-script-for-scheduling/151192/10 "2005-01-28T17:53:59Z")

</div>

> [@chanduhyd4u](#):
>
> This is actually our project requirement. I described here it as a small assignment, but actually we have about 150 procedures and some of them can be run parallally and some of them have dependency. Our objective is to write a shell script to schedule this task. I have no idea about how to start with and request your ideas/suggestions.

What a complicated request! You are trying to code a work flow system in a shell script.

If you're new to UNIX, you will certainly have trouble wrapping your arms around someone else's scripts.

Conceptually, here is what you want to do:

Your first goal is to build the framework for interacting with Oracle. I find that communicating to Oracle with a sqlplus coprocess to be very easy to control your various interactions.

```plaintext
sqlplus -s /nolog |&

```

Then you send messages to the coprocess.

```plaintext
print -p "connect un/pw@db"
print -p "prompt 'sql complete'"

```

Then you can read back Oracle's responses simillarly as follows:

```plaintext
while read -p LINE
do
    # This is necessary so that the loop doesn't freeze waiting for more output
    if print $LINE | grep 'sql complete'
    then
        break
    fi
    # More result analysis...
    if ...
done

```

This works for PL/SQL code wrappers, dynamic SQL, etc. In fact, the only thing that isn't so great is if you have 1000s of lines of results and you want to process them with the while read command.

The next thing that you need is a mechanism to drive your execution priorities. This can be done with one or more tables and defining some basic rules. The simplest will be the processing order.

You'll also want to identify the don't care procedures that can run in the background. This is more tricky but can be done fairly easy. I did this for my current project where I need to bulk load data using sqlldr and where I can't use direct path loads (this works at blazingly fast speeds; my tests showed nearly one million records per minute in one test case). I split the task into five threads and each thread loads a different portion of the file. You can sychronize information by wrapping the command in a pair of braces which causes the code to execute in a subshell. The direct the output to a common file for analyzing the results.

```plaintext
{
    sqlldr un/pw@db ...
    # put any kind of stuff inside the braces that will help you digest the processing results
    print result code=$?
} >> some_output_file_common_to_all_threads &

# save the process ID
mypidlist="$mypidlist $!"
...
# later on, when you want to synchronize the threads you have to wait on the background pids.
# The following wait is more reliable than the unix wait (for me that is :( )
while [ $(ps -p $(print ${mypidlist} |
          tr -d ' ' |
          nawk '{print substr($0,1,length($0)-1)}') |
          wc -l) -gt 1 ]
do
   sleep 2
done

# now process information related to your thread's output
while read LINE
do
    ...
done < some_output_file_common_to_all_threads

```

Your serially executed procedures/tasks run in the coprocessor and your "don't care procedures" can be run in their own sqlplus sessions independent of the coprocessor.

Coprocess exacmple:

```plaintext
print -p "SET SERVEROUTPUT ON"
print -p "BEGIN"
print -p " call_my_proc;"
print -p "END;"
print -p "/"
print -p "PROMPT 'sql complete'"

while read -p LINE
do
   ... same as before
done

```

Background example:

```plaintext
{
sqlplus -s /nolog <<EOF
connect us/pw@db
SET SERVEROUTPUT ON
BEGIN
     call_my_proc;
     call_another_one_if_you_like;
     DBMS_OUTPUT.PUT_LINE ('whatever message you want');
END;
/
EOF
} >> some_other_file_as_before &
mypids="$mypids $!"

```

If you really want to monitor status of the background procedures, use the facilities with Oracle for long oeprations and then use your coprocessor to periodically query the information. Or you can use your own table, whatever you want but the Oracle longops facility is designed for this simple kind of job communication.

This can also serve as a mechanism for synchronizing your background tasks.

This is a very simplistic description for solving your very complex requirement but I use this kind of technique now with great success.

Thomas

---

<div class="post-metadata">

**Author:** ![akar\_naveen](https://community.unix.com/letter_avatar/akar_naveen/32/6_449c533f586939b6e5f246632a46421f.png) [@akar\_naveen](https://community.unix.com/u/akar_naveen)\
**Post date:** [August 12, 2008, 5:42pm UTC](https://community.unix.com/t/shell-script-for-scheduling/151192/11 "2008-08-12T17:42:08Z")

</div>

I have an SFTP coprocess whose output is redirected to a file (as shown in the script below). Now how can I read the output of the coprocess:

exec 4\>$FTPLOG  
sftp nakarapu@serverx \>&4 2\>&4 |&  
print -p mget $filename  
print -p !echo "The FTP is complete"  
read -p LINE  
echo $LINE  
print -p bye  
wait

What changes to the above code can make it read the LINE from the coprocess.

(Btw, this is an attempt to synchronize with the coprocess. If you have a better way to know when the coprocess finishes the ftp, please let me know.)

Thanks in advance  
Naveen Akarapu

---

<div class="post-metadata">

**Author:** ![tmarikle](https://community.unix.com/letter_avatar/tmarikle/32/6_449c533f586939b6e5f246632a46421f.png) [@tmarikle](https://community.unix.com/u/tmarikle)\
**Post date:** [August 12, 2008, 5:54pm UTC](https://community.unix.com/t/shell-script-for-scheduling/151192/12 "2008-08-12T17:54:18Z")

</div>

> [@akar\_naveen](#):
>
> I have an SFTP coprocess whose output is redirected to a file (as shown in the script below). Now how can I read the output of the coprocess:
> 
> exec 4\>$FTPLOG  
> sftp nakarapu@serverx \>&4 2\>&4 |&  
> print -p mget $filename  
> print -p !echo "The FTP is complete"  
> read -p LINE  
> echo $LINE  
> print -p bye  
> wait
> 
> What changes to the above code can make it read the LINE from the coprocess.
> 
> (Btw, this is an attempt to synchronize with the coprocess. If you have a better way to know when the coprocess finishes the ftp, please let me know.)

I have never been able to get ssh, scp, or sftp to work via a coprocess due to how it forks a child process. You need to use expect or Perl expect; that's what I have used.

T

---

<div class="post-metadata">

**Author:** ![akar\_naveen](https://community.unix.com/letter_avatar/akar_naveen/32/6_449c533f586939b6e5f246632a46421f.png) [@akar\_naveen](https://community.unix.com/u/akar_naveen)\
**Post date:** [August 12, 2008, 6:21pm UTC](https://community.unix.com/t/shell-script-for-scheduling/151192/13 "2008-08-12T18:21:36Z")

</div>

expect is not installed in our systems.

But I am guessing there must be some kind of I/O indirection that allows me to read the output of the coprocess.

---

<div class="post-metadata">

**Author:** ![tmarikle](https://community.unix.com/letter_avatar/tmarikle/32/6_449c533f586939b6e5f246632a46421f.png) [@tmarikle](https://community.unix.com/u/tmarikle)\
**Post date:** [August 12, 2008, 6:24pm UTC](https://community.unix.com/t/shell-script-for-scheduling/151192/14 "2008-08-12T18:24:48Z")

</div>

> [@akar\_naveen](#):
>
> expect is not installed in our systems.
> 
> But I am guessing there must be some kind of I/O indirection that allows me to read the output of the coprocess.

```plaintext
read -p LINE

```

That would normally do it but, as I said, secure shell commands fork a child process and that I/O can't be captured by the coprocess. Sorry.

t