Hi guys,
I am new guy for unix. I need help writing the following shell script.
My requirement is:
I have few pl/sql procedures(lets say 10). I want to run some of them (say 5) parallally in the background and once these are completed trigger the next ones one after other.
Could some one give some ideas on how to keep track of the first 5 background processes and how it can be implemented.(some sample script would be appreciated.)
Thanks
Chandu
Welcome, this seems to be a homework question. You should post what you have worked out already. If this is a homework question, please read the Unix.com forum's Rules.
This may be a legitimate post. Create a temp table in your database, have each procedure post back to the table. This is how our project tends to do these sorts of things. Table is small and has columns such as: job ID, current status (in progress, success, failure) and a date/time column of the last update.Drop the table if it is not needed after your run.
This is actually our project requirement. I described here it as a small assignment, but actually we have about 150 procedures and some of them can be run parallally and some of them have dependency. Our objective is to write a shell script to schedule this task. I have no idea about how to start with and request your ideas/suggestions.
Google: could you be more specific. When the first set of procedures triggered in the background how do we monitor for its status and trigger the next dependent job when it is done. I mean do we need to use some kind script which keeps on checking this job status table in specific intervals?
thanks for your suggestions in advance.
We have a couple of tables set up:a job dependency table (holds job id and jod dependency ID among other things) and aother is a job master table (table containing all jobs and job ids) and finally a job status table. Shell scripts control the execution of stored procedures. Before a procedure is executed the shell runs a small sql statement to check the job status table to see if the current job is able to run. If the last status is success, and the dependency is also met everything is good. The controlling shell also updates the job status table based upon the status of the PL/SQL it kicked off (in progress, success, failure). this works well for us but we dont have 150 procedures to run in parallel either 
You may not need this but your first post mentions you are new to UNIX, disregard if you already know this. Here is a link that shows how you execute a stored procedure from a shell and capture output variables.
Once you follow google's tips you will have the individual jobs. Now let's say that you want job1, job2, and job3 to run in parallel followed by job4, job5, and job6 running in parallel. You could write a script:
#! /usr/bin/ksh
/path/to/job1 &
/path/to/job2 &
/path/to/job3 &
wait
/path/to/job4 &
/path/to/job5 &
/path/to/job6 &
wait
exit 0
The ampersands tell ksh to background the jobs. The "wait" without any parameters says to wait for all background jobs.
#! /usr/bin/bash
/path/to/job1 &
/path/to/job2 &
/path/to/job3 &
wait
/path/to/job4 &
/path/to/job5 &
/path/to/job6 &
wait
exit 0
join/to/job 1 % ...
pico
wait
exit 0,1
Hi:
Could you explain what does this extra code does.
-------
join/to/job 1 % ...
pico
wait
exit 0,1
-------
Thanks,
Chandu
What a complicated request! You are trying to code a work flow system in a shell script.
If you're new to UNIX, you will certainly have trouble wrapping your arms around someone else's scripts.
Conceptually, here is what you want to do:
Your first goal is to build the framework for interacting with Oracle. I find that communicating to Oracle with a sqlplus coprocess to be very easy to control your various interactions.
sqlplus -s /nolog |&
Then you send messages to the coprocess.
print -p "connect un/pw@db"
print -p "prompt 'sql complete'"
Then you can read back Oracle's responses simillarly as follows:
while read -p LINE
do
# This is necessary so that the loop doesn't freeze waiting for more output
if print $LINE | grep 'sql complete'
then
break
fi
# More result analysis...
if ...
done
This works for PL/SQL code wrappers, dynamic SQL, etc. In fact, the only thing that isn't so great is if you have 1000s of lines of results and you want to process them with the while read command.
The next thing that you need is a mechanism to drive your execution priorities. This can be done with one or more tables and defining some basic rules. The simplest will be the processing order.
You'll also want to identify the don't care procedures that can run in the background. This is more tricky but can be done fairly easy. I did this for my current project where I need to bulk load data using sqlldr and where I can't use direct path loads (this works at blazingly fast speeds; my tests showed nearly one million records per minute in one test case). I split the task into five threads and each thread loads a different portion of the file. You can sychronize information by wrapping the command in a pair of braces which causes the code to execute in a subshell. The direct the output to a common file for analyzing the results.
{
sqlldr un/pw@db ...
# put any kind of stuff inside the braces that will help you digest the processing results
print result code=$?
} >> some_output_file_common_to_all_threads &
# save the process ID
mypidlist="$mypidlist $!"
...
# later on, when you want to synchronize the threads you have to wait on the background pids.
# The following wait is more reliable than the unix wait (for me that is :( )
while [ $(ps -p $(print ${mypidlist} |
tr -d ' ' |
nawk '{print substr($0,1,length($0)-1)}') |
wc -l) -gt 1 ]
do
sleep 2
done
# now process information related to your thread's output
while read LINE
do
...
done < some_output_file_common_to_all_threads
Your serially executed procedures/tasks run in the coprocessor and your "don't care procedures" can be run in their own sqlplus sessions independent of the coprocessor.
Coprocess exacmple:
print -p "SET SERVEROUTPUT ON"
print -p "BEGIN"
print -p " call_my_proc;"
print -p "END;"
print -p "/"
print -p "PROMPT 'sql complete'"
while read -p LINE
do
... same as before
done
Background example:
{
sqlplus -s /nolog <<EOF
connect us/pw@db
SET SERVEROUTPUT ON
BEGIN
call_my_proc;
call_another_one_if_you_like;
DBMS_OUTPUT.PUT_LINE ('whatever message you want');
END;
/
EOF
} >> some_other_file_as_before &
mypids="$mypids $!"
If you really want to monitor status of the background procedures, use the facilities with Oracle for long oeprations and then use your coprocessor to periodically query the information. Or you can use your own table, whatever you want but the Oracle longops facility is designed for this simple kind of job communication.
This can also serve as a mechanism for synchronizing your background tasks.
This is a very simplistic description for solving your very complex requirement but I use this kind of technique now with great success.
Thomas
I have an SFTP coprocess whose output is redirected to a file (as shown in the script below). Now how can I read the output of the coprocess:
exec 4>$FTPLOG
sftp nakarapu@serverx >&4 2>&4 |&
print -p mget $filename
print -p !echo "The FTP is complete"
read -p LINE
echo $LINE
print -p bye
wait
What changes to the above code can make it read the LINE from the coprocess.
(Btw, this is an attempt to synchronize with the coprocess. If you have a better way to know when the coprocess finishes the ftp, please let me know.)
Thanks in advance
Naveen Akarapu
I have never been able to get ssh, scp, or sftp to work via a coprocess due to how it forks a child process. You need to use expect or Perl expect; that's what I have used.
T
expect is not installed in our systems.
But I am guessing there must be some kind of I/O indirection that allows me to read the output of the coprocess.
read -p LINE
That would normally do it but, as I said, secure shell commands fork a child process and that I/O can't be captured by the coprocess. Sorry.
t