You're really asking two questions here. I'm not certain that you're clear on the difference between a thread and a process. But I will assume that you are and proceed from there.
First, I do not know of a shell with thread support. So as far as I know, no you cannot do threads in a shell script.
But it quite easy to launch processes in a shell script. And if you have multiple cpu's concurrent processing is easy.
In fact, concurrent processing is easy from the command line. Consider a command like:
ls -lR | grep mp3
If you have two cpu's, it is quite possible that both processes will run at the same time. No big deal here. But I have rewritten slow shell scripts to take advantage of multiple cpu's and I have seen a boost in performance.
Besides pipelines, you can put programs in the background. These may find their way into another cpu.
by thread i meant a java thread, i beleive it creates a seperate process in the background for each thread., not sure.
what i want is
sample.sh
Start
|
|
Copy detailtableA from one database to another (database connection 1)
Copy detailtableB from one database to another (database connection 2)
....
Copy detailtableN from one database to another (database connection N)
(I wanna do this simultaneously since they are not dependent on each other)
|
|
Get some flag when everything is complete
|
|
Copy master table from one database to another
|
|
End
The data is very important and goes from a production box to an archive box, so can't afford to lose data here.
I was able to do this in a Java program using threads, spanning many threads for each database connection and acheive the work, its pretty good.
But i want to know if i can do this in a shell script and stored procedure for each copy (row by row commit) it would be better, the volume is huge.
Those three programs should run simultaneously. The script will stall at the wait command until they are all done.
Immediately after you launch a background job, its pid is available in $! and you can store this pid:
some_job &
pid=$!
And you can wait for this particular job to finish with:
wait $pid
This all works pretty well. The downside is that it is very hard for these jobs to communicate with one another or even to communicate results back to the parent. Temporary files are the usual solution.
100 processes is too much unless you have 50 cpu's or more. I would not create any more than 2 processes per cpu as a max. And 5 or 10 processes would be more reasonable.
And remember that when you do:
something &
unless "something is a c program that does not fork, it actually may be several processes itself. Your os may prohibit you from creating many processes. On HP-UX maxuprc is 50 by default.
When there are processes waiting on the run queue for a cpu, the scheduler will switch processes so that they get a shot at the cpu. That's called a context switch and it can be expensive. Each cpu typically has enough on-board storage to remember how to run a few processes. But more than that and stuff must be reloaded from memory.
Since you are after performance, you should benchmark it. Increase the number of processes until it is counter-productive and then stop.