bctl is a tool written in C (#+?) by DGPickett.
He shared the code, so you can compile it on your system.
tui-psm is tool written in bash by me. (psm stands for paralell script manager)
I shared the code and made it part of its own dependency (TUI).
There was a discussion wether to use kill , ps or /proc way to identify the processes.
For me, ps worked the best.
So, give the others a try, if they work better for you - switch, otherwise keep using ps.
hth
It looks like it should work just fine (although it creates an unneeded process every time through the loop) if you want to completely ignore some command input read from your input file, never run those commands through my_job , and do not want any record of the fact that you chose to ignore those input file command lines.
Every additional process you run, slows down your system for all users on your system and slows down your script (and reduces the number of input lines you'll be able to process). This is an expected side effect when you need to run a process. It is just wasteful when you create processes that are not needed!
For example, consider changing:
NUM_PROCS=`ps -u root | grep my_job | wc -l`
to:
NUM_PROCS=`ps -u root | grep -c my_job`
changing:
cat /home/gmark/rje/COMMANDS.csv | grep "A[BL][AE]" > ${LOCAL_CSV}
> ${REMOTE_CSV}
LOCAL_CSV_SIZE=`wc -l ${LOCAL_CSV} | sed "s;^ *;;" | sed "s; .*;;"`
to:
grep "A[BL][AE]" < /home/gmark/rje/COMMANDS.csv > ${LOCAL_CSV}
> ${REMOTE_CSV}
LOCAL_CSV_SIZE=`wc -l < ${LOCAL_CSV}`
etc.
You still haven't given any indication about what type of system you're using, how many CPUs (or cores) are in your system, and what else might be running on your system that needs run while you're running this stuff. Without that information, I have to assume that setting MAX_NUM_PROCS to 400 will run significantly slower than setting MAX_NUM_PROCS to the number of cores on your system -2.
I would also assume that waiting for a job to complete once in a while instead of running a loop that runs ps and grep throttled only by a sleep 1 would consume considerably fewer resources and actually complete more jobs. (But of course that would require you to look at some of the other suggestions that have been made suggesting other ways to limit the number of background jobs you're running simultaneously.)
Some of the coding decisions I've made are to allow rapid prototyping and debugging, such as "read" statements in the loop to allow me to step through it slowly. Eventually this will all be removed or changed.
I think the idea of putting a "wait" command in there occasionally is a great idea!
Inside "my_job", I'm basically making several database queries and then using these to decide whether to make a hardware operation, which might take minutes. "my_job" leaves some output data and status in a small file that I go around periodically to collect, and possibly spawn a new "my_job" to respond to.
Is there a possibility of the "my_job" just hanging that there is a more elegant way of checking and purging? I'd hate to have a "wait" command in line that has 20 jobs all of which will complete and then the 21st stops the system by hanging.
Is there some way of handling this, such as a "wait" that has a "time limit"?
@ Don, might this mean counting cpus or threads?
$ grep -i cores /proc/cpuinfo
cpu cores : 4
...
4 - 2 = 2 paralell running jobs?
Or if counting threads:
$ grep -i cores /proc/cpuinfo -c
8
$ grep cpu[0-99] /proc/stat -c
8
8 - 2 = 6 paralell running jobs?
EDIT:
@ gmark99
Did you try:
sleep 15
sleep 0.5
CPU's.
To get a count, count 'core id' lines.
$ grep -c 'core id' /proc/cpuinfo
2
$
Keep in mind, once the jobs are in the background, they are running, regardless if you sleep in the foreground, so that has no effect.
Furthermore, it cant hurt the give the cpu's a second break between the jobs it executes.
Also, you could implement that check in the loop itself rather than in the background as well.
In that case, you might even be able to 'leave out' the sleep.
On the other hand, if the 'purge-job' takes several minutes, as you say, just to check if a process is running or not, you have a serious performance issue anyway.
Which could be either too many jobs, or a badly coded parser.
(EDIT2: on rethinking, i'm very sure the issue is here, since with every loop you start a background job lasting several minutes and doing db and cpu checks)
EDIT:
@ Corona, ok thank you.
So for you this would mean no background jobs then?
While i could do 6?
With any at least less modern hardware one should be able to run at least one background job, or is one not?
I'm not sure how you get "no background jobs" from "two cores". Without timesharing, two cores can run two things.
We could both run 30 if we wanted to, they'd just have to timeshare. I'd have each core running 15 jobs at a little less than 1/15th speed. You'd have each core running 5 jobs at a little less than 1/5th speed.
For maximum efficiency, and assuming each process keeps itself busy 100% of the time, don't exceed the number of cores.
Also, there's things which running things in parallel won't help with -- I/O like deleting files, for example. Two processes deleting from one disk doesn't make it spin twice as fast.
In my case, "my_job" does some database queries and decides to call a function that will result in a cross-processor message and then wait for a result to be returned.
I would still benefit from a lot of jobs in parallel if they're just waiting for results from another machine, right? This waiting for outside resources is the basic reason for my wanting to run so many jobs in background.
Am I missing anything, or am I on the right track?
Well, how many queries can your database server really run at the same time? There's no point exceeding that. 5,000 processes sitting around waiting for the database server are no faster than 5 processes sitting around waiting for the database server.
Also, what exactly does your query do? Might it create lots of temporary tables, consume lots of memory, ask for things on opposite ends of the disk, lock tables so everyone else has to wait, etc? If you push any of your server's limits, be they bandwidth, disk, or CPU, you lose.
Spamming hundreds of network connections causes its own bottlenecks, too.
I think you should look into having one job doing n queries, instead n jobs doing 1 query. There's a lot of overhead in making and breaking connections over and over. Besides, hammering any kind of server like that is just rude. If you can do that, try 2 jobs doing n/2 queries and see if that's an improvement or not.
Do those 'my_job's handle the waiting on external resources?
If so and if they are idle - as in waiting on the resource to become available, do they sleep a little bit, or are they constantly attempting to 'read' (didnt find a better word in this context) from that outside response??
Have you heard of denial of service attacks? Flooding a database server with more requests than it can handle doesn't speed things up; it grinds service to a halt.
We assume that you want to increase transaction throughput rather than to increase the loads on your CPUs, disks, networks, etc. Increased loads on system facilities is not linearly related to throughput. Increasing the load on a loaded (or overloaded) resource frequently reduces throughput rather than increasing it.
Unfortunately we built the database custom as well, so we'll have to test out the capacity. In reference to another comment, each job does six queries, and they're not very time sensitive at all. If we have to throttle requests, we can do that. What guidance you folks have provided so far is invaluable. I'll explore the advice I've gotten and come back if I have more questions, which I'm very likely to have.
Thanks!!!
---------- Post updated 01-30-15 at 09:55 AM ---------- Previous update was 01-29-15 at 07:45 PM ----------
Reply to sea:
The "my_job" processes just wait for the returns of function calls that read the data or that are operating hardware.
---------- Post updated at 09:59 AM ---------- Previous update was at 09:55 AM ----------
I have a couple more questions.
If I execute a "wait" command, does that cause a wait until ALL the child background processes have completed?
It was suggested I put in a "sleep 3" command after an "exit"? Did I get the right? Or after a "done"? Why, specifically?
Thanks!
Yes.
Some shells let you 'wait PID', to wait for one process in particular, but not all.
Does 'not very time sensitive' mean 'very short and fast'?
In that case there could be a lot of benefit to doing the loop on the database instead of locally. The time it takes to make and break the connection could be on par with the time it takes to do the query, leading to quite a lot of waste.
I usually double the core count, as threads are i/o blocked half the time at least. The idea is to keep the count in a managable range, and not waste resources with excess parallelism. A tool like bctl, at the end of a command generating pipeline, manages this for you with very low overhead, and then there is GNU Parallel, a much more complex but flexible paralelism tool.
Most bash, perl, shell speedup involves examining things in a loop, avoiding shelling out (`cmd` or '$(cmd)' ) when there is a shell built in that can do it, or using sed/awk in the pipeline to enhance the data stream so that a 'while read' sort of loop in eliminated or kept simpler. For instance, sed can convert a stream data lines into a stream of commands that can flow to bctl or a shell.