Say you got a for loop where each execution has 0 dependence on the other. Thus ideally you'd like to executed them all concurrently rather than iteratively (if you had enough CPUs). We don't quite have that many CPUs but I would like to instead partition the iterations between them.
Or maybe even partition between CPU cores.
How to do either using bash commands? Thanks
You'd run the commands inside in the background, and each instance could run separately on one core. You'd have to be careful to limit it to the number of cores available.
Different commands in the same pipe chain can run concurrently with no further effort on your part, by the way.
Without seeing more details on what you want to run parallel, it's hard to give more details.
I'll rephrase. Say you have:
for VARIABLE in 1 2 3 4 5 .. N
do
command1
command2
commandN
done
How to tell each iteration to execute commands 1, 2, and N on a CPU and CPU core of my choosing? Lets not worry about how to partition them yet.
IT would help if you gave your UNIX OS flavor, too. Output of
uname -a
You can do what you request on some different UNIXes but the commands are totally different.
It's Yellow Dog Linux, although I could conceivably run it from a Red Had Enterprise Linux machine as well.
That's decided by the operating system. You run n commands simultaneously and if there's enough CPU's, it will run them concurrently.
for X in 1 2 3 4 5
do
command &
done
wait
I don't know if there's a standard utility to do so, but 'processor affinity mask' is what you're looking for if you really want to. But there's almost never any point -- the OS knows what CPU's are busy, and you don't.
Use the taskset command. You need some privilege to use it.
The syntax is:
for i in something
do
taskset [cpumask] command $i &
taskset [cpumask] command2 $i &
taskset [cpumask] command3 $i &
wait
done
[cpumaskl] is a hexadecimal mask : 0x00000001 is processor #1
you may also need to redirect stdout and stderr for the taskset.... & line, depending on what the command is.
Try the man page for taskset.
Yea that might be the better way to do it. I guess I could write logic to check how busy a CPU is but that would be time consuming. I didn't know it doesn't try to run everything on the same core, but that's good knowledge.
---------- Post updated at 12:11 PM ---------- Previous update was at 12:01 PM ----------
By the way how do you tell the OS to run a block of commands under the same background process such that there are multiple instances of these command blocks running in parallel?
And what would prevent your CPU busy-checking script making your CPU's busy? And how long would you expect one process to keep running on one CPU before it has to wait for something else? Should it sit idle while that process has to wait, or should another process run on it in the meantime?
What you are trying to write is an operating system, but you happen to already have one 
Unless you have a real good reason to want a process on a specific CPU, you probably don't need to do so.
There are good reasons, sometimes... Imagine you have two long-running, CPU intensive processes where the timing works out that, sometimes, they swap CPU's. A really intensive game on one processor perhaps, and Firefox in another. Switching processes from CPU to CPU wastes a bit of bandwidth sending cache bouncing around, causing both to lag. The OS, not being psychic, doesn't know they're destined to be long-running processes and may make less than ideal choices here. You, the user, know they're not quitting until you say so, so alter their CPU affinity masks to make sure they always execute on specific CPU's, reducing the lag.
I think you're confused on the definition of process -- if they're concurrent, and not threads, then by definition they're different processes -- so could you explain what you want in other terms?
The forementioned for loop, how to get each iteration of executing command1 command2 commandN in parallel in the same process (since they must be executed sequentially) such that the operating system might run them on different CPUs and cores?
Ah, so command1 command2 command3 must be executed sequentially, but the different instances can go in parallel.
for X in 1 2 3 4 5
do
( command1 ; command2 ; command3 ) &
done
wait
Again, be careful not to exceed the number of CPU's you have.