Hia all,
I am not sure how many people actually follow the HPC forum on unix.com, but you may be interested in discussing the following (academic) problem:
Assume you want to run a *very* large number (say 100.000) of very lightweight synchronous operations. As an example, assume that you want to run 100.000 instances of
sleep (3600); // thats one hour sleep
The trivial (aka braindead) approach would be
for ( int i = 0; i < 100000; i++ )
{
::sleep (3600);
}
Takes about 15 years to finish
One could start 1000 threads, and run a sleep in each of them. That reduces the runtime to 100 hours - still 4 days, and the system is totally idle all the time.
So, using more threads? Won't work, as the max-threads-per-process limit will be hit at some point.
So, spawn 100 processes which spawn 1000 threads each?
The max-threads-per-process limit is, on Linux, close to the max-threads-per-system limit, so that won't work. On other Unixes that is different, but I don't think you get 100.000 threads on a normal single CPU system. Do you?
So, what would your approach be?
I am not looking for a sleep replacement: so saying that I should set alarm or something similar is of not much use. Sleep is obviously only an example here - replace it with an extremely lightweight job, like running a very time consuming synchronous remote operation.
I am looking forward to the ideas you guys can come up with!
Cheers, Andre.