xargs -P

UNIX for Advanced & Expert Users
1

I discovered that GNU's xargs has a -P option to allow its processes to run in parallel. Great! Is this a GNU thing, or is it supported by other platforms as well?

2

xargs -P -- It definitely is not part of POSIX, and it isn't on HP UX, so it must be a GNU thing, like the time manipulations GNU date is capable of doing.

3

Thanks Jim. Anyone know about Solaris?

4

AFAIK it is GNU specific. It's not on Solaris 10 either:

SYNOPSIS
     /usr/bin/xargs [-t]  [-p]  [  -e  [eofstr]]  [-E eofstr]  [-
     I replstr]  [  -i  [replstr]]  [-L number]  [ -l [number]] [
     -n number [-x]] [-s size] [ utility [ argument...]]

     /usr/xpg6/bin/xargs [-t] [-p] [ -e [eofstr]] [-E eofstr]  [-
     I replstr]  [  -i  [replstr]]  [-L number]  [ -l [number]] [
     -n number [-x]] [-s size] [ utility [ argument...]]
5

Hi.

I was very surprised to learn about the additional capabilities of GNU/Linux xargs. It can be used as a throttle -- allow no more than n processes to be running at a time (from the processes it creates). More than that, xargs monitors the processes and if one of the n processes stops, xargs will pluck another of its arguments and fire off an additional process. Very nice, but not very well documented at all.

To see how this might work (if at all) on a non-GNU/Linux machine. I obtained the source from findutils - GNU Project - Free Software Foundation (FSF) onto an HP box, and compiled it. It took a long time, but it finished -- it compiles a few other things, e.g. the GNU version of find. The tests included in the source failed to run -- an item named runtest could not be located -- but I ran a small test of my own, essentially:

xargs -n 2 -P 4 $burn

where burn is a script that prints start, sleeps, prints stop. Omitting the script and some output, but showing, for the number of processes being 4, that whenever a process ends, another one is started, but never more than 4:

$ ./s1 data3

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: HP-UX, B.11.00, 9000/712
Distribution        : GenericSysName [HP Release B.11.00] (see /etc/issue)
GNU bash 2.05b.0

 Results:
 job  4 start  8 seconds at: 06:10:33
 job  3 start  6 seconds at: 06:10:33
 job  2 start  4 seconds at: 06:10:33
 job  1 start 50 seconds at: 06:10:33
 job  2 stop   4 seconds at: 06:10:37
 job  5 start 10 seconds at: 06:10:37
 job  3 stop   6 seconds at: 06:10:39
 job  6 start 12 seconds at: 06:10:39
 job  4 stop   8 seconds at: 06:10:41
 job  7 start 14 seconds at: 06:10:41
 job  5 stop  10 seconds at: 06:10:47
 job  8 start 16 seconds at: 06:10:48
 job  6 stop  12 seconds at: 06:10:51
 job  9 start 18 seconds at: 06:10:52
 job  7 stop  14 seconds at: 06:10:55
 job 10 start 20 seconds at: 06:10:56
 job  8 stop  16 seconds at: 06:11:04
 job  9 stop  18 seconds at: 06:11:10
 job 10 stop  20 seconds at: 06:11:16
 job  1 stop  50 seconds at: 06:11:23

So, for large-granularity parallel processing, this could be useful.

Of course, many people here could write a shell script to do much the same, but it is nice to have it done for you ... cheers, drl