Parallel processing in AIX (bash shell scripting)

Shell Programming and Scripting
1

Hi all,

I am struggling to build a utility which can do the parallel processing. I achieved same in Linux using xargs -P but same is not working on AIX.

I am building file copy utility where I will have all required info in a file (like souce file info and target location details), now i need to copy files which should run in parallel like 50 files copy in parallel.

your guidance is very much appreciated.

Thanks

2

Did you consider running the copy commands in a loop, sending some (not all!) into background?

Don't expect too much from 50 parallel copy actions performance wise - disk operation and memory might be the constraints.

3

There's usually no point trying to copy files in parallel. CPU is almost always faster than disk. Parallelizing can easily slow it down due to disk thrashing, and poison your filesystem with large amounts of fragmentation. I only know two situations where it's a win:

  • File operations with lots of CPU overhead, like extracting many large bzip files.
  • Software disk mirrors in Linux, where processes get arbitrarily assigned one disk each. (Only useful for reading.)

So could you explain your situation in more detail, please, so we can find out if this is one of those rare situations you'll come out ahead?

2 Likes