I don't know AIX and I don't have access to an AIX box. But maybe I can help get you pointed in the right direction. You are clearly having a problem with virtual memory, not physical memory. Stuff like how much unused ram you have is not relevant.
Most likely you are bumping into the max size of a data segment. You might try "/usr/bin/ulimit -a" to see if that command exists and if it displays a limit called "data" or something like that. I don't know how to increase this on AIX.
Yeah, malloc() tries to grow the data segment, and yes, you are currently limited to 131K.
And your stack is limited 32K bytes. There is also a text segment. However the sum of these, data+stack+text must be less than 32K if the "memory" line means what I think it means.
AIX must work differently than what I'm used to. Or maybe not...do a "man getrlimit". The stack stuff will be under RLIMIT_DATA and memory might be RLIMIT_AS.
Also try /usr/bin/ulimit -Ha to see if they are your hard limits too.
Actually, it can be. It's simply dumb, but not impossible. The kernel will check both limits as you attempt to increase the data segment size. You're guaranteed to bump into the memory limit first is all.
And look at your hard limits....they are unlimited. Your program can call getrlimit() to obtain the hard and soft limits. And it can call setrlimit() to raise the soft limit up to the hard limit.
As I said before, I don't know AIX and I could be wrong about some or all of this. But it would only take a few seconds to toss a setrlimit() into the program to see if I'm right.
But you're right about the memory leak, the best policy is to find it and fix it.