realloc fails in C : what next ?

Applications Programming
1

Hi all

I'm trying to use someone else's software, which has a realloc that fails in it. This is probably due to memory limitations, as it only happens when I use this software on huge datasets.

First question : how to diagnose if it's a soft or hard limitation? I mean, if it's due to my computer's memory limits, or to some limits the compiler / my C implementation imposes?
I'm running Mandriva Linux, 64 bit, with 4Gb RAM and 6GB swap, I don't think I'm reaching those limits.
The same failure happens on our computer cluster, which has about 60Gb of RAM ...

I used the makefile that came with the software, which adds these options to the gcc command :
CFLAGS = -O4 -funroll-loops -fomit-frame-pointer -finline-functions

More questions might follow depending on the answer, but the obvious question is : how to get around this limit ?

2

Well, you can check that the system isn't artificially imposing a limit on you by running `ulimit -m' which will tell you the maximum number of kB of memory the OS will allocate to you. If that reports anything other than "unlimited" you can increase the limit with `ulimit -m unlimited', but you need to be root to do this.

Other than that, you could always get realloc() to print the size of the memory it failed to move, to make sure you're not trying to move 2 GB of data or something ridiculous.

3 
[jos@faba ~]$ ulimit -m
unlimited

But you're right, the program is trying to do something ridiculous :

   k = com.npatt*com.ncatG*sizeof(double);
   if((com.fhK=(double*)realloc(com.fhK,k)) == NULL) 
    {
    printf("%d\n",k);
    error2("oom fhK");
}

Upon execution, the reported value for k is -169522176.

Thanks a lot for your suggestion, I hadn't thought about that !

jos

4

As an aside, whoever wrote the makefile was trying to be really aggressive with the optimizations. Perhaps they were using bad algorithms to start with, and thought to speed things up by cranking up the compiler optimizations. At any rate, these optimizing switches sometimes do, but sometimes don't, speed up execution time. Only a careful analysis (profiling) can provide the answer. At any rate, GCC offers up to -O3. -O4 is not an option, to my knowledge.

5

Thanks for pointing that out, kermit !
I'll keep that in mind for the next time I analyze large datasets.

jos

6

The omit-frame-pointer and inline-functions are enabled by default if using -O3 optimization - so this is all that is required

CFLAGS = -O3 -funroll-loops
7

...and funroll-loops isn't often helpful.

If you're still debugging you should have -O0 and -ggdb. No optimizations and debugging support.

8

Is the binary you're running a 64- or 32-bit binary?

If it's 64-bit, the use of the "%d" field in that printf() statement leads me to believe the original developer had no clear understanding of the implications of using a 64-bit memory model. That printf() field should be "%ld" for a 64-bit size_t value.

The variable "k" is defined as a "size_t", isn't it?

9

Assuming the version of GCC is recent enough (And I assume that it is, as "Mandriva" only started shipping in late 2005, I believe), %zu (supported under C99) is also available.

10

the "printf("%d\n",k)" part was my addition ... and indeed, I have no idea of how 32 or 64 bit architectures change that.

But anyway, k was declared as int, so the author of the program should have done a cast to size_t, I guess ?

11

sizeof() returns a size_t type. Instead of casting k, you would do

size_t k;