Not sure in which forum to post this. I'm trying here, in Programming.
I'm working on a PC with Intel Duo processor & 2GB of ram. OS is Ubuntu 10.04.
I'm having problems with a C++ program that makes extensive use of realloc(). It happens that as soon as the overall memory allocated(OS + user) reaches ~60% (tracked with System Monitor), realloc() fails to allocate additional memory and the program stops.
If the memory used stays below 60%, everything goes just fine.
Again, that 60% is reported by the System Monitor.
I also run Memtest86. It reports no problems with the ram.
Did someone ever experienced something similar? Any hint about where to look at?
Any feedback will be appreciated.
realloc() doesn't check physical memory. What matters is virtual memory availability.
There are several factors that might prevent realloc (or malloc for that matter) to fail including:
- 32 bit process addressing space
- Memory overcommitting configuration
- Area requested larger than free space available (obvious)
- Memory fragmentation (malloc/realloc must return a contiguous area of virtual memory).
- Bugs in your program leading to heap corruption
Inside your code are you checking and printing the error returned by realloc when it fails to allocate memory...and can you post that error message here.
How much memory does your system have?
The OP poster already stated 2GB of RAM was installed. Unfortunately, he doesn't tell how much swap is set if any, how memory overcommiting is configured, how large is the segment he tries to reallocate, to what size, and if this is a 32 or 64 bit program.
Thanks everyone. Your responses made me think more.
void * realloc ( void * ptr, size_t size );
In case that ptr is NULL, the function behaves exactly as malloc, assigning a new block of size bytes and returning a pointer to the beginning of it.
In case that the size is 0, the memory previously allocated in ptr is deallocated as if a call to free was made, and a NULL pointer is returned.
I'm saying it fails because it returns NULL although size is *not* zero.
And, again, it fails when the overall ram usage goes just above 60%.
The C++ program through realloc() is storing a few thousands text files in ram for postprocessing. Being the files size unknown, the ram for each individual file is re-allocated in increments of 500KB.
Although I cannot apriori exclude bugs in the C++ code, it works very well as long as the overall allocated memory stays below 60% of available ram (2GB).
About the system: it is 32b Ubuntu 10.04 on Intel Duo Core. Swap is ~6GB. RAM is 2GB. I'm not familiar with "memory over-committing config"...
I'm digging more into it. If you guys have any hint, I'll be glad to hear.
My best advice would be for you to switch to a 64 bit OS and application. You still don't give a clue about the allocated area sizes but I suspect that your process virtual memory is too fragmented for a contiguous segment large enough to be available.
Agreed... Though at 60% utilization you are also getting worryingly close to 32-bit address size limits. The heap can't use all 4 gigs because there's other things in the way.
Ok, more accurately but still quite roughly:
at the starting of the C++ program the OS is using ~0.3GB or ram. The program stores ~3000 text files of unknown size in ram, one by one, via realloc(). All the files should fit into less than 2GB and, anyway, when realloc() fails there are still ~0.8GB available (40% of 2GB). At that time realloc() has allocated 2-0.3-0.8 ~=0.9GB.
Each file is stored independently: realloc() makes space for one single file at each time. The space is allocated in increment of 500KB. It means that if a file is 800KB, realloc() works twice and allocates 1MB.
It happens that when the allocated memory reaches ~1.2GB (OS=0.3, program=0.9), realloc() returns NULL when attempting to allocated another 500KB.
As long as realloc() does not allocated more than those ~0.9GB, everything works just fine.
Thanks again.
You are focusing too much on RAM. realloc() doesn't care or even use RAM. What this function does is reserving (or releasing) virtual memory. Whether this virtual memory is backed by RAM or not doesn't matter.
You should monitor your process virtual memory usage and see if you don't exhaust it. You might also see how scattered your process virtual memory is with using "pmap -d pid".
And by the way, why aren't you directly allocating the file size ? or even better, why aren't you using mmap to directly load the files in memory ?
1 Like
Thanks! Will do and eventually will post again.
---------- Post updated at 04:52 PM ---------- Previous update was at 04:34 PM ----------
Because I do NOT know the size of the files. They are coming from the Internet. Instead of storing them first on HD and then loading them in memory, it's *much* faster receiving them directly in memory.
---------- Post updated at 06:25 PM ---------- Previous update was at 04:52 PM ----------
Ok, while I'm digging into something that I never faced (memory management), I'd like to take a short cut and ask you guys how I should interpret the output of vmstat and pmap.
In quiet mode, before launching the infamous C++ program here is the output of vmstat:
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 0 900908 64892 727084 0 0 7 14 154 62 4 1 95 0
Then I launch the program.
In the middle of its exec time `pmap -x PID | grep anon` gives (after grep'ing out spaces smaller than 1000):
59e17000 5904 - - - rw--- [ anon ]
5a3db000 1468 - - - rw--- [ anon ]
5a54b000 1468 - - - rw--- [ anon ]
5a6bb000 63468 - - - rw--- [ anon ]
5e6a0000 24108 - - - rw--- [ anon ]
60015000 6396 - - - rw--- [ anon ]
6083e000 189420 - - - rw--- [ anon ]
6c323000 165804 - - - rw--- [ anon ]
766f8000 39852 - - - rw--- [ anon ]
78de3000 1468 - - - rw--- [ anon ]
78f52000 16236 - - - rw--- [ anon ]
7a117000 119064 - - - rw--- [ anon ]
81652000 122016 - - - rw--- [ anon ]
88e6f000 34932 - - - rw--- [ anon ]
8b3e6000 3444 - - - rw--- [ anon ]
8b92d000 202704 - - - rw--- [ anon ]
97f9b000 94956 - - - rw--- [ anon ]
9dfb0000 69864 - - - rw--- [ anon ]
a25d4000 135300 - - - rw--- [ anon ]
aaaea000 114636 - - - rw--- [ anon ]
b1add000 1468 - - - rw--- [ anon ]
b1c4d000 1468 - - - rw--- [ anon ]
b1dbd000 91788 - - - rw--- [ anon ]
total kB 1541848
Just before realloc() fails here is again `pmap -x PID | grep anon` (after grep'ing out spaces smaller than 1000):
00adc000 1476 - - - rw--- [ anon ]
00ee4000 116112 - - - rw--- [ anon ]
080c9000 25708 - - - rw--- [ anon ]
099f1000 198768 - - - rw--- [ anon ]
15c0d000 1468 - - - rw--- [ anon ]
15d7d000 1468 - - - rw--- [ anon ]
15eed000 91020 - - - rw--- [ anon ]
1b846000 352764 - - - rw--- [ anon ]
310c5000 1956 - - - rw--- [ anon ]
312ae000 1468 - - - rw--- [ anon ]
3141e000 61992 - - - rw--- [ anon ]
35292000 5412 - - - rw--- [ anon ]
358d0000 31488 - - - rw--- [ anon ]
3797a000 49692 - - - rw--- [ anon ]
3aa78000 137268 - - - rw--- [ anon ]
4326f000 7872 - - - rw--- [ anon ]
43c09000 28536 - - - rw--- [ anon ]
459d1000 20172 - - - rw--- [ anon ]
46e79000 39360 - - - rw--- [ anon ]
495de000 34440 - - - rw--- [ anon ]
4b96a000 7872 - - - rw--- [ anon ]
4c304000 10824 - - - rw--- [ anon ]
4cf80000 208116 - - - rw--- [ anon ]
59b35000 8856 - - - rw--- [ anon ]
5a3db000 1468 - - - rw--- [ anon ]
5a54b000 1468 - - - rw--- [ anon ]
5a6bb000 63468 - - - rw--- [ anon ]
5e6a0000 24108 - - - rw--- [ anon ]
60015000 6396 - - - rw--- [ anon ]
6083e000 189420 - - - rw--- [ anon ]
6c323000 165804 - - - rw--- [ anon ]
766f8000 39852 - - - rw--- [ anon ]
78e5c000 17220 - - - rw--- [ anon ]
7a117000 119064 - - - rw--- [ anon ]
81652000 122016 - - - rw--- [ anon ]
88e6f000 34932 - - - rw--- [ anon ]
8b3e6000 3444 - - - rw--- [ anon ]
8b92d000 202704 - - - rw--- [ anon ]
97f9b000 94956 - - - rw--- [ anon ]
9dfb0000 69864 - - - rw--- [ anon ]
a25d4000 135300 - - - rw--- [ anon ]
aaaea000 114636 - - - rw--- [ anon ]
b1add000 1468 - - - rw--- [ anon ]
b1c4d000 1468 - - - rw--- [ anon ]
b1dbd000 91788 - - - rw--- [ anon ]
b7772000 65936 - - - rw--- [ anon ]
bb7d6000 3420 - - - rw--- [ anon ]
bbb2d000 43296 - - - rw--- [ anon ]
total kB 3112744 - - -
How should I interpret these info?
---------- Post updated at 06:29 PM ---------- Previous update was at 06:25 PM ----------
Mistakenly, I posted some further info as a reply to myself. Can you guys pls take a look at my last post?
Thanks!
Why would that be? All memory is swap backed technically, this is just disk-backed instead. If you save it on disk it'll be in cache, and cache is memory too. It gets paged in on-demand just like any other memory, with the advantage of not wasting swap space. It's quite efficient actually.
---------- Post updated at 12:08 AM ---------- Previous update was at 12:06 AM ----------
That you've used up 3 gigabytes of address space if not memory, way beyond the limit I'd have suspected your program would begin to bug out on. In any case the reason for the errors is clear. You're running out of address space.
mmap() might be able to fill in the corners you're not using, but there's not a lot left to be had.
Yes. Either buy some ram and move to 64 bit or fix your application design. The OS is better suited to store thousands of files than a user application.
If you feel 3+ GB of memory isn't what your application should use, maybe is there a memory leak in it. You might use valgrind to investigate that.
When you're reading from a network, writing to disk is going to be much, much faster than your network speeds unless your disks are stone tablets with data being carved into them by gnomes.
And what do you plan on doing WHEN you run into a bug and your program crashes in production with all these files in RAM? Did you even think of that?
The realloc() function changes the size of the memory object pointed to by ptr to the size specified by size. The contents of the object will remain unchanged up to the lesser of the new and old sizes. If the new size of the memory object would require movement of the object, the space for the previous instantiation of the object is freed. If the new size is larger, the contents of the newly allocated portion of the object are unspecified. If size is 0 and ptr is not a null pointer, the object pointed to is freed. If the space cannot be allocated, the object remains unchanged.