Hi there...
I am maitaining an archaic application which is using mmap for file transfering/routing. There are over 500 instances of the application running without any issues for almost 2 decade. Now, the problem is that the on one particular server (HP-UX 11), sometimes the mmap is failing when processing big files (size of just 8 MB). The error message I am getting is ENOMEM which corresponds to the fact that there is no sufficient memory for the mmap. But the application support folks says there is enough memory in the server and they tuned kernel maxdsiz, maxssiz but still the problem happens occassionally. Could you please let me know what other kernel values or any other things need to tuned to avoid this situation. Also, any other possible solutions are welcome.
Thanks,
Tobsinte
A place to start would be the HP-UX mmap man page It has a very different description of the meaning of ENOMEM than the one you mention. In fact it says the meaning depends on the setting of MAP_FIXED and you didn't mention whether or not that is set.
Also you should check for OS patches that might affect mmap.
Thanks a lot for the information.
I've checked mmap man pages for more information. MAP_FIXED is not set from the program instead the "flags" argument carries a value MAP_SHARED. But I still couldn't figure out what exactly causes the mmap to return ENOMEM. What else can I possibly check for?
Also since I am not from the IT operations team (who maintains the HP-UX servers) I dont have access to check for the patches since they are the one's who install patches etc. But is there any specific things (like some specific patches info) I can check with operation team to check for me. (else they will keep asking me what to check!)
If you don't have access to the server, it is not obvious how you might with the problem. 
You (or someone) will need a service contract to obtain patches, so I would advise using that contract to open a call with HP and ask them to suggest patches for your version of HP-UX that might help with a mmap problem.
Anyway, the man page says " [ENOMEM] ... Or, MAP_FIXED was not specified and there is insufficient room in the address space to effect the mapping." So if it's not an OS problem, it is probably a memory leak. Maybe you could add some code to induce a core dump when mmap returns ENOMEM, then examine the core dump to see where the virtual addresses went. (But if you can't access the server, I do not know you might examine a core dump.)
thanks a lot for your suggestion.
I have requested for the list of patches applied to this particular server and later would compare it with the servers which dont have any issue (any mmap related patches were missed or not). I am 100% sure there is not any memory leak as these product is running fine on over 500+ servers around the world for almost 2 decade. There is something wrong on this particular server.