Just a question about subprocesses.. Lately one of our servers has started to throw out the following error:
SYSTEM ERROR: Too many subprocesses, cannot fork. Errno=12
We've already increased the threshold twice. Its now up to 8000 and the swap space has also been increased. We had rebooted the server to clear out any old processes that may have been running but we have since hit the same error.
Whats the best way to find whats creating so many subprocesses? I'm not 100% sure what the error actually means..
Things like this do not usually just happen. Has the server just been updated or has a new application / script been installed. The "cannot fork2" error is usually associated with the machine running low on memory or CPU resource. It can also be associated with leaky applications which eat up one or other or both of these resources. When the error occurs can anything be run on the server or is just in a "hung" state?
If the server is accessible when the error occurs run the ps -ef | more command and check the process I/Ds that are using an over amount of CPU or memory.
The server is used as a Model region. Code is tested in the region for Release purposes.. I suspect some new code has gone in and is causing the issue..
When the fork error occurs it usually freezes up the machine.. I'm not sure if I can run the ps -ef command at that specific moment when the error occurs..
Like i said before we've increase the subprocess threshold but I'm just worried we're delaying the inevitable..
As I surmised a new script or something like this has been installed and most likely is running an endless loop or something similar which is just eating up CPU. Try running the ps -ef | more command or Topas and check for excessive CPU usage.
Which output does lsattr -El sys0 | grep maxuprocs show. Maybe you have to go here to a higher value? Oracle servers for example need here values between 2048 and 20480.