Hi. We have a HACMP two node (Active passive cluster).
Recently the primary node went to shutdown state and all the resource groups
moved to the secondary node.
People say that this has happened because of the /var filesystem was 100% utilized in the primary node. Is this true?
Will it switch over to the secondary node, if the /var filesystem was 100% utilized in the primary node ?
You might want to consult your /tmp/hacmp.out for the reasons, why your RG moved.
There is no such file found (/tmp/hacmp.out) in both the nodes
What version of HACMP are you using? Maybe check /var/hacmp/log for the hacmp.out. If you got the version you might check the corresponding documentation to find your hacmp.out like for example
IBM InfoCenter - Understanding the hacmp.out log file
Default for your cluster logs is /var/hacmp/.....
Short answer: Yes.
Long answer: sth. in the OS failed and therefore the cluster followed. It is the AIX server that cannot live with /var getting 100% full. Ironically this failure might have been caused by the HA software logging to many information in the /var filesystem. Works as designed...
If the server malfunctions the cluster is expected to failover. Nothing wrong here.
If you did not guess by now how to work around this classic error here is my tip for you: an HACMP admin never installs HACMP/PowerHA on an AIX server without creating a separate filesystem (e.g. /var/hacmp/) for the cluster logs first.