PowerHA 7.1.2 two node cluster problem

Hi,

We have PowerHA 7.1.2 two-node cluster with dh NodeA and NodeB. We had some HW problems and lost the secondary node NodeB. The cluster configuration have two resource groups one for the application (Informix) and second for disk heartbeat. So in an attempt to recover the NodeB, we took a sysback from NodeA and restore it, after we overcome our HW issue. We change the hostnames, IP addresses, rhost/hosts files, mount the existing disks and volume groups, but we cant sync and verify the cluster communication. Usually we get comm error during sync and verify.

migcheck[490]: read() error, nodename=NodeA, rc=0
migcheck[490]: read() error, nodename=NodeB, rc=0

WARNING: A communication error was encountered trying to get the VRMF from remote nodes. Please make sure clcomd is running

Verifying Cluster Configuration Prior to Starting Cluster Services.

There are no active cluster nodes to verify against.
Verifying node(s): NodeA requested to start

ERROR: Comm error found on node: NodeA.
ERROR: Comm error found on node: NodeB.
ERROR: No more alive nodes left

Cluster services and RG are ONLINE on NodeA. During verbose sync and verify we notice that NodeB cant resolve NodeA.

Verifying that all IP labels resolve to correct IP address on all nodes.
Node: NodeB

IDS ()	PASS

NodeA ()	FAIL

NodeAHB ()	FAIL

NodeB ()	PASS

NodeBHB ()	PASS

I suspect there is something left from the sysback from the NodeA, maybe in the ODM database, but I can't find out for sure.

Every comment, help, and solution is welcome. I really need to get this cluster up and running.

BR,
Ivan Efremovski

Firstly, I'm not an AIX or PowerHA expert so I'm answering in generic terms with knowledge of cluster technology and how it all works.

When you configure a two node cluster, NodeA knows that it is NodeA, and NodeB knows that it is NodeB. If a node fails then the cluster software suite (whatever one it is) detects the problem and fails-over all services to the surviving node. When the problem with the failed node is resolved, the cluster suite will recover to normal service by copying/mirroring data from the surviving node to the recovered node, etc, etc.

IMHO, you should never restore one node's backup to the other node because if you restore NodeA's backup onto NodeB you end up with two NodeA's. The cluster configuration is lost. Anyway, the whole point of using a cluster suite is you shouldn't need to do anything like that. Recovery should be automatic once you've pressed the button to tell the software to normalize.

Let's see what other forum members on here say, especially the AIX/PowerHA experts.

Also you did not say what was your HW issue, and what you done, especially if you had to change some HW
For instance changing a disk of rootvg can end with not having anymore disk heartbeat, new network card vould be unknown to HACMP config etc...
And as said previously you should never restore a node with an image of the other unless they are absolutely identical AND you know what/where to modify what in the HA config files
Normally you would have restored from nodeB's mksysb, and after restore from your backup to the last version, then HA has scripts to launch to verify its integrity, after you went through the config files and put all up to date...
And you execute the script till you have no errors, and only then can you try to communicate with the other node otherwise you risk to corrupt your HA, and its really laborious to get it back in a coherent state

Hi guys,

So the HW problem the customer had is with the disk array, in other words, all data from the machine was lost and we didn't have a backup to restore from. So I needed to build the whole machine from scratch, VIOS, LPARs etc. In the case with HACMP CAA with repo disks I managed to successfully restore the lost nodes with sysback from the alive node, but no success with PoweHa 7.1.2 nodes. This system is very old and still in production, with many custom application configurations (Informix) and no documentation. So in order to avoid installations from scratch, I wanted to try with sysback from the alive node.

That said, is there any way to:

1. Manage to work with sysback from alive node and modify the HA config files? The nodes are identical.
2. Reinstall the lost node and restore the cluster configuration snapshot in order to bring it back?
3. Install and configure a new node in the cluster from scratch.

Every comment, help, and solution is welcome. I really need to get this cluster up and running.

BR,
Ivan Efremovski

I tried to contact our AIX guru on this forum, she will need quite more information to help you because so far the situation is not all that clear:
Were the 2 nodes LPAR on the same physical machine?
More important: what were the disks you had, what went wrong with the disk array and how was it solved? what kind of connection? IF you were on a bay (e.g. EMC or HDS ) can the storage guys recreate to the identical the hmm I forgot the term, but its the equivalent of the Net mac address ...

What is unclear also is what is now running on nodeA, if the appication resource group is still running, what have you lost on nodeB? the OS with the HA software?

as you essentially built the primary node twice, I am guessing that your ODM has still cluster entries somewhere pointing to the wrong node.
The easiest way to fix this is probably to remove the second node (so that you only have one node left in the cluster. Then remove and reinstall the ha software to have a fresh start. Make sure that any reserve locks are removed from all disks on that second node. Then make sure that /etc/hosts, /usr/es/sbin/cluster/etc/rhosts and /etc/cluster/rhosts know all the IPs within the cluster and are identical. Make sure as well that all IPs are having the correct and identical subnet masks. Finally add the node back to the cluster and sync the cluster.
BTW - having heartbeat disk as a resource group is not only wrong but completely unneccessary. As long as the disk is configured correctly, thats all you need. And make sure that you have the absolute identical HA version and identical ha fixes and OS version on both nodes.

Thank you so much for your help zxmaus, big hugs

Thank you for your answer.

First to explain the architecture. We have two separate Power p740 machines, PowerVM installed and configured. NodeA and NodeB LPAR's were located on separate physical machines. rootvg is installed on the local vio, presented to the LPAR's via vscsi. With the HW problem on the second physical machine, we lost all rootvg and vio data and we had to rebuild it from scratch. The resource and hb disks, are presented from the SAN (EMC Unity 400). Additionally, we didn't have backups from the lost LPARs, for example, NodeB.

There are no duplicate UUIDs, but the HACMPcluster ODM entry on the restored NodeB, shows NodeA listed as nodename. Cluster is stable on the NodeA and all resource groups are online.

I have concerns about removing the NodeB from the cluster because I think it will remove something from the active NodeA and destroy my cluster. Maybe I should isolate NodeB from the network and start from there.

All HA and OS software versions are identical.

Please advice.

BR,
Ivan

if you remove the second node from your cluster, it should not do anything to your primary node. The ODM on you primary node should not have any wrong entries, so its very likely safe to do - however I do understand your concerns so doing it under downtime would be my choice. As long as your primary node does not talk to your secondary node, you essentially do not have a cluster right now.
I am not sure how good your HA skills are, but if you are afraid anything happens to your data, you can always remove your storage and any application start/stop scripts you might be using first to make it extra safe. In worst case you could remove the cluster completely and start over from scratch.

So I will try to create AP, based on my research and your insight/recommendations. @zxmaus

1. Schedule maintenance window.
2. Shutdown the NodeB (sysback image from NodeA).
3. Bring the resources Offline and take a new sysback image node NodeA.
4. Take snapshot cluster configuration backup.
5. Clone or take snapshots from the SAN disks.
6. Remove the NodeB node, from the smit hacmp NodeA. (not sure this step will work, or I need to force the removal)
7. Power on the NodeB and uninstall all HA software from it.
8. Install the HA software and add the node NodeB back to the cluster. Same configuration from before.
9. Hope for the best

Every comment, help, and solution is welcome. I really need to get this cluster up and running.

BR,
Ivan

run on good node:
/usr/es/sbin/cluster/utilities/clRGinfo -v

and look at the node names.

/usr/es/sbin/cluster/utilities/clshowres
especially look for Service IP Label, Filesystems, Volume Groups

on "bad" node:
look if the nodename on second machine is correct:
/usr/es/sbin/cluster/utilities/get_local_nodename

if you get back the same name as on good system, well you produced an exact and unusable copy on the second machine. TRY to set the local node name (smitty hacmp....) WITHOUT starting cluster-services.

if not:
look if main cluster-processes are running:
lssrc -g clcomdES
lssrc -g cluster

if not, use smitty hacmp to get the cluster processes started.

Hi,

First to answer @JMn and @zxmaus questions:

Good node IDSa, bad node IDSst (image from IDSa).

IDSst:

root@IDSst:/# /usr/es/sbin/cluster/utilities/get_local_nodename
IDSst
root@IDSst:/# lssrc -g clcomdES
Subsystem Group PID Status
clcomdES clcomdES 9240638 active
root@IDSst:/# lssrc -g cluster
Subsystem Group PID Status
clstrmgrES cluster 9044032 active

root@IDSst:/# lsattr -E -l inet0 -a hostname
hostname IDSst Host Name True
root@IDSst:/# odmget -q 'attribute = hostname' CuAt

CuAt:
name = "inet0"
attribute = "hostname"
value = "IDSst"
type = "R"
generic = "DU"
rep = "s"
nls_index = 24

So, found that the HCMPcluster odm on the IDSst still has the wrong name:

root@IDSst:/# odmget HACMPcluster

HACMPcluster:
id = 1574567870
name = "IDS_cluster"
nodename = "IDSa"
sec_level = "Standard"
sec_level_msg = ""
sec_encryption = ""
sec_persistent = ""
last_node_ids = ""
highest_node_id = 0
last_network_ids = "0"
highest_network_id = 0
last_site_ids = ""
highest_site_id = 0
handle = 1
cluster_version = 11
reserved1 = 0
reserved2 = 0
wlm_subdir = ""
settling_time = 0
rg_distribution_policy = "node"
noautoverification = 0
clvernodename = ""
clverhour = 0
clverstartupoptions = 0

I tried and changed the hostname to IDSst, with the command: printf 'HACMPcluster:\n\tnodename=IDSst' | odmchange -o HACMPcluster and restarted the node. After this change, verify and sync on node IDSa went without error, but the IDSst node showed the resources groups as OFFLINE on both nodes:

root@IDSst:/# /usr/es/sbin/cluster/utilities/clRGinfo

Group Name State Node

IDS_rg
OFFLINE IDSa
OFFLINE IDSst

rg_diskhbmulti
OFFLINE IDSa
OFFLINE IDSst

There is still something in the IDSst configuration, that prevents starting and joining the cluster. At this point, I think that removing the node, or maybe adding a new node is the better approach here. Don't want to break something and can't go back.

Every comment, help, and solution is welcome. I really need to get this cluster up and running.

BR,
Ivan

Basically, and it has been suggested in a long list above - I am only trying to storyline the process.

A. Tell HA what happened, ie node b was lost forever. So, on nodea, remove node b, and sync the cluster.

B. Build a new node B. Cloning from sysback is probably fine.

As it is a clone, it might think it is part of a cluster. Use the HA command to remove cluster. Iirc, no sync is necessary ( or even possible).

So, now we have a new node b, ready to be added to a cluster, or almost. Cl_verify will be the loop you will be in until all the shared resources are setup properly. That is, it might not work immediately.

C. On node a, tell HA, i want to add a node named b. Provide the necessary info and sync the cluster. This will verify the node is ready for adding as a node.

This is called a topology change. Next is the resource change.

D. We now have a two node cluster, but node b has no resources. Add node b to any rg you want, and sync. Again, clverify will run and verify it’s readiness.

At this point you should have a functional two node cluster.

Hope this helps!!

First of all, thank you for your answers and inputs, they definitely point me in the right direction. Two weeks ago i managed to bring the cluster up and running, doing the procedure i wrote before:

1. Schedule maintenance window.
2. Shutdown the NodeB (sysback image from NodeA).
3. Bring the resources Offline and take a new sysback image node NodeA.
4. Take snapshot cluster configuration backup.
5. Clone or take snapshots from the SAN disks.
6. Remove the NodeB node, from the smit hacmp NodeA. (not sure this step will work, or I need to force the removal)
7. Power on the NodeB and uninstall all HA software from it.
8. Install the HA software and add the node NodeB back to the cluster. Same configuration from before.

Everything went ok and the cluster is now healthy again.

Special thx to @zxmaus @MichaelFelt @JMn @vbe @hicksd8.

BR,
Ivan