1. Deploy automatic virtual IP failover using Oracle Private Cloud Appliance
  2. Deploy Automatic Virtual IP Failover

Deploy Automatic Virtual IP Failover

Corosync and Pacemaker must be installed on clusters before integrating with Private Cloud Appliance and testing failover of the secondary IP.

Install on Clusters

  1. Install Corosync and Pacemaker packages.
    yum install corosync pacemaker pcs
yum install python36-oci-cli
    Setup proxy as per the lab requirements to reach yum repos.
  2. Take the backup of heartback configuration of the clusters.
    sudo cp /usr/lib/ocf/resource.d/heartbeat/IPaddr2 /usr/lib/ocf/resource.d/heartbeat/IPaddr2.bck
  3. Run firewalld, defined per customer security needs for cluster.
  4. Update the /etc/hosts file to add the node's IP and host name information for all cluster nodes to setup a local DNS system.
    Example: 
    <Instance-Node1 Private IP> node1-name
<Instance-Node2 Private IP> node2-name
  5. Create a user and set a password.
    sudo passwd hacluster
  6. Start and enable the cluster services.
    sudo systemctl start pcsd
sudo systemctl enable pacemaker
sudo systemctl enable corosync
sudo systemctl enable pcsd
  7. Verify cluster authentication.
    sudo pcs cluster auth <node1-name> <node2-name>
<node1-name> : Cluster Node1/Hostname
<node2-name> : Cluster Node1/Hostname
    Example:
    sudo pcs cluster auth a-node1 a-node2
Username: hacluster
Password: 
a-node1: Authorized
a-node2: Authorized
  8. Cluster setup.
    sudo pcs cluster setup --name <clustername> <node1-name> <node2-name>
<clustername> : Cluster Name
<node1-name> : Cluster Node1/Hostname
<node2-name> : Cluster Node1/Hostname
    Example:
    sudo pcs cluster auth a-node1 a-node2
Username: hacluster
Password: 
a-node1: Authorized
a-node2: Authorized
[root@a-node1 opc] # sudo pcs cluster setup HACluster a-node1 a-node2
Error: A cluster name (--name <name>) is required to setup a cluster
[root@a-node1 opc] # sudo pcs cluster setup --name HACluster a-node1 a-node2
Destroying cluster on nodes: a-node1, a-node2...
a-node1: Stopping Cluster (pacemaker)...
a-node2: Stopping Cluster (pacemaker)...
a-node2: Successfully destroyed cluster
a-node1: Successfully destroyed cluster
Sending 'pacemaker_remote authkey' to 'a-node1', 'a-node2'
a-node1: successful distribution of the file 'pacemaker_remote authkey'
a-node2: successful distribution of the file 'pacemaker_remote authkey'
Sending cluster config files to the nodes...
a-node1: Succeeded
a-node2: Succeeded
Synchronizing pcsd certificates on nodes a-node1, a-node2...
a-node1: Success
a-node2: Success
Restarting pcsd on the nodes in order to reload the certificates...
a-node1: Success
a-node2: Success
  9. Start cluster for all cluster nodes from any of the cluster nodes.
    sudo pcs cluster start --name clustername -–all
    Example:
    sudo pcs cluster start --name HACluster --all
a-node1: Starting Cluster (corosync)...
a-node2: Starting Cluster (corosync)...
a-node2: Starting Cluster (pacemaker)...
a-node1: Starting Cluster (pacemaker)...
  10. Set pacemaker property.
    sudo pcs property set stonith-enabled=false
sudo pcs property set no-quorum-policy=ignore
  11. Verify the running cluster status.
    sudo pcs cluster status
    Example:
    sudo pcs cluster status
Cluster Status:
Stack: corosync
Current DC: a-node2 (version 1.1.23-1.0.1.el7_9.1-9acf116022) - partition with quorum
Last updated: Fri Aug 19 03:07:25 2022
Last change: Fri Aug 19 03:06:13 2022 by root via cibadmin on a-node1
2 nodes configured
0 resource instances configured
PCSD Status:
a-node1: Online
a-node2: Online
  12. Set the OCI config as per your Private Cloud Appliance settings on all cluster nodes. Setting up the config profile is very important to connect to Private Cloud Appliance.
    Example: i.e. /root/.oci/config
    [DEFAULT]
user=<User-ocid1>
fingerprint=<fingerprint>
key_file=<Key-Location>
tenancy=<Tenancy ocid1>
region=<PCA FQDN>

Define the Heartbeat Settings on Clusters and Integrate with Private Cloud Appliance X9-2 Instances for VIP Failover

  1. For Private Cloud Appliance run:
    sudo sed -i '633i\export OCI_CLI_CERT_BUNDLE=/root/.oci/ca-chain.cert.pem\' /usr/lib/ocf/resource.d/heartbeat/IPaddr2
    You can instead pass the Private Cloud Appliance certificate with the assign-private-ip command. For example: --cert-bundle <Certification Location for Private Cloud Appliance> option.
    Here's an example for Oracle Linux 7.9. For Oracle Linux 8 and Centos operating systems, look for the add_interface () function line number in IPaddr2, and change the line number accordingly when updating Linux HA IPaddr2 resource entries. 
    sudo sed -i '628i\server="`hostname -s`"\' /usr/lib/ocf/resource.d/heartbeat/IPaddr2
sudo sed -i '629i\node1vnic="ocid1.vnic.pca.NODE1-vNIC-OCID"\' 
/usr/lib/ocf/resource.d/heartbeat/IPaddr2
sudo sed -i '630i\node2vnic="ocid1.vnic.pca.NODE2-vNIC-OCID"\' 
/usr/lib/ocf/resource.d/heartbeat/IPaddr2
sudo sed -i '631i\vnicip="10.212.15.13"\' /usr/lib/ocf/resource.d/heartbeat/IPaddr2
sudo sed -i '632i\export LC_ALL=C.UTF-8\' /usr/lib/ocf/resource.d/heartbeat/IPaddr2
sudo sed -i '633i\export LANG=C.UTF-8\' /usr/lib/ocf/resource.d/heartbeat/IPaddr2
sudo sed -i '633i\export OCI_CLI_CERT_BUNDLE=/root/.oci/ca-chain.cert.pem\' 
/usr/lib/ocf/resource.d/heartbeat/IPaddr2
sudo sed -i '634i\touch /tmp/error.log\' /usr/lib/ocf/resource.d/heartbeat/IPaddr2
sudo sed -i '635i\##### OCI/IPaddr Integration\' /usr/lib/ocf/resource.d/heartbeat/IPaddr2
sudo sed -i '636i\ if [ $server = "node1" ]; then\' 
/usr/lib/ocf/resource.d/heartbeat/IPaddr2
sudo sed -i '637i\ oci network vnic assign-private-ip --unassign-if-already-assigned --vnic-id $node1vnic --ip-address $vnicip >/tmp/error.log 2>&1\' 
/usr/lib/ocf/resource.d/heartbeat/IPaddr2
sudo sed -i '638i\ else \' /usr/lib/ocf/resource.d/heartbeat/IPaddr2
sudo sed -i '639i\ oci network vnic assign-private-ip --unassign-if-already-assigned --vnic-id $node2vnic --ip-address $vnicip >/tmp/error.log 2>&1\' 
/usr/lib/ocf/resource.d/heartbeat/IPaddr2
sudo sed -i '640i\ fi \' /usr/lib/ocf/resource.d/heartbeat/IPaddr2
  2. Implement modifications for code in IPaddr2.
    • Replace ocid1.vnic.pca.NODE1-vNIC-OCID and ocid1.vnic.pca.NODE2-vNIC-OCID with your own OCI VNIC's (virtual network interface card) OCIDs.
    • Replace node1 and node2 hostname entries with your own clusternodes hostnames.
    • In OCI_CLI_CERT_BUNDLE, define the CERT bundle location for Private Cloud Appliance.
    • For your VNIC IP, define the VNIC IP as per your configuration, subnet, VCN, and make sure this is a unique IP and not allocated to any other VNIC.

Setup the Cluster Resource

To setup the cluster resource, run:

pcs resource create <Cluster-Resource-Name> ocf:heartbeat:IPaddr2 ip=10.212.15.13 cidr_netmask=24 op monitor interval=20

Note:

  • The cidr_netmask=24 in the Pacemaker command is dependent on the subnet size being /24.
  • The ip=10.212.15.13 is the secondary private IP.

Example:

pcs status
Cluster name: HACluster
Stack: corosync
Current DC: a-node2 (version 1.1.23-1.0.1.el7_9.1-9acf116022) - partition with quorum
Last updated: Fri Aug 19 03:34:51 2022
Last change: Fri Aug 19 03:34:44 2022 by root via cibadmin on a-node1
2 nodes configured
1 resource instance configured
Online: [ a-node1 a-node2 ]
Full list of resources:
HAFailover (ocf::heartbeat:IPaddr2): Started a-node1
Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled

Test Failover of the Secondary IP

To test failover of the secondary IP run:

sudo pcs resource move <cluster-name> <node2-name>
<clustername> : Cluster Name
<node2-name> : Cluster Node1/Hostname

For example:

sudo pcs resource move HAFailover a-node2

Verify Successful Failover

Resources should be started on node2. To verify successful failover, run:

# pcs status