6.17 Expanding an Oracle RAC Cluster in Oracle Linux KVM Using OEDACLI

You can expand an existing Oracle RAC cluster on Oracle Linux KVM by adding guests using the Oracle Exadata Deployment Assistant command-line interface (OEDACLI).

OEDACLI is the preferred method if you have a known, good version of the OEDA XML file for your cluster.

Note:

During the execution of this procedure, the existing Oracle RAC cluster nodes along with their database instances incur zero downtime.

Note:

During deployment, the cloned guest inherits various configuration attributes from the source guest, including the client network configuration and the backup network configurations (if present).

If all of the KVM hosts have the same network configuration, then the inherited attributes work as expected.

However, if the new KVM host uses a different physical network configuration, deployment of the cloned guest will fail. This situation is most likely when an Exadata system contains different versions of compute node hardware. For example, when adding an X10M server to an X8M-2 rack.

In this case, you must manually adjust the relevant network definition by using the ALTER NETWORK command before deployment. Contact Oracle Support for details.

Use cases for this procedure include:

• You have an existing Oracle RAC cluster that uses only a subset of the database servers of an Oracle Exadata Rack, and now the nodes not being used by the cluster have become candidates for use.
• You have an existing Oracle RAC cluster on Oracle Exadata that was recently extended with additional database servers.
• You have an existing Oracle RAC cluster that had a complete node failure and the node was removed and replaced with a newly re-imaged node.

Before preforming the steps in this section, the new database servers should have been set up as detailed in Adding a New Database Server to the Cluster, including the following:

• The new database server is installed and configured on the network with a KVM host.
• Download the latest Oracle Exadata Deployment Assistant (OEDA); ensure the version you download is the July 2019 release, or later.
• You have an OEDA configuration XML file that accurately reflects the existing cluster configuration. You can validate the XML file by generating an installation template from it and comparing it to the current configuration. See the OEDACLI command SAVE FILES.
• Review the OEDA Installation Template report for the current system configuration to obtain node names and IP addresses for existing nodes. You will need to have new host names and IP addresses for the new nodes being added. The new host names and IP addresses required are:
  • Administration host names and IP addresses (referred to as ADMINNET) for the KVM host and the guests.
  • Private host names and IP addresses (referred to as PRIVNET) for the KVM host and the guests.
  • Integrated Lights Out Manager (ILOM) host names and IP addresses for the KVM host.
  • Client host names and IP addresses (referred to as CLIENTNET) for the guests.
  • Virtual IP (VIP) host names and IP addresses (referred to as VIPNET) for the guests.
  • Physical rack number and location of the new node in the rack (in terms of U number)

• Each KVM host has been imaged or patched to the same image in use on the existing database servers. The current system image must match the version of the /EXAVMIMAGES/ System.first.boot.*.img file on the new KVM host node.

  Note:

  The ~/dom0_group file referenced below is a text file that contains the host names of the KVM hosts for all existing and new nodes being added.

  Check that the image version across all KVM hosts are the same.

  dcli -g ~/dom0_group -l root "imageinfo -ver"
  
  exa01adm01: 19.2.0.0.0.190225
  exa01adm02: 19.2.0.0.0.190225
  exa01adm03: 19.2.0.0.0.190225

  If any image versions differ, you must upgrade the nodes as needed so that they match.

  Ensure that the System.first.boot version across all KVM hosts matches the image version retrieved in the previous step.

  dcli -g ~/dom0_group -l root "ls  -1 /EXAVMIMAGES/System.first.boot*.img" 
  exa01adm01:  /EXAVMIMAGES/System.first.boot.19.2.0.0.0.190225.img
  exa01adm02:  /EXAVMIMAGES/System.first.boot.19.2.0.0.0.190225.img
  exa01adm03:  /EXAVMIMAGES/System.first.boot.19.2.0.0.0.190225.img

  If any nodes are missing the System.first.boot.img file that corresponds to the current image, then obtain the required file. See the “Supplemental README note” for your Exadata release in My Oracle Support Doc ID 888828.1 and look for the patch file corresponding to this description, “DomU System.img OS image for V.V.0.0.0 VM creation on upgraded dom0s”

• Place the klone.zip files (gi-klone*.zip and db-klone*.zip) in the /EXAVMIMAGES location on the freshly imaged KVM host node you are adding to the cluster. These files can be found in the/EXAVMIMAGES directory on the KVM host node from where the system was initially deployed.

The following examples show how to add a new KVM host node named exa01adm03 that will have a new guest named exa01adm03vm01. The steps show how to extend an existing Oracle RAC cluster onto the guest using OEDACLI commands. The existing cluster has KVM host nodes named exa01adm01 and exa01adm02 and guest nodes named exa01adm01vm01 and exa01adm02vm01.

1. Add the KVM host information to the OEDA XML file using the CLONE COMPUTE command.

   In the following examples, the OEDA XML file is assumed to be in: unzipped_OEDA_location/ExadataConfigurations.

   OEDACLI> LOAD FILE NAME=exa01_original_deployment.xml 
   
   OEDACLI> CLONE COMPUTE SRCNAME=exa01adm01 TGTNAME=exa01adm03
   OEDACLI> SET ADMINNET NAME=exa01adm03,IP=xx.xx.xx.xx
   OEDACLI> SET PRIVNET NAME1=exa01adm03-priv1,IP1=xx.xx.xx.xx,NAME2=exa01adm03-priv2,IP2=xx.xx.xx.xx
   OEDACLI> SET ILOMNET NAME=exa01adm03-c,IP=xx.xx.xx.xx
   OEDACLI> SET RACK NUM=NN,ULOC=XX 
   
   OEDACLI> SAVE ACTION
   OEDACLI> MERGE ACTIONS FORCE
   OEDACLI> SAVE FILE NAME=exa01_plus_adm03_node.xml

   At this point we have a new XML file (exa01_plus_adm03_node.xml) that has the new compute node KVM host in the configuration. This file will be used in the following steps.

2. Add the new guest information to the OEDA XML file using the CLONE GUEST command and deploy the guest.

   • The first example shows how to control deployment of the new guest by using a WHERE clause in the CLONE GUEST command to specify the name of each step. If you choose to perform deployment this way, you must run all of the other deployment steps in order as follows:

     OEDACLI> LOAD FILE NAME=exa01_plus_adm03_node.xml 
     
     OEDACLI> CLONE GUEST SRCNAME=exa01adm01vm01 TGTNAME=exa01adm03vm01 WHERE STEPNAME=CREATE_GUEST
     OEDACLI> SET PARENT NAME=exa01adm03
     OEDACLI> SET ADMINNET NAME=exa01adm03vm01,IP=xx.xx.xx.xx
     OEDACLI> SET PRIVNET NAME1=exa01db03vm01-priv1,IP1=xx.xx.xx.xx,NAME2=exa01db03vm01-priv2,IP2=xx.xx.xx.xx
     OEDACLI> SET CLIENTNET NAME=exa01client03vm01,IP=xx.xx.xx.xx
     OEDACLI> SET VIPNET NAME=exa01client03vm01-vip,IP=xx.xx.xx.xx
     
     OEDACLI> SAVE ACTION
     OEDACLI> MERGE ACTIONS
     OEDACLI> DEPLOY ACTIONS
     
     OEDACLI> CLONE GUEST SRCNAME=exa01adm01vm01 TGTNAME=exa01adm03vm01 WHERE STEPNAME=CREATE_USERS

     OEDACLI> SAVE ACTION
     OEDACLI> MERGE ACTIONS
     OEDACLI> DEPLOY ACTIONS
     
     OEDACLI> CLONE GUEST SRCNAME=exa01adm01vm01 TGTNAME=exa01adm03vm01 WHERE STEPNAME=CELL_CONNECTIVITY

     OEDACLI> SAVE ACTION
     OEDACLI> MERGE ACTIONS
     OEDACLI> DEPLOY ACTIONS
     
     OEDACLI> CLONE GUEST SRCNAME=exa01adm01vm01 TGTNAME=exa01adm03vm01 WHERE STEPNAME=ADD_NODE

     OEDACLI> SAVE ACTION
     OEDACLI> MERGE ACTIONS
     OEDACLI> DEPLOY ACTIONS
     
     OEDACLI> CLONE GUEST SRCNAME=exa01adm01vm01 TGTNAME=exa01adm03vm01 WHERE STEPNAME=EXTEND_DBHOME

     OEDACLI> SAVE ACTION
     OEDACLI> MERGE ACTIONS
     OEDACLI> DEPLOY ACTIONS
     
     OEDACLI> CLONE GUEST SRCNAME=exa01adm01vm01 TGTNAME=exa01adm03vm01 WHERE STEPNAME=ADD_INSTANCE

     OEDACLI> SAVE ACTION
     OEDACLI> MERGE ACTIONS
     OEDACLI> DEPLOY ACTIONS

   • Alternatively, you can perform all of the deployment steps using one CLONE GUEST command by omitting the WHERE clause. For example:

     OEDACLI> LOAD FILE NAME=exa01_plus_adm03_node.xml 
     
     OEDACLI> CLONE GUEST SRCNAME=exa01adm01vm01 TGTNAME=exa01adm03vm01
     OEDACLI> SET PARENT NAME=exa01adm03
     OEDACLI> SET ADMINNET NAME=exa01adm03vm01,IP=xx.xx.xx.xx
     OEDACLI> SET PRIVNET NAME1=exa01db03vm01-priv1,IP1=xx.xx.xx.xx,NAME2=exa01db03vm01-priv2,IP2=xx.xx.xx.xx
     OEDACLI> SET CLIENTNET NAME=exa01client03vm01,IP=xx.xx.xx.xx
     OEDACLI> SET VIPNET NAME=exa01client03vm01-vip,IP=xx.xx.xx.xx
     
     OEDACLI> SAVE ACTION
     OEDACLI> MERGE ACTIONS
     OEDACLI> DEPLOY ACTIONS

   Regardless of the deployment method, for each step OEDACLI displays progress information similar to the following:

   Deploying Action ID : 39 CLONE GUEST SRCNAME=exa01adm01vm01 TGTNAME=exa01adm03vm01 where STEPNAME=ADD_INSTANCE 
   Deploying CLONE GUEST 
   Cloning Guest 
   Cloning Guest  :  exa01adm03vm01.example.com_id 
   Adding new instance for database [dbm] on exa01adm03vm01.example.com 
   Setting up Huge Pages for Database..[dbm] 
   Adding instance dbm3 on host exa01adm03vm01.example.com 
   Successfully completed adding database instance on the new node [elapsed Time [Elapsed = 
   249561 mS [4.0  minutes] Fri Jun 28 13:35:52 PDT 2019]] 
   Done...
   Done

3. Save the current state of the configuration and generate configuration information.

   OEDACLI> SAVE FILES LOCATION=/tmp/exa01_plus_adm03_config

   The above command writes all the configuration files to the directory /tmp/exa01_plus_adm03_config. Save a copy of these files in a safe place since they now reflect the changes made to your cluster.

4. Gather an Exachk report and examine it to ensure the cluster is in good health.