4 Recovery of the Exalogic Control Stack from Hardware Failures and Corruption

Exalogic Control is a comprehensive software management stack providing onboarded capabilities for Exalogic machine, vDC management, and monitoring.

This chapter contains the following sections:

4.1 Recovering from a Hardware Failure

This section describes how you can recover each component of the Exalogic Control stack if a compute node running an Exalogic Control vServer crashes.

4.1.1 Recovering the Database vServer from a Hardware Failure

The database repositories for Oracle VM Manager (ovs) and Enterprise Manager Ops Center (emoc, emoc_ro) are deployed to a database vServer. By default, the database vServer is deployed to the first compute node in the first pool. If the database vServer crashes, both Oracle VM manager and Enterprise Manager Ops Center will stop being operational.

After the compute node is restored to its previous state, do the following to start the database vServer:

  1. Log in to the compute node as the root user.

  2. Change directory to /OVS/Repositories/*/VirtualMachines.

    hostname#cd /OVS/Repositories/*/VirtualMachines
    
  3. Find the absolute path to the virtual machine configuration file for the database vServer.

    Run the following grep command to identify the correct configuration file corresponding to the Exalogic Control database vServer:

    hostname# grep -i ExalogicControlDB */vm.cfg
    

    The output is similar to:

    0004fb00000600002c18bee8647fb8f7/vm.cfg:OVM_simple_name = 'ExalogicControlDB'
    
  4. Start the database vServer by using the xm create command.

    xm create absolute_path_to_vm.cfg
    

    Example:

    xm create 0004fb00000600002c18bee8647fb8f7/vm.cfg
    
  5. Verify whether the database vServer and all the database processes are running by logging in to the vServer.

  6. Restart the Oracle VM Manager process:

    1. Log in to the Oracle VM Manager vServer as root.

    2. Run service ovmm stop to stop the Oracle VM Manager process.

    3. Run service ovmm start to start the Oracle VM Manager process.

    4. Verify whether Oracle VM Manager started correctly by logging in to the Oracle VM Manager console.

  7. Restart the Proxy Controller and Enterprise Controller processes:

    1. Log in to each of the Proxy Controller vServers as root.

    2. Run proxyadm stop to stop the Proxy Controller.

    3. Run proxyadm start to start the Proxy Controller.

    4. Log in to the Enterprise Controller vServer as root.

    5. Run satadm stop to stop the Enterprise Controller.

    6. Run satadm start to start the Enterprise Controller.

    7. Verify whether the Proxy Controller and Enterprise Controller vServers restarted successfully by logging in to the Exalogic Control BUI.

4.1.2 Recovering the Oracle VM Manager vServer from a Hardware Failure

The Oracle VM Manager is deployed to the ovmm vServer. By default, it is deployed to the first compute node in the first pool. If the Oracle VM Manager vServer crashes, Enterprise Manager Ops Center functionality will be affected.

After the compute node is restored to its previous state, do the following to start the Oracle VM Manager vServer:

  1. Log in to the compute node as the root user.

  2. Change directory to /OVS/Repositories/*/VirtualMachines.

    cd /OVS/Repositories/*/VirtualMachines
    
  3. Find the absolute path to virtual machine configuration file for the Oracle VM Manager vServer.

    Run the following grep command to identify the correct configuration file:

    Hostname# grep -i ExalogicControlOVMM */vm.cfg
    

    The output is similar to:

    0004fb000006000088afde54f9794d32/vm.cfg:OVM_simple_name = 'ExalogicControlOVMM'
    
  4. Start the Oracle VM Manager vServer using the xm create command.

    xm create absolute_path_to_vm.cfg
    

    Example:

    xm create 0004fb000006000088afde54f9794d32/vm.cfg
    
  5. Verify whether the Oracle VM Manager vServer and all the Oracle VM processes are running by logging in to the vServer.

  6. Verify whether Enterprise Manager Ops Center is fully functional by logging in to the Exalogic Control BUI.

4.1.3 Recovering the Proxy Controller vServer from a Hardware Failure

The Proxy Controller component is deployed as two vServers. By default, the second vServer (pc2) is deployed to the second compute node in the first pool and the first proxy controller (pc1) vServer is deployed to the third compute node in the first pool. Enterprise Manager Ops Center functionality will be affected if either of the Proxy Controller vServers crash.

After the compute nodes are recovered to their previous state, do the following to start the Proxy Controller vServer:

  1. Log in to the compute node as the root user.

  2. Change directory to /OVS/Repositories/*/VirtualMachines.

    cd /OVS/Repositories/*/VirtualMachines
    
  3. Find the absolute path to virtual machine configuration file for the Proxy Controller vServer.

    Run the following grep command to identify the correct configuration file:

    Hostname# grep -i ExalogicControlOpsCenterPC* */vm.cfg
    

    The output is similar to:

    0004fb0000060000821f3e60a6d3502d/vm.cfg:OVM_simple_name = 'ExalogicControlOpsCenterPC2'
     
    0004fb000006000084a183dbe7c3dba0/vm.cfg:OVM_simple_name = 'ExalogicControlOpsCenterPC1'
    
  4. Start the Proxy Controller vServer by using the xm create command.

    xm create absolute_path_to_vm.cfg
    

    Example:

    xm create 0004fb0000060000cf01f02c2fb5adaf/vm.cfg
    
  5. Verify whether Enterprise Manager Ops Center is fully functional by logging in to the Exalogic Control BUI.

4.1.4 Recovering the Enterprise Controller vServer from a Hardware Failure

The Enterprise Controller component of the Enterprise Manager Ops Center is deployed to the enterprise controller vServer. By default, the enterprise controller vServer is deployed to the fourth compute node in the first pool. All provisioning and lifecycle management functionality is affected if the enterprise controller vServer is unavailable. Once the compute node is recovered to its previous state, follow the steps to start the enterprise controller vServer:

  1. Log in to the compute node as the root user.

  2. Change the directory to /OVS/Repositories/*/VirtualMachines:

    cd /OVS/Repositories/*/VirtualMachines
    
  3. Find the absolute path to virtual machine configuration file for the Proxy Controller vServer.

    Run the following grep command to identify the correct configuration file:

    Hostname# grep -i ExalogicControlOpsCenterEC1 */vm.cfg
    

    The output is similar to:

    0004fb0000060000cf01f02c2fb5adaf/vm.cfg:OVM_simple_name = 'ExalogicControlOpsCenterEC1'
    
  4. Manually start the enterprise controller vServer using the xm create command. The syntax for the command is:

    xm create absolute_path_to_vm.cfg
    

    Example:

    xm create 0004fb0000060000cf01f02c2fb5adaf/vm.cfg
    
  5. Validate that Ops Center is fully functional by logging in to the Ops Center BUI.

4.2 Backing Up and Recovering Oracle VM Manager

This section provides the backup and recovery steps for the Oracle VM Manager and its repository.

Save the backup to the NFS location you created for the Exalogic Control stack, as described in Chapter 2, "Backup and Recovery Locations" (for example, /export/Exalogic_Backup/control_metadata).

Create a directory for storing the backups of the Oracle VM manager under this NFS share (for example, /export/Exalogic_Backup/control_metadata/ovmm).

4.2.1 Backing Up Oracle VM Manager

To back up Oracle VM Manager, you should back up the Oracle VM Manager configuration file, and the Oracle VM Manager database schema. By default, this schema is named ovs, and this name is used in the backup example; when you run this procedure, replace the schema name with your own.

The Oracle VM Manager configuration file is stored at the following location on the Oracle VM manager vServer:

/u01/app/oracle/ovm-manager-3/.config

This configuration file contains database connection information, ports, and the UUID used by Oracle VM Manager.

The following is an example of this configuration file:

DBHOST=<hostname of database server>
SID=<oracle SID>
LSNR=<listener port number defaults 1521>
APEX=<application express port number defaults 8080>
OVSSCHEMA=<database schema name for oracle vm manager defaults ovs>
WLSADMIN=<weblogic server admin defaults weblogic>
OVSADMIN=<oracle vm manager administrator name defaults admin>
COREPORT=<oracle vm manager core port defaults 54321>
UUID=<oracle vm manager uuid>

To back up Oracle VM Manager, do the following:

  1. Back up or copy the Oracle VM Manager configuration file located at:

    /u01/app/oracle/ovm-manager-3/.config
    
  2. As the root user, shut down Oracle VM Manager.

    # /sbin/service ovmm stop
    
  3. Back up the Oracle VM Manager database schema.

    1. Log in to any compute node on the Exalogic machine.

    2. Mount the NFS location created for the exalogic control stack.

    3. Log in to the Oracle database vServer from the compute node using the IP address of the IPoIB-admin interface.

    4. Log in to the operating system as the oracle user.

      You can use the su - oracle command as the root user if you do not have the password for the oracle user.

    5. Find the version of Oracle Database installed by navigating to the following directory:

      /u01/app/oracle/product/
      
    6. Set the ORACLE_HOME, PATH and ORACLE_SID environment variables by running the following commands:

      export ORACLE_HOME=/u01/app/oracle/product/ProductVersion/dbhome_1
      

      ProductVersion is the version of Oracle Database installed on your Exalogic rack that you found in step e.

      export PATH=$ORACLE_HOME/bin:$PATH
      
      export ORACLE_SID=elctrldb
      
    7. Navigate to the following directory:

      /u01/app/oracle/product/ProductVersion/dbhome_1/bin
      
    8. Export the schema using the using the exp command:

      exp ovs/password grants=y compress=y file=/tmp/ovsbackup.dmp
      
    9. FTP the backup file to the compute node from step a. Use the mount point of the NFS share as the destination in the ftp command.

    10. Store the Oracle VM Manager database schema backup along with the Oracle VM Manager configuration file.

4.2.2 Restoring Oracle VM Manager

To restore Oracle VM Manager, and the Oracle VM Manager database schema from a backup, you must have performed the steps to back up Oracle VM Manager as described in Section 4.2.1, "Backing Up Oracle VM Manager."

Note:

The OVS repository and the Oracle VM Manager must always be restored together. They cannot be restored as individual components.

  1. In certain cases it may be necessary to either re-install or upgrade the Oracle VM Manager. For more information, see the following documentation:

    Log in to the Oracle VM Manager vServer and then perform the install using the runInstaller.sh --uuid uuid command and provide the UUID from the previous manager installation you created a backup from. The UUID can be found in the Oracle VM Manager configuration file.

    Note:

    The Oracle VM Manager UUID is also persisted in the /etc/sysconfig/ovmm file. If the system disk of the server on which you are installing or restoring Oracle VM Manager was not wiped entirely, the existing UUID is still present and will be detected when running the installer.

    • The --uuid option overrides this existing UUID.

    • If no UUID is present in /etc/sysconfig/ovmm, the --uuid option adds the UUID to the file.

    Example:

    # ./runInstaller.sh --uuid 0004FB000000100002CB7F2DFFA8D8
    

    When the Oracle VM Manager installer prompts for installation information other than passwords, reuse the same user names for the Oracle Database schema, Oracle WebLogic Server and Oracle VM Manager administration user, as set out in the backup of the Oracle VM Manager configuration file. You must set the passwords again as the passwords are not backed up and cannot be restored.

  2. After installation, reinstallation or upgrade, stop Oracle VM Manager before you restore the backup:

    # /sbin/service ovmm stop
    
  3. Log in to any compute node in the Exalogic machine.

  4. Mount the NFS location created for the Exalogic control stack.

  5. Log in to the Oracle Database vServer from the compute node using the IP address of the IPoIB-admin interface.

  6. Log in to the operating system as the oracle user. You can su - oracle as the root user if you do not have the password for the oracle user.

  7. Find the version of Oracle Database installed by navigating to the following directory:

    /u01/app/oracle/product/
    
  8. Set the ORACLE_HOME, PATH and ORACLE_SID environment variables by running the following commands:

    export ORACLE_HOME=/u01/app/oracle/product/ProductVersion/dbhome_1
    

    ProductVersion is the version of Oracle Database installed on your Exalogic rack that you found in step e.

    export PATH=$ORACLE_HOME/bin:$PATH
    
    export ORACLE_SID=elctrldb
    
  9. Log in to the Oracle Database as the sys or system user to delete the Oracle VM Manager administration user. The default Oracle VM Manager administration user is ovs.

    $ sqlplus system/password
    SQL> drop user ovs cascade;
    

    Then, re-create the Oracle VM Manager administration user, with the necessary grants:

    SQL> create user ovs identified by password;
    SQL> grant connect, resource to ovs;
    SQL> exit;
    
  10. FTP the backup file of the OVS schema from the NFS location mounted on the compute node from Step 1 to a temporary location on the database vServer.

  11. Restore the OVS schema by importing it from the backup file.

    # imp ovs/password file=/tmp/ovmmovsbackup.dmp full=y
    
  12. Restart Oracle VM Manager.

    # /sbin/service ovmm start
    

4.3 Recovering Oracle VM Manager After Database Corruption

This section provides the procedure for recovering from corruption of the Oracle VM Manager data in the database.

Save the backups for the Oracle VM Manager and Enterprise Manager Ops Center repositories to the NFS locations created as described in Chapter 2, "Backup and Recovery Locations" (for example, /export/Exalogic_Backup/control_metadata).

This procedure has the following steps:

Step 1: Shut Down the Enterprise Manager Ops Center VMs

Step 2: Cleanup the Oracle VM Manager RDBMS Schema

Step 3: Rediscover the Environment

Step 4: Start the Enterprise Manager Ops Center Control VMs

Step 5: Synchronize the Clocks of the Control vServers

Step 6: Verify Whether Enterprise Manager Ops Center is Running

Step 1: Shut Down the Enterprise Manager Ops Center VMs

When the Oracle VM Manager database gets corrupted, the Oracle VM Manager can no longer operate. As a result Enterprise Manager Ops Center cannot perform any of the Exalogic cloud management functions.

The Enterprise Manager Ops Center control VMs must be shut down in the following order:

  1. ExalogicControlOpsCenterEC1 VM

  2. ExalogicControlOpsCenterPC1 VM

  3. ExalogicControlOpsCenterPC2 VM

First, determine the OVS server on which the Enterprise Manager Ops Center control VMs are running. Note that the VMs must be restarted on the same OVS servers later.

Note:

By default, the VMs run on the following OVS servers:

  • ExalogicControlOpsCenterEC1 VM: cn04

  • ExalogicControlOpsCenterPC1 VM: cn03

  • ExalogicControlOpsCenterPC2 VM: cn02

  1. Launch a web browser and log in to the Oracle VM Manager web console.

  2. Expand Server Pools under the Home section.

  3. Under the Serverpool1 pool, expand the OVS servers one by one till you find the three VMs listed earlier.

There are several ways to shut down the Enterprise Manager Ops Center control VM instances. This section documents two of them.

  • Using Oracle VM Manager web console

    1. Select the ExalogicControlOpsCenterEC1 VM and click Stop.

    2. Select the ExalogicControlOpsCenterPC1 VM and click Stop.

    3. Select the ExalogicControlOpsCenterPC2 VM and click Stop.

    Wait for each VM to be shut down before proceeding.

  • Using Oracle VM Manager CLI

    Do the following for each of the Exalogic Control VMs, in the order listed earlier:

    1. SSH to the Oracle VM Manager VM by using the EoIB-external-mgmt IP address assigned to that VM.

    2. From inside the Oracle VM Manager VM, SSH to the OVS server (dom0) where the VM is running using the IPoIB-ovm-mgmt IP address assigned to that OVS server. If you used the default ECU settings, then the IP address will be:

      192.168.23.N
      

      N is the index of the OVS server.

      Example:

      For cn01, the IP will be:

      192.168.23.1
      
    3. Find the UUID of the VM that must be shut down.

      This can be done using the Oracle VM Manager web console: Select the VM instance. The ID will be shown in the window on the right. You can confirm that the VM runs on this OVS server by running the following command:

      xm list
      

      If the control VMs were never stopped, then they will have index 1 on the respective OVS server.

    4. To gracefully shut down the VM, run the following command:

      xm shutdown UUID
      

      Optionally, you can monitor the VM while it is shutting down by running this command:

      xm console UUID
      
    5. Verify that the VM instance has shut down by running the following command:

      xm list
      

      The UUID of the VM that was stopped should not be listed.

      Note:

      The VM can be shut down forcefully. Oracle does not recommend this process. Use it only if the previous steps do not work:

      xm destroy UUID
      

Step 2: Cleanup the Oracle VM Manager RDBMS Schema

  1. Stop the Oracle VM Manager service.

    1. SSH as root to the Oracle VM Manager VM using its EoIB-external-mgmt IP address.

    2. Run the following command:

      service ovmm stop
      
  2. Fix any RDBMS corruptions.

    For example, if there is a table containing a corrupted row, that row must be deleted using normal RDBMS tools such as SQL*Plus. If the RDBMS error ORA-1555 is present in any of the log files, then there is possibly a corrupted BLOB value. The section below covers how to confirm and delete that row.

    Note:

    If more than one table contains corrupted BLOBs you should repeat this procedure for each table. You should drop the corrupted_lob_data table by using the following statement:

    SQL> drop table corrupted_lob_data;
    

    The following are the steps to find the corrupted BLOB value and to delete it:

    1. SSH as root to the Oracle VM Manager VM using its EoIB-external-mgmt IP address.

    2. From the Oracle VM Manager VM, SSH as root to the DB VM instance using its IPoIB-admin IP address. The default IP address is 192.168.20.10.

      Run the following commands to connect to the Oracle VM Manager database schema:

      # ssh root@192.168.20.10
      The authenticity of host '192.168.20.10 (192.168.20.10)' can't be established.
      RSA key fingerprint is f6:14:37:f9:ef:45:ba:48:73:76:35:7f:a9:e0:99:ab.
      Are you sure you want to continue connecting (yes/no)? yes
      Warning: Permanently added '192.168.20.10' (RSA) to the list of known hosts.
      root@192.168.20.10's password:
      Last login: Wed May 16 13:39:58 2012
      [root@elir-db ~]# su - oracle
      [oracle@elir-db root]$ export ORACLE_HOME=/u01/app/oracle/product/DatabaseVersion/dbhome_1
      

      DatabaseVersion is the version of Oracle DB installed on your Exalogic machine.

      [oracle@elir-db root]$ export ORACLE_SID=elctrldb
      [oracle@elir-db root]$ cd /u01/app/oracle/product/DatabaseVersion/dbhome_1/bin
      [oracle@elir-db bin]$ ./sqlplus ovs@elctrldb
       
      SQL*Plus: Release 11.2.0.1.0 Production on Wed May 16 13:39:58 2012
       
      Copyright (c) 1982, 2009, Oracle.  All rights reserved.
       
      Enter password:
       
      Connected to:
      Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
      With the Partitioning, OLAP, Data Mining and Real Application Testing options
       
      SQL>
      
    3. Create a new temporary table for storing all row IDs of the corrupted LOBs. In this example, we will call it corrupted_lob_data.

      SQL> create table corrupted_lob_data (corrupted_rowid rowid);
      
    4. Execute the following SQL script to detect BLOB data corruptions:

      SQL> set concat off
       
      declare
        error_1555 exception;
        pragma exception_init(error_1555,-1555);
        num number;
      begin
        for cursor_lob in (select rowid r, &&lob_column from &table_owner.&table_with_lob) loop
          begin
            num := dbms_lob.instr (cursor_lob.&&lob_column, hextoraw ('889911')) ;
          exception
            when error_1555 then
              insert into corrupted_lob_data values (cursor_lob.r);
              commit;
          end;
        end loop;
      end;
      /
      

      After this step, the following prompts are displayed:

      Enter value for lob_column    : m_data
      Enter value for table_owner   : ovs
      Enter value for table_with_LOB: You need to get this information from your application log file, for example, Mgr_Eventlog, run this for each table you saw in the log prior to the exception
      

      In the end all row IDs of the corrupted LOBs are inserted into the corrupted_lob_data table created earlier.

    5. Check whether any corrupted BLOBs were found by executing the following query:

      SQL> select * from corrupted_lob_data;
       
      CORRUPTED_ROWID
      ---------------------
      AAEWBsAAGAAACewAAC
      AAEWBsAAGAAACewAAF
      AAEWBsAAGAAACewAAG
       
      3 rows selected
      
    6. Remove the corrupted BLOB value, by doing one of the following:

      Empty the affected LOBs:

      SQL> update table_name_with_corrupted_blob set m_data = empty_blob()
           where rowid in (select corrupted_rowid from corrupted_lob_data);
      

      Delete the rows with the corrupted BLOB value:

      SQL> delete from table_name_with_corrupted_blob where rowid in (select corrupted_rowid from corrupted_lob_data);
      

    Notes:

    The SQL*Plus tool is available only on the RDBMS VM. You can access that VM through SSH from the Oracle VM Manager VM to the RDBMS VM using the IPoIB-admin IP address assigned to that VM.

    The following My Oracle Support documents describe how to find corrupted BLOB segments:

    • 833635.1: Export Fails with ORA-2354 ORA-1555 ORA-22924 and How to Confirm LOB Segment Corruption Using Export Utility?

    • 787004.1: Export Receives the Errors ORA-1555 ORA-22924 ORA-1578 ORA-22922

  3. Delete all tables in the Oracle VM Manager RDBMS schema.

    1. SSH as root to the Oracle VM Manager VM using its EoIB-external-mgmt IP address.

    2. Gather some metadata information about the current Oracle VM Manager instance by running the following commands:

      [root@elir-ovmm ovm-manager-3]# cat /u01/app/oracle/ovm-manager-3/.config
      DBHOST=192.168.20.10
      SID=elctrldb
      LSNR=1521
      APEX=8080
      OVSSCHEMA=ovs
      WLSADMIN=weblogic
      OVSADMIN=admin
      COREPORT=54321
      UUID=0004fb0000010000e224fcdfc21df2d2
      BUILDID=3.0.3.240
      

      You can find the UUID assigned to the Oracle VM Manager instance by running the following command. This UUID is required if Oracle VM Manager needs to be reinstalled.

      [root@elir-ovmm ovm-manager-3]# cat /etc/sysconfig/ovmm
      UUID=0004fb0000010000e224fcdfc21df2d2
      RUN_OVMM=YES
      
    3. Delete all tables in the Oracle VM Manager RDBMS schema by running the following commands and getting the exact values for the parameters from file /u01/app/oracle/ovm-manager-3/.config.

      # cd /u01/app/oracle/ovm-manager-3/ovm_upgrade/bin/
      # bash ./ovm_upgrade.sh --dbuser=OVSSCHEMA --dbpass=OVSSCHEMA_PASSWORD --dbhost=DBHOST --dbport=LSNR --dbsid=SID --deletedb
      

      Example, using default values:

      # cd /u01/app/oracle/ovm-manager-3/ovm_upgrade/bin/
      # bash ./ovm_upgrade.sh --dbuser=ovs --dbpass=default_password --dbhost=192.168.20.10 --dbport=1521 --dbsid=elctrldb --deletedb
      

      After running this command, the Oracle VM Manager RDBMS schema should still exist but it should be empty (no tables in it).

Step 3: Rediscover the Environment

  1. Start up the Oracle VM Manager service.

    # service ovmm start
    
  2. Discover all the OVS servers as follows:

    1. Launch a web browser and access the Oracle VM Manager web console by using the following URL:

      http://OVMM_VM_EoIB-external-mgmt_IP_Address:7002/ovm/console
      
    2. Expand the Hardware section.

    3. Select the Hardware tab.

    4. Click the Discover Servers button

      The Discover Servers window is displayed.

    5. From inside the Oracle VM Manager VM, SSH to the OVS server (dom0) where the VM is running using the IPoIB-ovm-mgmt IP address assigned to that OVS server. If you used the default ECU settings, then that IP address will be:

      192.168.23.N
      

      where N is the index of the OVS server

      Example:

      For cn01, the IP will be:

      192.168.23.1
      
    6. Enter the Oracle VM Agent password.

    7. Click OK.

    8. Wait till all the servers are discovered. At this point, all OVS servers should be OK.

      If any errors occur, you should select the OVS server that has errors. Then, go to the Events tab and acknowledge all events.

  3. Register the File Server.

    1. While still in the Hardware pane, select the Storage tab.

    2. Click on the Discover a File Server button.

      The Discover a File Server wizard is displayed.

    3. Enter the following values:

      Name: Generic Network File System

      Access Host: IPoIB-storage_IP_address_of_the_storage

      Note:

      In a quarter rack at default settings, the above IP address is 192.168.21.9

    4. Click Next.

    5. Select all the OVS servers and move them to the Selected Servers section.

    6. Click Next.

    7. Select only the nfs:/export/ExalogicRepo file system.

    8. Click Finish.

  4. Present the repository to all servers

    1. Expand the Home pane.

    2. Select Server Pools.

    3. Click on the Repositories tab on the right side of the page.

    4. Select the exlcontrol_repo repository (it will be the only one listed).

    5. Click on Present-Unpresent Selected Repository icon (represented by up and down green arrows).

    6. Present the repository to all OVS servers by moving all servers to the Present to Server(s) side.

    7. Click OK.

  5. Refresh the repository by selecting it and clicking Refresh Selected Repository Content in the toolbar.

  6. Click on the Servers and VMs tab.

  7. Rediscover all the servers by repeating the following procedure for every OVS server in each server pool:

    1. Select the OVS server.

    2. Click the Rediscover Server button in the toolbar.

      Alternatively, right-click and select Rediscover Server.

    After this step, all running VM instances should be shown under their respective OVS servers. Only the stopped VM instances should be listed under Unassigned Virtual Machines, which should include the three Enterprise Manager Ops Center control VMs.

    Note:

    The storage volumes assigned to each VM will be discovered but instead of displaying their names, Oracle VM Manager will display their UUIDs. This is not a problem.

Step 4: Start the Enterprise Manager Ops Center Control VMs

The Enterprise Manager Ops Center VMs should be started in the following order:

  1. ExalogicControlOpsCenterPC2

  2. ExalogicControlOpsCenterPC1

    Wait for five minutes before proceeding.

  3. ExalogicControlOpsCenterEC1

Refer to where the VMs were running, as noted in the Step 1: Shut Down the Enterprise Manager Ops Center VMs step.

Do the following for each VM:

  1. Select the VM from the Unassigned Virtual Machines list.

  2. Click Migrate in the toolbar.

    Alternatively, right-click on the VM name, and select Migrate.

  3. Select the OVS server in which the VM should run.

  4. Select the migrated VM. Expand the OVS server in which the VM should run.

  5. Click Start in the toolbar.

    Alternatively, right-click on the VM name and select Start.

Step 5: Synchronize the Clocks of the Control vServers

  1. Stop the components of the Exalogic Control stack, as described in Section 5.2.1, "Stopping the Components of the Exalogic Control Stack."

  2. Log in as root on the first compute node.

  3. Set up SSH for all the control vServers by running the following commands:

    #/opt/exalogic.tools/tools/setup-ssh.sh -H 192.168.23.10 -P password
    #/opt/exalogic.tools/tools/setup-ssh.sh -H 192.168.23.11 -P password
    #/opt/exalogic.tools/tools/setup-ssh.sh -H 192.168.23.12 -P password
    #/opt/exalogic.tools/tools/setup-ssh.sh -H 192.168.23.13 -P password
    #/opt/exalogic.tools/tools/setup-ssh.sh -H 192.168.23.14 -P password
    
  4. Create a file with the IP addresses of all the control vServers as follows:

    > 192.168.23.10
    > 192.168.23.11
    > 192.168.23.12
    > 192.168.23.13
    > 192.168.23.14
    
  5. Ensure that the time matches for all control vServers by running the following commands:

    # /opt/exalogic.tools/tools/dcli -g IPAddressesFile 'date'
     
    # /opt/exalogic.tools/tools/dcli -g IPAddressesFile 'service ntpd stop'
     
    # /opt/exalogic.tools/tools/dcli -g IPAddressesFile 'ntpd -gq'
     
    # /opt/exalogic.tools/tools/dcli -g IPAddressesFile 'service ntpd start'
     
    # /opt/exalogic.tools/tools/dcli -g IPAddressesFile 'date'
    

    Note:

    To synchronize the time between control vServers, you may have to repeat the ntpd related commands.

  6. Start the components of the Exalogic Control stack, as described in Section 5.2.5, "Starting the Exalogic Control Stack."

Step 6: Verify Whether Enterprise Manager Ops Center is Running

After the ExalogicControlOpsCenterEC1 VM is up, it may take 10 to 15 minutes for the Enterprise Manager Ops Center to completely start and become accessible from a web browser.

Launch a web browser and access the Exalogic Control BUI. Enterprise Manager Ops Center should be fully functional at this point.