A Recovery of the Exalogic Control Stack From Hardware Failures

This appendix describes how to recover components of the Exalogic Control stack if a Oracle VM Server node running an Exalogic Control or Proxy Controller vServer crashes.

Note:

Use the procedures in this appendix only when directed to in Section 3.2.2.4, "Reimaging and Recovering Oracle VM Server Nodes in a Virtual Environment."

A.1 Recovering the Exalogic Control vServer From Hardware Failures

Oracle VM Manager, Enterprise Manager Ops Center, and Database are deployed in the Exalogic Control vServer. By default, the Exalogic Control vServer is deployed to the first Oracle VM Server node in the first server pool. If the Exalogic Control vServer crashes, both Oracle VM manager and Exalogic Control are not operational.

Migrate the Exalogic Control vServer, by doing the following:

  1. Identify the Oracle VM Server node to which you should migrate the Exalogic Control vServer.

    Failed Nodes Migrate to Node
    Node 1 Node 4
    Node 4 Node 1
    Nodes 1 and 4 Any running node

  2. Start the Exalogic Control vServer, by performing the following steps:

    1. Log in as the root user on the Oracle VM Server node you identified in the previous step.

    2. Find the absolute path to the virtual machine configuration file for the Exalogic Control vServer.

      Run the following grep command to identify the correct configuration file corresponding to the Exalogic Control vServer:

      # grep "ExalogicControl" /OVS/Repositories/*/*/*/vm.cfg

      Note:

      On an Exalogic rack that was upgraded to EECS 2.0.6 from EECS 2.0.4, the ExalogicControl vServer is called ExalogicControlOVMM.

      Example:

      # grep "ExalogicControl" /OVS/Repositories/*/*/*/vm.cfg

/OVS/Repositories/0004fb00000300007d4eef3af41ca807/VirtualMachines/0004fb00
00060000014361b5c6f404/vm.cfg:OVM_simple_name='ExalogicControlOpsCenterPC1'
 
/OVS/Repositories/0004fb00000300007d4eef3af41ca807/VirtualMachines/0004fb00
0006000040e5af16d3288845/vm.cfg:OVM_simple_name = 'ExalogicControl'
 
/OVS/Repositories/0004fb00000300007d4eef3af41ca807/VirtualMachines/0004fb00
0060000cbf5bca84ab4b65/vm.cfg:OVM_simple_name='ExalogicControlOpsCenterPC2

      In this example, the absolute path of the Exalogic Control vServer is as follows:

      /OVS/Repositories/0004fb00000300007d4eef3af41ca807/VirtualMachines/0004fb000006000040e5af16d3288845/vm.cfg

    3. Start the Exalogic Control vServer, by using the xm create command as follows:

      xm create absolute_path_to_vm.cfg

      Example:

      xm create /OVS/Repositories/0004fb00000300007d4eef3af41ca807/VirtualMachines/0004fb000006000040e5af16d3288845/vm.cfg

      Exalogic Control can take at least five minutes to start.

  3. Verify that the Exalogic Control vServer is running, by logging in to the Exalogic Control BUI as the root user.

  4. If the Proxy Controller vServers are down, follow the procedure in Section A.2, "Recovering the Proxy Controller vServers From Hardware Failures" to start the vServers manually.

  5. Identify the IP addresses of the Proxy Controller vServers, by doing the following:

    1. Log in to any Oracle VM Server node on the Exalogic rack as the root user.

    2. If the ExalogicControl share is not mounted in the mnt/ExalogicControl directory, mount it.

    3. Navigate to the /mnt/ExalogicControl/ECU_ARCHIVE directory.

    4. Extract the archive of the ECU files called ecu_log-date&time_stamp.tgz.

      The following files are extracted ecu_run_time.tgz, ecu_home.tgz, ecu_archive.tgz.

    5. Extract ecu_run_time.tgz that contains the ECU configuration files.

      A directory called ecu that contains all the configuration files as extracted.

    6. Navigate to the extracted ecu directory.

      cd ecu

    7. Identify the IP addresses of the Proxy Controller 1 and 2 vServers respectively, by running the following commands:

      # grep ecu_pc_IPoIB-admin_primary ops_center.properties
# grep ecu_pc_IPoIB-admin_secondary ops_center.properties

    8. Note the IP addresses of the Proxy Controller vServers.

  6. After restarting the Exalogic Control vServer, restart the Proxy Controllers by doing the following:

    1. Log in to any Oracle VM Server node on the Exalogic rack as the root user.

    2. SSH to the Proxy Controller vServer you want to restart, with the IP address you identified in step 5, as the root user.

      Example:

      # ssh root@192.168.20.12

    3. Stop the Proxy Controller services by running the following command:

      [root@hostname-pc1 ~]#  /opt/sun/xvmoc/bin/proxyadm stop -w

proxyadm: Shutting down Proxy Controller using SMFlite...
application/scn/proxy-available:default... ... stopped.
application/scn/uce-proxy:default... ... stopped.
application/management/common-agent-container:scn-proxy... ... stopped.
application/scn/proxy-enable:default... ... stopped.
proxyadm: Proxy Controller services have stopped

    4. Start the Proxy Controller services by running the following command:

      [root@hostname-pc1 ~]#  /opt/sun/xvmoc/bin/proxyadm start -w

proxyadm: Starting Proxy Controller with SMFlite...
application/scn/proxy-enable:default... ...started.
application/scn/uce-proxy:default... ...started.
application/management/common-agent-container:scn-proxy... ...started.
application/scn/proxy-available:default... ...started.
proxyadm: Proxy Controller services have started

    5. Run the exit command to exit the Proxy Controller vServer.

    6. Repeat steps a to e for the other Proxy Controller vServer.

A.2 Recovering the Proxy Controller vServers From Hardware Failures

The Proxy Controller is deployed as two vServers. By default, the first Proxy Controller vServer is deployed to the first Oracle VM Server node in the first pool and the second vServer is deployed to the second Oracle VM Server node in the first pool. If either of the Proxy Controller vServers is down, Exalogic Control functionality is affected.

Start the Proxy Controller vServers manually, by doing the following:

  1. Verify that the Exalogic Control vServer is running, by logging in to the Exalogic Control BUI as the root user.

  2. Identify the IP addresses of the Proxy Controller vServers, by doing the following:

    1. Log in to any Oracle VM Server node on the Exalogic rack as the root user.

    2. If the ExalogicControl share is not mounted in the mnt/ExalogicControl directory, mount it.

    3. Navigate to the /mnt/ExalogicControl/ECU_ARCHIVE directory.

    4. Extract the archive of the ECU files called ecu_log-date&time_stamp.tgz.

      The following files are extracted ecu_run_time.tgz, ecu_home.tgz, ecu_archive.tgz.

    5. Extract ecu_run_time.tgz that contains the ECU configuration files.

      A directory called ecu that contains all the configuration files as extracted.

    6. Navigate to the extracted ecu directory.

      cd ecu

    7. Identify the IP addresses of the Proxy Controller 1 and 2 vServers respectively, by running the following commands:

      # grep ecu_pc_IPoIB-admin_primary ops_center.properties
# grep ecu_pc_IPoIB-admin_secondary ops_center.properties

    8. Note the IP addresses of the Proxy Controller vServers.

  3. Identify the Oracle VM Server node on which you should start the Proxy Controller vServer.

    Proxy Controller Failed Nodes Migrate to Node

    Proxy Controller 1

    		 Node 1 Node 3
    Node 3 Node 1
    Nodes 1 and 3 Any running node that is not running Proxy Controller 2

    Proxy Controller 2

    		 Node 2 Node 4
    Node 4 Node 2
    Nodes 2 and 4 Any running node that is not running Proxy Controller 1

  4. Start the Proxy Controller vServer, by doing the following:

    1. Log in as the root user to the Oracle VM Server node you identified in the previous step.

    2. Find the absolute path to the virtual machine configuration file for the Proxy Controller vServer. Run the following grep command to identify the configuration file corresponding to the Proxy Controller vServer:

      # grep "ExalogicControlOpsCenterPC" /OVS/Repositories/*/*/*/vm.cfg

      Example:

      # grep "ExalogicControlOpsCenterPC" /OVS/Repositories/*/*/*/vm.cfg

/OVS/Repositories/0004fb00000300007d4eef3af41ca807/VirtualMachines/0004fb00
00060000014361b5c6f404/vm.cfg:OVM_simple_name='ExalogicControlOpsCenterPC1'
 
/OVS/Repositories/0004fb00000300007d4eef3af41ca807/VirtualMachines/
0000cbf5bca84ab4b658/vm.cfg:OVM_simple_name = 'ExalogicControlOpsCenterPC2

      In this example, the absolute paths of the Proxy Controller 1 and Proxy Controller 2 vServers respectively are as follows:

      /OVS/Repositories/0004fb00000300007d4eef3af41ca807/VirtualMachines/0004fb0000060000014361b5c6f404/vm.cfg

/OVS/Repositories/0004fb00000300007d4eef3af41ca807/VirtualMachines/0004fb0000060000cbf5bca84ab4b658/vm.cfg

    3. Start the Proxy Controller vServer by using the xm create command as follows:

      xm create absolute_path_to_vm.cfg

      Example:

      xm create /OVS/Repositories/0004fb00000300007d4eef3af41ca807/VirtualMachines/0004fb0000060000014361b5c6f404/vm.cfg

      In this example, you are starting the Proxy Controller 1 vServer.

  5. Verify whether the Proxy Controller vServer is up by doing the following:

    1. Log in to any Oracle VM Server node on the Exalogic rack as the root user.

    2. From the Oracle VM Server node, log in to the Proxy Controller vServer you started as the root user.

      Example:

      # ssh root@192.168.20.12

    3. Verify the Proxy Controller is online, by running the following command:

      [root@hostname-pc1 ~]# /opt/sun/xvmoc/bin/proxyadm status

    4. If the Proxy Controller is offline, run step 6 in Section A.1, "Recovering the Exalogic Control vServer From Hardware Failures."

  6. If you want to restart the other Proxy Controller vServer, repeat steps 4 and 5.