3.3 Oracle PCA 2.3 – Using the Oracle PCA Upgrader

UPGRADE BOTH MANAGEMENT NODES CONSECUTIVELY

With the Oracle PCA Upgrader, the two management node upgrade processes are theoretically separated. Each management node upgrade is initiated by a single command and managed through the Upgrader, which invokes the native Oracle VM Manager upgrade mechanisms. However, you must treat the upgrade of the two management nodes as a single operation.

During the management node upgrade, the high-availability (HA) configuration of the management node cluster is temporarily broken. To restore HA management functionality and mitigate the risk of data corruption, it is critical that you start the upgrade of the second management node immediately after a successful upgrade of the first management node.

NO MANAGEMENT OPERATIONS DURING UPGRADE

The Oracle PCA Upgrader manages the entire process to upgrade both management nodes in the appliance. Under no circumstances should you perform any management operations – through the Oracle PCA Dashboard or CLI, or Oracle VM Manager – while the Upgrader process is running, and until both management nodes have been successfully upgraded through the Upgrader. Although certain management functions cannot be programmatically locked during the upgrade, they are not supported, and are likely to cause configuration inconsistencies and considerable repair downtime.

Once the upgrade has been successfully completed on both management nodes, you can safely execute appliance management tasks and configuration of the virtualized environment.

As of Release 2.3.4, a separate command line tool is provided to manage the Controller Software update process. A version of the Oracle PCA Upgrader is included in the Controller Software .iso image. However, Oracle recommends that you download and install the latest stand-alone version of the Upgrader tool on the management nodes. The Oracle PCA Upgrader requires only a couple of commands to execute several sets of tasks, which were scripted or manual steps in previous releases. The new tool is more robust and easily extensible, and provides a much better overall upgrade experience.

A more detailed description of the Oracle PCA Upgrader is included in the introductory chapter of this book. Refer to Section 1.8, “Oracle Private Cloud Appliance Upgrader”.

3.3.1 Optional: Rebooting the Management Node Cluster

Although not strictly necessary, it may be useful to reboot both management nodes before starting the appliance software update. This leaves the management node cluster in the cleanest possible state, ensures that no system resources are occupied unnecessarily, and eliminates potential interference from processes that have not completed properly.

Rebooting the Management Node Cluster

  1. Using SSH and an account with superuser privileges, log into both management nodes using the IP addresses you configured in the Network Setup tab of the Oracle PCA Dashboard. If you use two separate consoles you can view both side by side.

    Note

    The default root password is Welcome1. For security reasons, you must set a new password at your earliest convenience.

  2. Run the command pca-check-master on both management nodes to verify which node owns the master role.

  3. Reboot the management node that is NOT currently the master. Enter init 6 at the prompt.

  4. Ping the machine you rebooted. When it comes back online, reconnect using SSH and monitor system activity to determine when the secondary management node takes over the master role. Enter this command at the prompt: tail -f /var/log/messages. New system activity notifications will be output to the screen as they are logged.

  5. In the other SSH console, which is connected to the current active management node, enter init 6 to reboot the machine and initiate management node failover.

    The log messages in the other SSH console should now indicate when the secondary management node takes over the master role.

  6. Verify that both management nodes have come back online after reboot and that the master role has been transferred to the other manager. Run the command pca-check-master on both management nodes.

    If this is the case, proceed with the software update steps below.

3.3.2 Installing the Oracle PCA Upgrader

The Oracle PCA Upgrader is a separate application with its own release plan, independent of Oracle PCA. Always download and install the latest version of the Oracle PCA Upgrader before you execute any verification or upgrade procedures.

Downloading and Installing the Latest Version of the Oracle PCA Upgrader

  1. Log into My Oracle Support and download the latest version of the Oracle PCA Upgrader.

    The Upgrader can be found under patch ID 28900934, and is included in part 1 of a series of downloadable zip files. Any updated versions of the Upgrader will be made available in the same location.

    To obtain the Upgrader package, download this zip file and extract the file pca_upgrader-<version>.el6.noarch.rpm.

  2. Copy the downloaded *.rpm package to the master management node and install it.

    [root@ovcamn05r1 ~]# pca-check-master
    NODE: 192.168.4.3  MASTER: True
    root@ovcamn05r1 tmp]# rpm -ivh pca_upgrader-1.0-98.el6.noarch.rpm
    Preparing..########################################### [100%]
    1:pca_upgrader########################################### [100%]
    Caution

    Always download and use the latest available version of the Oracle PCA Upgrader.

  3. If the version of the Oracle PCA Upgrader you downloaded, is newer than the version shipped in the Controller Software ISO, then upgrade to the newer version. From the directory where the *.rpm package was saved, run the command rpm -U pca_upgrader-1.0-98.el6.noarch.rpm.

  4. Repeat the *.rpm upgrade on the second management node.

    The Oracle PCA Upgrader verifies the version of the Upgrader installed on the second management node. Only if the version in the ISO is newer, the package on the second management node is automatically upgraded in the process. If you downloaded a newer version, you must upgrade the package manually on both management nodes.

3.3.3 Verifying Upgrade Readiness

The Oracle PCA Upgrader has a verify-only mode. It allows you to run all the pre-checks defined for a management node upgrade, without proceeding to the actual upgrade steps. The terminal output and log file report any issues you need to fix before the system is eligible for the next Controller Software update.

Tip

The Oracle PCA Upgrader pre-checks for the management nodes are equivalent to the pre-upgrade validation procedure required before the installation of Oracle PCA Controller Software Release 2.3.1, 2.3.2 and 2.3.3.

Note

The Oracle PCA Upgrader cannot be stopped by means of a keyboard interrupt or by closing the terminal session. After a keyboard interrupt (Ctrl+C) the Upgrader continues to execute all pre-checks. If the terminal session is closed, the Upgrader continues as a background process.

Verifying the Upgrade Readiness of the Oracle PCA

  1. Go to Oracle VM Manager and make sure that all compute nodes are in Running status. If any server is not in Running status, resolve the issue before proceeding. For instructions to correct the compute node status, refer to the support note with Doc ID 2245197.1.

  2. Perform the required manual pre-upgrade checks. Refer to Section 8.12, “Running Manual Pre- and Post-Upgrade Checks in Combination with Oracle PCA Upgrader” for instructions.

  3. Log in to My Oracle Support and download the required Oracle PCA software update.

    You can find the update by searching for the product name Oracle Private Cloud Appliance, or for the Patch or Bug Number associated with the update you need.

    Caution

    Read the information and follow the instructions in the readme file very carefully. It is crucial for a successful Oracle PCA Controller Software update and Oracle VM upgrade.

  4. Make the update, a zipped ISO, available on an HTTP or FTP server that is reachable from your Oracle PCA. Alternatively, if upgrade time is a major concern, you can download the ISO file to the local file system on both management nodes. This reduces the upgrade time for the management nodes, but has no effect on the time required to upgrade the compute nodes or the Oracle VM database.

    The Oracle PCA Upgrader downloads the ISO from the specified location and unpacks it on the management node automatically at runtime.

  5. Using SSH and an account with superuser privileges, log in to the master management node through its individually assigned IP address, not the shared virtual IP.

    Note

    During the upgrade process, the interface with the shared virtual IP address is shut down. Therefore, you must log in using the individually assigned IP address of the management node.

    The default root password is Welcome1. For security reasons, you must set a new password at your earliest convenience.

  6. From the master management node, run the Oracle PCA Upgrader in verify-only mode. The target of the command must be the stand-by management node.

    [root@ovcamn05r1 ~]# pca-check-master
    NODE: 192.168.4.3  MASTER: True
    root@ovcamn05r1 ~]# pca_upgrader -V -t management -c ovcamn06r1 -g 2.3.4 \
    -l http://<path-to-iso>/ovca-2.3.4-b000.iso.zip
    
    Please refer to log file /var/log/pca_upgrader.log for more details.
    
    Beginning PCA Management Node Pre-Upgrade Checks...
    Validate the Image Provided                        1/30
    Shared Storage Mounted Check                       2/30
    Oracle VM Minimum Version Check                    3/30
    [...]
    IB Symbol Errors Check                             26/30
    Network Configuration Check                        27/30
    OSA Disabled Check                                 28/30
    Server Update Configuration Check                  29/30
    OVM Warning/Error Check                            30/30
    
    ---------------------------------------------------------------------------
    PCA Management Node Pre-Upgrade Checks		Passed
    ---------------------------------------------------------------------------
    Validate the Image Provided                        Passed
    Shared Storage Mounted Check                       Passed
    Oracle VM Minimum Version Check                    Passed
    Server Name Check                                  Passed
    OS Check                                           Passed
    [...]
    IB Symbol Errors Check                             Warn
    Network Configuration Check                        Passed
    OSA Disabled Check                                 Passed
    Server Update Configuration Check                  Passed
    OVM Warning/Error Check                            Passed
    ---------------------------------------------------------------------------
    Overall Status                                     Passed
    ---------------------------------------------------------------------------
    
  7. As the verification process runs, check the console output for test progress. When all pre-checks have been completed, a summary is displayed. A complete overview of the verification process is saved in the file /var/log/pca_upgrader.log.

    Some pre-checks may result in a warning. These warnings are unlikely to cause issues, and therefore do not prevent you from executing the upgrade, but they do indicate a situation that should be investigated. When an upgrade command is issued, warnings do cause the administrator to be prompted whether to proceed with the upgrade, or quit and investigate the warnings first.

  8. If pre-checks have failed, consult the log file for details. Fix the reported problems, then execute the verify command again.

  9. Repeat this process until no more pre-check failures are reported. When the system passes all pre-checks, it is ready for the Controller Software update.

3.3.4 Executing a Controller Software Update

During a Controller Software update, the virtualized environment does not accept any management operations. After successful upgrade of the management node cluster, the compute nodes must also be upgraded in phases, and additional firmware upgrades on several rack components may be required. When you have planned all these upgrade tasks, and when you have successfully completed the upgrade readiness verification, your environment is ready for a Controller Software update and any additional upgrades.

No upgrade procedure can be executed without completing the pre-checks. Therefore, the upgrade command first executes the same steps as in Section 3.3.3, “Verifying Upgrade Readiness”. After successful verification, the upgrade steps are started.

Note

The Oracle PCA Upgrader cannot be stopped by means of a keyboard interrupt or by closing the terminal session.

After a keyboard interrupt (Ctrl+C) the Upgrader continues the current phase of the process. If pre-checks are in progress, they are all completed, but the upgrade phase does not start automatically after successful completion of all pre-checks. If the upgrade phase is in progress at the time of the keyboard interrupt, it continues until upgrade either completes successfully or fails.

If the terminal session is closed, the Upgrader continues as a background process.

Upgrading the Oracle PCA Controller Software

  1. Using SSH and an account with superuser privileges, log in to the master management node through its individually assigned IP address, not the shared virtual IP.

    Note

    During the upgrade process, the interface with the shared virtual IP address is shut down. Therefore, you must log in using the individually assigned IP address of the management node.

    The default root password is Welcome1. For security reasons, you must set a new password at your earliest convenience.

    NO MANAGEMENT OPERATIONS DURING UPGRADE

    Under no circumstances should you perform any management operations – through the Oracle PCA Dashboard or CLI, or Oracle VM Manager – while the Upgrader process is running, and until both management nodes have been successfully upgraded through the Upgrader.

  2. From the master management node, run the Oracle PCA Upgrader with the required upgrade parameters. The target of the command must be the stand-by management node.

    [root@ovcamn05r1 ~]# pca-check-master
    NODE: 192.168.4.3  MASTER: True
    root@ovcamn05r1 ~]# pca_upgrader -U -t management -c ovcamn06r1 -g 2.3.4 \
    -l http://<path-to-iso>/ovca-2.3.4-b000.iso.zip
    
    Please refer to log file /var/log/pca_upgrader.log for more details.
    
    Beginning PCA Management Node Pre-Upgrade Checks...
    [...]
    
    ***********************************************************************
    Warning: The management precheck completed with warnings. 
    It is safe to continue with the management upgrade from this point 
    or the upgrade can be halted to investigate the warnings. 
    ************************************************************************
    Do you want to continue? [y/n]: y

    After successfully completing the pre-checks, the Upgrader initiates the Controller Software update on the other management node. If any errors occur during the upgrade phase, tasks are rolled back and the system is returned to its original state from before the upgrade command.

    Rollback works for errors that occur during these steps:

    • downloading the ISO

    • setting up the YUM repository

    • taking an Oracle VM backup

    • breaking the Oracle PCA HA model

    Beginning PCA Management Node upgrade for ovcamn06r1
    Disable PCA Backups                              1/16
    Download ISO                                     2/16
    Setup Yum Repo                                   3/16
    Take OVM Backup                                  4/16
    Break PCA HA Model                               5/16
    Place PCA Upgrade Locks                          6/16
    Perform Yum Upgrade                              7/16
    Perform Oracle VM upgrade                        8/16
    Relink to Shared Storage                         9/16
    Install PCA UI                                   10/16
    Reboot                                           11/16
    Run Post-Upgrade Sync Tasks                      12/16
    Remove PCA Upgrade Locks                         13/16
    Cleanup ISO Backup Directories                   14/16
    Restore PCA Backups                              15/16
    Upgrade is complete                              16/16
    PCA Management Node upgrade of ovcamn06r1 completed after 74 minutes
    ---------------------------------------------------------------------------
    PCA Management Node Pre-Upgrade Checks           Passed
    --------------------------------------------------------------------------- 
    Validate the Image Provided                      Passed
    Shared Storage Mounted Check                     Passed
    [...]
    
    ---------------------------------------------------------------------------
    PCA Management Node Upgrade                      Passed
    ---------------------------------------------------------------------------
    Disable PCA Backups                              Passed
    Download ISO                                     Passed
    [...]
    Upgrade is complete                              Passed
    
    ---------------------------------------------------------------------------
    Overall Status                                   Passed
    ---------------------------------------------------------------------------
    PCA Management Node Pre-Upgrade Checks           Passed
    PCA Management Node Upgrade                      Passed
    PCA Management Node Post Upgrade Checks          No Automated Postchecks
    Tip

    When the ISO is copied to the local file system of both management nodes, the management node upgrade time is just over 1 hour each. The duration of the entire upgrade process depends heavily on the size of the environment: the number of compute nodes and their configuration, the size of the Oracle VM database, etc.

    If you choose to copy the ISO locally, replace the location URL in the pca-upgrader command with -l file:///<path-to-iso>/ovca-2.3.4-b000.iso.zip.

  3. Monitor the progress of the upgrade tasks. The console output provides a summary of each executed task. If you need more details on a task, or if an error occurs, consult the log file. You can track the logging activity in a separate console window by entering the command tail -f /var/log/pca_upgrader.log. The example below shows several key sections in a typical log file.

    Note

    Once the upgrade tasks have started, it is no longer possible to perform a rollback to the previous state.

    # tail -f /var/log/pca_upgrader.log
    
    [2018-11-01 12:40:16 391761] INFO (pca_upgrader:51) Starting PCA Upgrader...
    [2018-11-01 12:40:16 391761] DEBUG (util:411) the dlm_locks command output is debugfs.ocfs2 1.8.6
    [?1034hdebugfs: dlm_locks -f /sys/kernel/debug/o2dlm/ovca/locking_state
    Lockres: master                            Owner: 0    State: 0x0 
    Last Used: 0      ASTs Reserved: 0    Inflight: 0    Migration Pending: No
    Refs: 3    Locks: 1    On Lists: None
    Reference Map: 1 
     Lock-Queue  Node  Level  Conv  Cookie           Refs  AST  BAST  Pending-Action
     Granted     0     EX     -1    0:1              2     No   No    None
    
    debugfs: quit
    [2018-11-01 12:40:16 391761] INFO (util:420) This node (192.168.4.3) is the master
    [2018-11-01 12:40:16 391761] DEBUG (pca_upgrader:75) To verify upgrade readiness of the rack.
    [2018-11-01 12:40:16 391761] DEBUG (run_util:15) Writing 391854 to /var/run/ovca/upgrader.pid
    [2018-11-01 12:40:16 391854] DEBUG (process_flow:31) Execute precheck steps for component: management
    [2018-11-01 12:40:18 391854] INFO (precheck_management:68) [Validate the Image Provided
    (Verify that the image exists and is the correct filetype for upgrade)] Passed 
    [2018-11-01 12:40:18 391854] INFO (precheck_management:68) [Shared Storage Mounted Check
    (Checking shared storage MGMT_ROOT, Yum, and Incoming are mounted on the management nodes)] Passed 
    The check succeeded on the management nodes.
    
    [2018-11-01 12:40:22 391854] DEBUG (ovmSession:211)
    ------------------------Entering getServerInfo--------------------------
    [2018-11-01 12:40:23 391854] INFO (ovm:103) Calling ovmwsh script [get_server_info.py]
    with args ('--server=e0:31:8d:9a:00:10:ff:ff:ff:ff:ff:ff:ff:20:00:08',)
    [2018-11-01 12:40:32 391854] INFO (ovm:145) ovmwsh script finished with success
    STDOUT: Looking for [server] [e0:31:8d:9a:00:10:ff:ff:ff:ff:ff:ff:ff:20:00:08]
    [...]
    
    [2018-11-01 12:42:20 391854] INFO (precheck_management:68)[OS Check (Checking the
    management nodes and compute nodes are running the correct Oracle Linux version)] Passed 
    The check succeeded on the management nodes. The check succeeded on all the compute nodes.
    [...]
    
    [2018-11-01 12:42:25 391854] INFO (precheck_utils:421) Checking storage...
    Checking server pool...
    Checking pool filesystem...
    [...]
    
    [2018-11-01 12:45:53 391854] INFO (precheck_utils:1305) Checking the existence of IB symbol errors.
    [2018-11-01 12:45:56 391854] WARNING (precheck_utils:1311)
    WARNING: Found the following IB symbol errors:
    Errors for "ovcasw22r1"
       GUID 0x139702010013ac port 1: [VL15Dropped == 3164]
    Errors for "ovcasw15r1"
       GUID 0x139702010013ec port 1: [VL15Dropped == 9]
    Errors for 0x2128f569d8a0a0 "SUN DCS 36P QDR nm2-36p-a 192.168.4.202"
       GUID 0x2128f569d8a0a0 port ALL: [LinkDownedCounter == 6] [PortRcvErrors == 1]
       [PortRcvSwitchRelayErrors == 36] [PortXmitDiscards == 247] [VL15Dropped == 1]
       [PortXmitWait == 17742]
       GUID 0x2128f569d8a0a0 port 0: [PortXmitWait == 127]
    [...]
    
    [2018-11-01 12:47:41 391854] INFO (precheck_management:68) [OVM Warning/Error Check
    (Checking the existence of OVM warnings or errors)] Passed 
    The check succeeded. No OVM warnings or errors found.
    [2018-11-01 12:47:41 391854] INFO (precheck_management:75) 
    
    ****** Management Node Pre-Upgrade Check Summary ******
    [...]
    
    [2018-11-01 12:59:11 397738] DEBUG (process_flow:92)
    Successfully completed precheck. Proceeding to upgrade.
    
    [2018-11-01 12:59:11 397738] DEBUG (process_flow:31)
    Execute upgrade steps for component: management
    [2018-11-01 12:59:11 397738] INFO (rack_utils:24) Querying rpm for pca_upgrader details
    [2018-11-01 12:59:11 397738] INFO (rack_utils:26)
    Name        : pca_upgrader                 Relocations: (not relocatable)
    Version     : 1.0                               Vendor: Oracle America
    Release     : 89.el6                        Build Date: Wed Oct 17 15:52:09 2018
    Install Date: Thu Nov  1 12:37:53 2018      Build Host: x86-ol6-builder-04.us.oracle.com
    Group       : System Environment/Daemons    Source RPM: pca_upgrader-1.0-89.el6.src.rpm
    Size        : 545617                           License: GPL
    Signature   : RSA/8, Wed Oct 17 15:52:17 2018, Key ID 72f97b74ec551f03
    URL         : http://www.oracle.com/virtualization
    Summary     : PCA upgrader
    Description : Private Cloud Appliance Upgrader.
    
    [2018-11-01 12:59:11 397738] INFO (rack_utils:43)
    Setting pca_upgrader version to 1.0-89.el6 for shared script execution
    [2018-11-01 12:59:11 397738] DEBUG (mn_upgrade_utils:1144)
    Remove lock file: /nfs/shared_storage/pca_upgrader/locks/FAIL
    [2018-11-01 12:59:11 397738] DEBUG (mn_upgrade_steps:68)
    Successfully completed remove_upgrade_status_flag
    [2018-11-01 12:59:11 397738] DEBUG (mn_upgrade_steps:602)
    Successfully completed update_scripts
    [2018-11-01 12:59:11 397738] DEBUG (mn_upgrade_utils:1209)
    Calling service crond status on ovcamn05r1
    [2018-11-01 12:59:11 397738] INFO (mn_upgrade_steps:84)
    Stopping crond for [ovcamn05r1] to prevent PCA backups during the upgrade process
    [...]
    
    [2018-11-01 14:05:22 397738] DEBUG (update:489) Trying to backup MGMT image.
    [2018-11-01 14:05:22 397738] DEBUG (update:490) /nfs/shared_storage/mgmt_image.old
    [2018-11-01 14:05:22 397738] DEBUG (update:527)
    Installing new OS image: /tmp/tmpHIAtLl/ovca-2.3.4-b000.iso.zip
    [2018-11-01 14:08:43 397738] INFO (mn_upgrade_steps:242) Successfully downloaded ISO
    [2018-11-01 14:08:54 397738] DEBUG (mn_upgrade_steps:283) Successfully completed setup_mn_yum_repo
    [2018-11-01 14:08:54 397738] DEBUG (mn_upgrade_utils:213)
    Shutting down Oracle VM services before taking the Oracle VM backup
    [...]
    
    [2018-11-01 14:13:32 397738] INFO (mn_upgrade_utils:479) Beginning Yum Upgrade
    [2018-11-01 14:13:32 397738] INFO (mn_upgrade_utils:484) Output from the Yum upgrade will not be
    posted to this logfile in real time, but instead will be posted once the yum upgrade has completed.
    To view what yum is doing in real time on ovcamn06r1, login to ovcamn06r1 and tail /var/log/messages
    Removing obsolete packages prior to upgrade
    Beginning Yum upgrade
    Setting up Upgrade Process
    Resolving Dependencies
    [...]
    Transaction Summary
    ================================================================================
    Install       5 Package(s)
    Upgrade     265 Package(s)
    
    Total download size: 386 M
    Downloading Packages:
    --------------------------------------------------------------------------------
    [...]
    
    Oracle VM upgrade script finished with success
    STDOUT: Oracle VM Manager Release 3.4.5 Installer
    Oracle VM Manager Installer log file:
    /var/log/ovmm/ovm-manager-3-install-2018-11-01-142102.log
    Verifying upgrading prerequisites ...
    Starting Upgrade ...
    Upgrading from version 3.4.4.0000 to version 3.4.5.0000
    Running full database backup ...
    [...]
    Installation Summary
    --------------------
    Database configuration:
      Database type               : MySQL
      Database host name          : localhost
      Database name               : ovs
      Database listener port      : 1521
      Database user               : ovs
    Weblogic Server configuration:
      Administration username     : weblogic
    Oracle VM Manager configuration:
      Username                    : admin
      Core management port        : 54321
      UUID                        : 0006c9000000100055d6f23a1e992277
    [...]
    
    STDOUT: Looking for [tenant group] [Rack1_ServerPool]
    Looking for [storage array] [OVCA_ZFSSA_Rack1]
    Finding the LUN that is used as the heartbeat device in tenant group by page83id.
    [...]
    
    Successfully completed PCA management node post upgrade tasks

    When the upgrade tasks have been completed successfully, the master management node is rebooted, and the upgraded management node assumes the master role. The new master management node's operating system is now up-to-date, and it runs the new Controller Software version and upgraded Oracle VM Manager installation.

    Tip

    Rebooting the management node is expected to take up to 10 minutes.

    To monitor the reboot process and make sure the node comes back online as expected, log in to the rebooting management node ILOM.

    Broadcast message from root@ovcamn05r1 (pts/2) (Thu Nov  1 15:00:00 2018):
    
    Management Node upgrade succeeded. The master manager will be rebooted to initiate failover in one minute.
    
    Broadcast message from root@ovcamn05r1
    (/dev/pts/2) at 15:01 ...
    
    The system is going down for reboot NOW!
    Please refer to log file /var/log/pca_upgrader.log for more details.
  4. Log into the upgraded management node, which has now become the master management node. Use its individually assigned IP address, not the shared virtual IP.

    [root@ovcamn06r1 ~]# pca-check-master
    NODE: 192.168.4.4  MASTER: True
    
    [root@ovcamn06r1 ~]# head /etc/ovca-info
    ==== Begin build info ====
    date: 2018-11-01
    release: 2.3.4
    build: 000
    === Begin compute node info ===
    compute_ovm_server_version: 3.4.5
    compute_ovm_server_build: 2.3.4-000
    compute_rpms_added:
    osc-oracle-s7k-2.1.2-4.el6.noarch.rpm
    ovca-support-2.3.4-118.el6.noarch.rpm
    NO MANAGEMENT OPERATIONS DURING UPGRADE

    Under no circumstances should you perform any management operations – through the Oracle PCA Dashboard or CLI, or Oracle VM Manager – while the Upgrader process is running, and until both management nodes have been successfully upgraded through the Upgrader.

  5. From the new master management node, run the Oracle PCA Upgrader command again. The target of the command must be the stand-by management node, which is the original master management node from where you executed the command for the first run.

    root@ovcamn06r1 ~]# pca_upgrader -U -t management -c ovcamn05r1 -g 2.3.4 \
    -l http://<path-to-iso>/ovca-2.3.4-b000.iso.zip
    
    Please refer to log file /var/log/pca_upgrader.log for more details.
    Beginning PCA Management Node Pre-Upgrade Checks...
    [...]
    
    ***********************************************************************
    Warning: The management precheck completed with warnings. 
    It is safe to continue with the management upgrade from this point 
    or the upgrade can be halted to investigate the warnings. 
    ************************************************************************
    Do you want to continue? [y/n]: y
    Beginning PCA Management Node upgrade for ovcamn05r1
    [...]
    
    ---------------------------------------------------------------------------
    Overall Status                                   Passed
    ---------------------------------------------------------------------------
    PCA Management Node Pre-Upgrade Checks           Passed
    PCA Management Node Upgrade                      Passed
    PCA Management Node Post Upgrade Checks          No Automated Postchecks
    
    Broadcast message from root@ovcamn06r1 (pts/1) (Thu Nov  1 18:00:00 2018):
    
    Management Node upgrade succeeded. The master manager will be rebooted to initiate failover in one minute.
    
    Broadcast message from root@ovcamn05r1
    (/dev/pts/1) at 18:01 ...
    
    The system is going down for reboot NOW!
    Please refer to log file /var/log/pca_upgrader.log for more details.

    The upgrade steps are executed the same way as during the first run. When the second management node is rebooted, the process is complete. At this point, both management nodes run the updated Oracle Linux operating system, Oracle PCA Controller Software, and Oracle VM Manager. The high-availability cluster configuration of the management nodes is restored, and all Oracle PCA and Oracle VM Manager management functionality is operational again. However, do not perform any management operations until you have completed the required manual post-upgrade checks.

  6. Perform the required manual post-upgrade checks on management nodes and compute nodes. Refer to Section 8.12, “Running Manual Pre- and Post-Upgrade Checks in Combination with Oracle PCA Upgrader” for instructions.

When the management node cluster upgrade is complete, proceed with the following tasks: