6 Patching OMCe

This section describes how to apply a patch to the services comprising an OMCe stack, and roll back the patch as necessary.

You can patch an OMCe stack from the Service Details page of the Service Manager, or by using the Patch Tool which is a command line script. Patching occurs at the level of individual services, but because OMCe is a stack all services within it must be patched at the same time. You cannot patch just some of the services comprising an OMCe stack.

Download the Patch Tool

The Patch Tool is part of the Admin tools zip which you can get from the OMCe Downloads page on OTN at http://www.oracle.com/technetwork/topics/cloud/downloads/mobile-suite-3636471.html. Save the zip to a local location and unzip it.

You must have Python 3 installed on the same machine from where you will run the patch. You can get it from https://www.python.org/downloads/.

Note:

The OMCe Admin tools use the executable python. If the version of Python 3 you have has the python3 executable you should either modify the patch.sh/patch.cmd scripts to use python3, or symlink python to the python3 executable.

On OSX, if you get an error SSL: CERTIFICATE_VERIFY_FAILED when you run the tool it is probably because Python 3.6 on OSX has no certificates and can’t validate any SSL connections. You can install the certifi package of certificates. See the ReadMe at /Applications/Python\ 3.6/ReadMe.rtf .

OMCe Service Instances

The OMCe stack is comprised of a number of services which you can see in the Stack Overview page for your OMCe stack. The name of the service has the stack name as the first part of the service name:

• {stack}CORE

• {stack}CCC

• {stack}BOTSCFG

• {stack}BOTSCON

• {stack}BOTSINT

• {stack}BOTSPIP

• {stack}BOTSMGM

• {stack}OMCE

• {stack}CXACFG

• {stack}CXAPOD

• {stack}1CXAANA

• {stack}1CXACOL

Typical Workflow for Patching an OMCe Stack

Consider the typical workflow for patching an OMCe stack, as described in the following table.

Task	Description	More Information
Learn about approved patches	View approved patches displayed on the Patching page periodically.	Viewing Patch Details
Perform optional prechecks	Learn about problems that would cause patching to fail, so you can address those problems before you try to apply a patch.	Performing a Patch Precheck
Apply a patch	Initiate a patching operation to update the service instances which comprise and OMCe stack.	Patching OMCe Services
Roll back a patch	Initiate a rollback operation to return a service to its previous patch level.	Rollback a Patch with the PaaS CLI

Configuring the Patch Script

If you just run the patch script, you will be prompted several times to enter information about your OMCe log in and IDCS instance. You can avoid this by using patch.json, which is provided as part of the OMCe Admin Tools. You may want to leave the value for psmPassword blank, in which case you will be prompted for the password when the script is run.

The default patch.json is:

{
  "psmUser": "<OMCe admin user> ",
  "psmPassword": "<password for OMCe admin user>",
  "psmUri" : "<URI for PSM Stack Manager>",
  "psmIdentityDomain" : "<IDCS domain>"
}

The URI for the PSM Stack Manager has the form https://region-prefix.oraclecloud.com, where the region-prefix is the URL prefix based on region. Typical values include:

• United States: psm.us

  For example: https://psm.us.oraclecloud.com

• Europe: psm.europe

  For example: https://psm.europe.oraclecloud.com

• Australia: psm.aucom

  For example: https://psm.aucom.oraclecloud.com

Viewing Patch Details

Use the Patching page to view information about the available patches, as well as your patch history.

1. From the Instance Overview page, use the Administration tile to view the number of available patches.
2. If any patches are available, click the Administration tile.
3. Click the Patching tab.
4. In the Available Patches section click the icon for a patch.

   An overlay displays with more details about the patch.

You can also use the patch script to check whether the stack is ready to be patched. Use:

./patch.sh -s {stackName} patch_precheck

The response will list the services and the stack is ready to patch when the status for all services is Ready to apply patch.

Before applying any patches, Oracle recommends that you perform a precheck. See Performing a Patch Precheck.

Performing a Patch Precheck

You can optionally perform a precheck operation before applying a patch in order to identify and remedy any issues.

The precheck operation helps detect potential patching problems such as:

• Disk space shortage

• Missing database connectivity

• Servers not running

• Storage access failure

The precheck operation does not check whether another administration task (backup, restoration, or scaling) is in progress, which prevents patching.

1. Access your service console.
2. Click the name of the service instance to which you want to apply a patch.
3. Click the Administration tile.
4. Click the Patching tab.
5. Beside the patch that you want to precheck, click PSU , and then select Precheck.
6. To determine whether the prechecks are completed, periodically click Refresh .

   The Precheck results link is displayed.

   • If prechecks pass, a green check mark icon is displayed.

   • If prechecks fail, a red exclamation mark icon is displayed.

7. Click Precheck results.
8. If there are precheck failures, address the issues, and then run Precheck again.

After addressing any precheck failures, you are ready to apply the patch. See Patching OMCe Services.

Upgrading from 17.4.5 to 18.1.1

If you have previously patched from 17.4.3 to 17.4.5, then before you patch to 18.1.1 there is an additional step you need to perform. You have to connect to the Core VM using ssh and run a script, and then run the patch.

Note:

This only applies to OMCe instances that were provisioned on release 17.4.3 and have been patched to 17.4.5. If you are patching directly from 17.4.3 to 18.1.1 you don’t need to run the pre-patch script.

How do I check whether my OMCe instance may be affected?

If you are unsure whether your OMCe instance was initially provisioned on 17.4.3 and patched to 17.4.5, you can check the patching and rollback history.

1. Navigate to the Patching page. Click Administration tab on the left, then select Patch.

2. Expand Patch and Rollback History. If you can see that the 17.4.5 patch has been applied then your OMCe instance was provisioned at 17.4.3 and you need to perform this step.

How to run the pre-patch script

1. Find the Core VM’s IP address.

   1. From OMCe Stack Overview, click the resource with the name <stack>CORE. See View Your OMCe Stack.

   2. In the Service Overview for the Mobile Core POD Details, expand Resources and note the Public IP, for example 192.0.2.1.

2. Connect to the Core VM using ssh and change to the oracle user. You can log in as the default user, opc. The opc user has sudo privileges.

   1. Use:

      $ ssh -i <private_key_file> opc@<Core IP address>

      For example, ssh -i my_private_rsa_keyfile opc@192.0.2.1

   2. Switch to the oracle user using:

      $ sudo su - oracle

3. Save the script below as prepatch_omce_18_1_1.sh in a convenient location, give exec permissions for current user, and run the script.

4. Check that the output looks correct. It should look similar to this:

   Updating: /u01/data/domains/<domain-name>/init-info/domain-info.xml
   Updating: /u01/data/domains/<domain-name>/init-info/config-path.properties
   Updating: /u01/data/domains/<domain-name>/init-info/startscript-unsub.xml

   If it is, you can continue to patch to 18.1.1.

The pre-patch script

Save this as prepatch_omce_18_1_1.sh in a suitable location.

#!/bin/bash

_domain_configs="domain-info.xml config-path.properties startscript-unsub.xml"

_domain_init_info_dir=$(find "/u01/data/domains" -type d -name "init-info")

for f in ${_domain_configs}; do
  _fpath="${_domain_init_info_dir}/${f}"
  _backup="${_fpath}.bak"
  if [ -f "${_backup}" ]; then
    echo "Skipping ${f} because its backup already exists at ${_backup}"
    continue
  fi
  echo "Updating: ${_fpath}"
  sed -i.bak "s/17.4.3.0.0/17.4.5.0.0/g" "${_fpath}" 
done

Patching OMCe Services

Do not apply a patch to service instances:

• If one or more of the service instance has any ongoing maintenance operations, such as backup, restoration or scaling

• If any database deployment associated with one or more of the service instances is undergoing a patching operation

Patch All OMCe Services

Oracle recommends that you use the patch script to patch all the services in an OMCe stack. Use the command:

./patch.sh -s {stackName} patch

 If you accidentally exit the script before it has completed don't panic.  The patch operations that are currently in progress will continue.  Simply make the following call to check status a few times until you see no running activities on any of the service instances.

./patch.sh -s {stackName} status

Once you see that there are no running instances, re-run the patch command.  The patching command will not modify service instances that have already been patched, and will resume patching where it is required.

Patch Individual OMCe Services

You can patch services one by one. There is a recommended order in which the OMCe components should be patched, and within the components, the order in with the services should be patched. Use this list to guide you through the order of applying patches.

Note:

Oracle recommends that you use the patch script to patch all the services in an OMCe stack in one go. This information is provided in case there is a problem with the script and you are advised to patch services one by one.

1. Core

   1. {stack}CORE

   2. {stack}CCC

2. Bots

   1. {stack}BOTSCFG

   2. {stack}BOTSCON

   3. {stack}BOTSINT

   4. {stack}BOTSPIP

   5. {stack}BOTSMGM

   6. {stack}OMCE

3. Analytics

   1. {stack}CXACFG

   2. {stack}CXAPOD

   3. {stack}1CXAANA

   4. {stack}1CXACOL

Applying a Patch Using the Patch Script

Patch an individual service using:

./patch.sh -t {stackType} -n {serviceName} patch

Applying a Patch from the Service Details Page

If you do not want to use the patch script you can use the Patching page of the Service Details page for the service you want to patch.

1. Navigate to the Patching page. Click Administration tab on the left, then select Patch.

2. Click to the right of the patch you want to apply and select Patch. The Patch Service dialog is displayed.

3. In the Notes field enter a description for this patching operation. You must enter a note. If you do not, an error message will be displayed and you will not be able to patch.

4. Click Patch.

   The patching operation begins, and the patches are applied in a rolling fashion, having minimal impact on the functioning of the service.

   • A message stating that a patch job has been submitted for the service is displayed in the upper left corner of the console.

   • The Administration tile displays a message about the version of patch being applied. It also shows the time of the backup that occurred before the patching operation started.

   • The menu for this patch and all other available patches are disabled.

   You can follow the progress of the patch by clicking the Overview tab. Use the at the upper right to refresh the in-progress operation messages.

Upgrading Bots Samples After Patching

If you are patching from 17.4.3 to 17.4.5, you will have to recreate the sample of financial Bots. The new financial bot will work with the 17.4.5 samples container.

1. After patching your system, go to the Sample Bot container.

2. Select the Recreating Sample Bots tab and download financialBot.json.

3. In the patched environment, import the downloaded financialBot.json.

Troubleshoot Patch Issues

Sometimes patching goes wrong. When the patch has been unsuccessful rollback is automatically performed to restore the service instance or instances to their original state. But sometimes there are issues where you need to intervene to resolve issues so that you can re-attempt the patch.

Troubleshoot Using the Patch Script

The patch script writes logging information to a file called patch.log which is in the same directory as the patching script. patch.log contains information you will find useful when you are diagnosing the problem that has caused the patch failure.

You can gather even more information by re-running the patch operation specifying a log level of "DEBUG". Use:

./patch.sh -t {serviceType} -n {serviceName} -ll DEBUG patch

Troubleshoot from the Patching Page

You can use information on the Patching Page of the Service Details page for the service to diagnose why a patch or a rollback operation has failed to complete successfully regardless of whether you ran the script from the patch script or from the Service Details page.

1. Navigate to the Patching page. Click Administration tab on the left, then select Patch. Failed operations have a red triangle on the icon.

2. Click the icon to show details.

   
   Description of the illustration patch-fail-details.png

   Keep a note of the Job ID in case you have to contact support for assistance.

Rolling Back a Patch

Use the patch script or the Patching page to roll back a patch that you previously applied.

If you have to roll back a patch, you do this by rolling back individual services so that they are all rolled back to the same version.

Note:

You must ensure that all services are patched or rolled back to the same version for OMCe to function correctly. This means that if one of the services fails to upgrade, all services that have been patched must be rolled back.

Rollback Using the Patch Script

Use the following commands to rollback an OMCe stack or an individual stack.

Check stack is ready to have rollback applied:

./patch.sh -s {stackName} rollback_precheck

Check service is ready to have rollback applied:

./patch.sh -t {serviceType} -n {serviceName} rollback_precheck

Rollback the OMCe stack:

./patch.sh -s {stackName} rollback

Rollback an individual service:

./patch.sh -t {serviceType} -n {serviceName} rollback

Rollback From the Service Details Page

Use the Patching page of the Service Details to rollback one service at a time. Rollback the services in reverse order that the patches were applied. See Patch Individual OMCe Services.

1. Navigate to the Patching page. Click Administration tab on the left, then select Patch.

2. Click to the right of the patch you want to apply and select Rollback.