Upgrading your OSM Cloud Native Deployment

Overview of the Upgrade Steps

This sections outlines the upgrade steps:

Download the OSM 7.5.0 CNTK version.
Install or upgrade WebLogic Kubernetes Operator (WKO). See "Installing WebLogic Kubernetes Operator".
Decide on the Ingress Controller. See "Ingress Controller".
Build the OSM, DB Installer, OSMGW, and RTUX images, using the same OSM cloud native version as the toolkit. See "Creating OSM Cloud Native Images" for building the images.
Updating the specification files. See "Updating Specification Files".
Upgrading to OSM cloud native 7.5.0. See "Upgrading to OSM Cloud Native 7.5.0".

Installing WebLogic Kubernetes Operator

This section provides information about installing WebLogic Kubernetes Operator.

WKO Monitoring Mechanism

Once WKO is installed in a Kubernetes cluster, it needs to know which namespaces will have OSM instances. Of interest are two mechanisms:

List: WKO is given an explicit list of namespaces. This is done by upgrading the WKO helm chart to put in a new value for the list (one that takes the existing value and either adds a new namespace or removes one of the namespaces). This mechanism is supported in WKO 3.x and 4.x and is the mechanism in OSM 7.4.1 CNTK as well (upto 7.4.1.0.13). With this, the user needs RBAC access to the WKO namespace, which is likely administratively problematic in a large shared environment.
Label: When WKO is installed, it is given a label to look for in namespaces. Any namespace in the cluster that has that label is monitored. Adding or removing that label from a namespace has the effect of adding or removing that namespace from WKO's monitoring. This mechanism is supported in WKO 4.x. With this, the user needs RBAC access only to the OSM project namespace and not to WKO namespace.

For OSM cloud native, the recommendation is to use the Label mechanism for WKO 4.x and especially for WKO 4.0.8 and newer.

Operator Installation

Before starting with the procedure in this section, refer to OSM Compatibility Matrix for release 7.5.0 for the list of supported WKO versions. It is mandatory to have WKO installed with label-based monitoring mechanism for OSM 7.5.0.

If your existing WebLogic Kubernetes Operator satisfies the OSM 7.5.0 version requirements and uses the label-based monitoring mechanism, no action is required. Skip to the "Ingress Controller" section.

If you need to upgrade, it is recommended to install the new WKO into a new namespace. As individual namespaces are deregistered from old WKO and registered to new WKO, eventually, the old WKO will no longer be managing any namespaces. It can then be safely uninstalled.

Refer to Weblogic Kubernetes Operator documentation for operator installation at: https://oracle.github.io/weblogic-kubernetes-operator/managing-operators/
During installation, it is strongly recommended to choose a specific label that this version of WKO must look for, rather than relying on the default label. Relying on the default label causes problems when multiple WKOs need to be installed for any reason (such as during a phased WKO upgrade).
To provide the custom label, use the domainNamespaceLabelSelector parameter as shown below.

It is recommended that the custom label you select is descriptive and unique to this WKO. In the example below, this is achieved by specifying the full WKO version.

# Eg: For installing WKO version 4.1.2 with a custom label wlsko412=enabled, use below after configuring the right helm repo
$ helm install $WLSKO_NS \
 weblogic-operator/weblogic-operator \
 --namespace $WLSKO_NS \
 --version 4.1.2 \
 --set "domainNamespaceLabelSelector=wlsko412\=enabled"

Unregistering and Registering the Namespace with Weblogic Operator

Using your older CNTK, unregister your namespace forcefully with operator using "unregister-namespace.sh" script

# Set required variables with earlier CN values (before upgrade)
$ export OSM_CNTK=<osm_7.4.1_cntk>
$ export WLSKO_NS=<operator_namespace_before_upgrade>
$ export WLSKO_HOME=<path_for_operator_home>
$ $OSM_CNTK/scripts/unregister-namespace.sh -p $PROJECT -t wlsko -f

Install/upgrade your operator by following weblogic operator documentation.

Register OSM namespace with recent weblogic operator using "register-namespace.sh" script.

# Set required variables with OSM CN 7.5.0 values
$ export OSM_CNTK=<osm_7.5.0_cntk>
$ export WLSKO_NS=<operator_namespace_before_upgrade>
$ $OSM_CNTK/scripts/register-namespace.sh -p $PROJECT -t wlsko -l <custom-label>

Ingress Controller

OSM cloud native supports any annotations based ingress controller.

If you were using Traefik ingress controller earlier, you can continue to use it. However, support for Traefik is deprecated in favour of ingress controllers that support the generic Kubernetes ingress API. See "Working with Ingress, Ingress Controller, and External Load Balancer" for more details. If you wish to transition to such an Ingress Controller, it is recommended to remove the existing Traefik ingress objects (delete-ingress.sh) and unregister the namespace from Traefik (unregister-namespace.sh -t traefik) before beginning the upgrade process. As part of upgrading, add the required annotations for the Ingress object to your 7.5 specification files, ensure Ingress objects in the namespace will be monitored by the replacement Ingress Controller and create the new Ingress (create-ingress.sh with the 7.5 toolkit.

Updating Specification Files

This section focuses on the delta in the specification files between OSM 7.4.1 and OSM 7.5.0 releases.

It is strongly recommended that you begin by copying over the sample specification files from the OSM 7.5.0 CNTK. You can then re-make the customizations from your current specification files one by one.

Updating the Project Specification

This section describes the sections that you update in the project specification:

OSM cloud native 7.5.0 adds new capabilities (such as OSMGW and RTUX microservices) for which there are corresponding sections in the project and instance specification files. See to "Creating OSM Cloud Native Images" for details.
The default value for Ingress Controller has been set to GENERIC. If you wish to retain Traefik, change this to TRAEFIK.
If you use GENERIC ingress controller, the required annotations need to be added to the "ingress" section. In the project specification file, ingress.annotations lists the required annotations if you are using Nginx ingress controller. Adjust the annotations based on the ingress controller that you use.
In OSM 7.4.1 cloud native, there is a separate step to deploy WME WAR file to the instance as a custom archive. In 7.5.0 cloud native, WebLogic Monitoring Exporter can be enabled and disabled by changing the value of enabled in the instance.yaml. By default, it is set to true, which creates a sidecar container for WME using the default image for the version of WebLogic Operator in use. If you want to use a different image than the one that WKO uses, you have a provision to do that by updating WME in the project specification file.
There are new capabilities added for deploying cartridges. OSM cloud native cartridge deployment supports container image built with par file. Either the url or the Image field should be populated for a particular cartridge but not both. See "Cartridge par Sources" for more details.
Update the requiredTargetSystems section if your cartridge needs to integrate with external systems using REST API.
Defining osmWLSTargetNodes restricts all OSM cloud native WebLogic pods and DB Installer pods to worker nodes, that match the label conditions and for these pods and it will take precedence over osmcnTargetNodes.
The remaining sections for which there are no changes from OSM cloud native 7.4.1 need to be copied as-is to 7.5.0 based on your usage of the respective functionality in OSM cloud native 7.5.0.

Updating the Instance Specification

This section describes the sections that you update in the instance specification:

If you are planning to use log masking functionality, uncomment and use the logMaskingCustomRegexes section.
OSM 7.5.0 supports fine graining the log level for OSM core using the instance specification file. Uncomment and use the log section for overriding the default log level.
WebLogic Monitoring Exporter can be enabled and disabled by changing the value of weblogicMonitoringExporter.enabled in the instance.yaml file. By default, this is set to true.
OSM 7.5.0 introduces support for FluentD as a sidecar container for reading the logs from OSM Servers and DB Installer. Enable the fluentdLogging section for making use of this functionality. See "Configuring Fluentd Logging" for more details.
The storage volume must specify "pvc" to be used for persistent storage or as "emptydir" to share with a custom sidecar which is enabled via "sidecar.enabled" in instance specification file.
```
storageVolume:
  enabled: true
  type: pvc # Acceptable values are pvc and emptydir
  volumeName: storage-volume
  pvc: storage-pvc # Specify this only in case type is pvc
```
If you want to enable a sidecar other than FluentD, make use of the section sidecar.enabled.
For enabling SSL using generic Ingress, add the annotations required for OSM. The ssl.ingress.annotations section in the instance specification file lists the sample annotations required for OSM if you are using Nginx ingress controller. Add the annotations based on the ingress controller that you choose.
In the 7.4.1 cloud native, instance.yaml file contains the loadBalancerPort section. In OSM 7.5.0 cloud native, it is renamed to inboundGateway. As part of the inboundGateway section, provide the host and port details of the actual ingress point or the loadbalancer.
OSM cloud native 7.5.0 introduces support for Host Aliases with which you can achieve hostname resolution inside the pods. Refer to the hostAliases element and its comments in the instance specification for more details.
Update the osm-gateway and osmRuntimeUX sections as per "Configuring Target Systems for Events and System Interactions".
If you have defined targetSystems in the project specification file, provide the details of the respective target system in the instance specification file.
Copy the remaining sections for which there are no changes from OSM cloud native 7.4.1 as-is to 7.5.0 based on your usage of the respective functionality in OSM cloud native 7.5.0.

Updating Shape Specification

If you have a custom shape specification, identify the standard shape on which your custom shape is based. Make a copy of this standard shape from the OSM 7.5.0 CNTK and make your customization one-by-one to it.

Upgrading to OSM Cloud Native 7.5.0

Oracle recommends that upgrade the existing OSM 7.4.1 cloud native application to 7.5.0 cloud native using the same project namespace and the instance name.

Prerequisites for the Upgrade

Check OSM Compatibility Matrix for details about the required and supported versions of the pre-requisite software, before proceeding with the upgrade.

Note:

Perform the following steps using the existing 7.4.1 Cloud Native toolkit specification files.

Delete the Instance

To delete an instance, run the following:

$OSM_CNTK/scripts/delete-instance.sh -p project -i instance

Delete the Ingress

To delete an ingress, run the following:

$OSM_CNTK/scripts/delete-ingress.sh -p project -i instance

If you are switching from Traefik as part of this upgrade, to stop Traefik from monitoring this namespace, run the following:

$OSM_CNTK/scripts/unregister-namespace.sh -t traefik -p project

Preparation Steps for the Upgrade

This section lists preparatory steps for the upgrade:

Take a backup of the working OSM 7.4.1 project and instance specification files or rely on your source code management system to maintain versioning.
Take the OSM 7.4.1 DB schema backup before the upgrade.

Take a backup of the following secrets and also if there are any custom OSM secrets created for 7.4.1 cloud native before the upgrade.

<project>-<instance>-database-credentials
<project>-<instance>-embedded-ldap-credentials
<project>-<instance>-weblogic-credentials
<project>-<instance>-rcudb-credentials
<project>-<instance>-opss-wallet-password-secret
<project>-<instance>-runtime-encryption-secret
<project>-<instance>-openldap-credentials
<project>-<instance>-osmcn-cred-<user> (If you have this secret created. Ignore, if the secret do not exist.)
<project>-<instance>-saf-<remote-system> (If you have this secret created. Ignore, if the secret do not exist.)

Run the following command on each secret which would copy the output to the secret.yaml file. Maintain a separate file for each secret.
```
kubectl get secret -n project secretName -o yaml > secret.yaml
```
Set the $OSM_CNTK environment variable to point to the 7.5.0 cloud native toolkit.
Copy the specification files updated in your earlier steps to your $SPEC_PATH location and rename them to match the same as existing 7.4.1 cloud native specification files (to match the existing project name and the instance name).
(Optional) If you use a newer version of WKO, install the newer version WKO in a new namespace and register the project namespace with it.
(Optional) If you replace Traefik with another ingress controller, install the new ingress controller.

Updating the Secrets

Note:

Starting with the below steps, you will use the OSM 7.5.0 cloud native toolkit and specification files.

Update Existing Secrets

As a pre-requisite to using the toolkit for upgrading the OSM 7.4.1 cloud native environment, you must update the below secrets for access to the following:

OSM database
RCU DB
OSM system users

Secrets can be updated using manage-instance-credentials as below:

$OSM_CNTK/scripts/manage-instance-credentials.sh -p project -i instance update osmdb,rcudb,osmldap

In OSM 7.5.0 cloud native, when osmldap is chosen, the script prompts the password for a new user "osm-gateway-internal", which is used to access the OSM Gateway microservices.

Creating New Secrets

Creating OIDC Secret for OSMGW and RTUX Microservices

This section describes how to create new secrets and credentials.

Note:

For more details about these secrets, see "Reference of Secrets Created by the Scripts".

Creating Secrets for an OSM Upgrade

If you are upgrading from a release prior to OSM 7.5.0, you should create the following secrets. If the upgrade is from OSM 7.5.0, you can update these existing secrets.

To create a new secret, run the following command:

$OSM_CNTK/scripts/manage-instance-credentials.sh -p project -i instance create secret

To update an existing secret, run the following command:

$OSM_CNTK/scripts/manage-instance-credentials.sh -p project -i instance update secret

You must create the OSMGW and RTUX secrets to access OSM Gateway and RTUX microservices which are secured with OIDC (OpenID connect) by default.

Run the following script to create the required secrets for access to OSM Gateway services. Make sure you have the required credentials available before running this script.

$OSM_CNTK/scripts/manage-instance-credentials.sh -p project -i instance create oidc

where oidc specifies the details of the OIDC IDP that will be used to authenticate access to the gateway API.

This command creates secret project-instance-oidc-credentials.

Creating TLS Secrets

If you have SSL turned on for incoming connections (by setting ssl.incoming to true in the specification files), then you must create wlstls and gatewaytls secrets as below to provide the required certificate information. This information is provided securely to the ingress controller.

This secret carries the application or ingress TLS credentials for OSM.

Note:

If you have the

secrets
                            project-instance-osm-tls-cert

, project-instance-admin-tls-cert and project-instance-t3-tls-cert already created as part of 7.4.1, skip this step and proceed to gatewaytls secret creation.

$OSM_CNTK/scripts/manage-instance-credentials.sh -p project -i instance create wlstls

This carries the application and ingress TLS credentials for microservices.

$OSM_CNTK/scripts/manage-instance-credentials.sh -p project -i instance create gatewaytls

Creating FluentD Credentials

If FluentD is enabled as a sidecar, then create the credentials for its connection to Elastic Search server.

$OSM_CNTK/scripts/manage-instance-credentials.sh -p project -i instance create fluentd

Creating Target Systems Credentials

If security schemes for the target systems are enabled, then create secrets for each target system separately.

$OSM_CNTK/scripts/manage-target-system-credentials.sh -p project -i instance -n securitySchemeName -t authenticationType create

Creating SAML SSO Secret

If SAML SSO is enabled, you should create a secret that carries the archive file that is needed by the OSM Weblogic domain to support SAML2 SSO.

$OSM_CNTK/scripts/manage-instance-credentials.sh -p project -i instance create samlsso

Upgrading the OSM DB Schema

Make sure you have the updated specification files for OSM 7.5.0 in your $SPEC_PATH.

Run the following command to upgrade the existing OSM schema:

$OSM_CNTK/scripts/install-osmdb.sh -p project -i instance -s $SPEC_PATH -c 1

Back-out procedure

Perform this procedure if the DB Schema upgrade does not work as expected. Use the 7.5.0 cloud native toolkit and specification files for this procedure.

To perform back-out procedure:

Drop the OSM Schema using DB installer code 8.

$OSM_CNTK/scripts/install-osmdb.sh -p project -i instance -c 8

Delete all the secrets that were created while updating the secrets (including fluentd, gatewaytls, target systems).

$OSM_CNTK/scripts/manage-instance-credentials.sh -p project -i instance delete oidc,osmdb,rcudb,wlsadmin,osmldap,opssWP,wlsRTE,fluentd,gatewaytls

$OSM_CNTK/scripts/manage-target-system-credentials.sh -p project -i instance -n security_scheme_name delete (optional)

Restore the OSM 7.4.1 schema backup that was taken during the "Preparation Steps for the Upgrade" task.
Copy the 7.4.1 cloud native backup specification files to $SPEC_PATH location and set the $OSM_CNTK variable to point to the 7.4.1 CNTK.
Import all the secrets from the backup files taken before the upgrade. Run the following command using each secret file to import the secret.
```
kubectl apply -f secret.yaml
```

Create the instance.

$OSM_CNTK/scripts/create-instance.sh -p project -i instance -s $SPEC_PATH

Create the ingress.

$OSM_CNTK/scripts/create-ingress.sh -p project -i instance -s $SPEC_PATH

OSM Application Upgrade

Make sure you have the updated specification files for OSM 7.5.0 in your $SPEC_PATH.

Run $OSM_CNTK/scripts/create-instance.sh to create an instance with OSM 7.5.0 cloud native artifacts.

$OSM_CNTK/scripts/create-instance.sh -p project -i instance -s $SPEC_PATH

Once the create-instance.sh script execution is completed, admin and managed server pods will be in the running state.

Create the ingress.

$OSM_CNTK/scripts/create-ingress.sh -p project -i instance -s $SPEC_PATH

Back-out procedure (in case OSM application upgrade didn't work as expected):

With 7.5.0 cloud native toolkit and specification files, follow below steps:

Delete the OSM cloud native instance.

$OSM_CNTK/scripts/delete-instance.sh -p project -i instance

Perform the DB backout procedure as described above.