13 Upgrading your OSM Cloud Native Deployment

This chapter provides instructions for upgrading OSM cloud native deployment release 7.4.1 or later to OSM cloud native 8.0.

Overview of the Upgrade Steps

This sections outlines the upgrade steps:

• Download the OSM 8.0 CNTK version.
• Install or upgrade WebLogic Kubernetes Operator (WKO). See "Installing WebLogic Kubernetes Operator".
• Decide on the Ingress Controller. See "Ingress Controller".
• Build the OSM, DB Installer, OSMGW, RTUX and OCA images, using the same OSM cloud native version as the toolkit. See "Creating OSM Cloud Native Images" for building the images.
• Updating the specification files. See "Updating Specification Files".
• Upgrading to OSM Cloud Native 8.0. See "Upgrading to OSM Cloud Native 8.0".

Installing WebLogic Kubernetes Operator

This section provides information about installing WebLogic Kubernetes Operator.

WKO Monitoring Mechanism

Once WKO is installed in a Kubernetes cluster, it needs to know which namespaces will have OSM instances. Of interest are two mechanisms:

• List: WKO is given an explicit list of namespaces. This is done by upgrading the WKO helm chart to put in a new value for the list (one that takes the existing value and either adds a new namespace or removes one of the namespaces). This mechanism is supported in WKO 3.x and 4.x and is the mechanism in OSM 7.4.1 CNTK as well (upto 7.4.1.0.13). With this, the user needs RBAC access to the WKO namespace, which is likely administratively problematic in a large shared environment.
• Label: When WKO is installed, it is given a label to look for in namespaces. Any namespace in the cluster that has that label is monitored. Adding or removing that label from a namespace has the effect of adding or removing that namespace from WKO's monitoring. This mechanism is supported in WKO 4.x. With this, the user needs RBAC access only to the OSM project namespace and not to WKO namespace.

For OSM cloud native, the recommendation is to use the Label mechanism for WKO 4.x and especially for WKO 4.0.8 and newer.

Operator Installation

Before starting with the procedure in this section, refer to OSM Compatibility Matrix for release 8.0 for the list of supported WKO versions. It is mandatory to have WKO installed with label-based monitoring mechanism for OSM 8.0.

If your existing WebLogic Kubernetes Operator satisfies the OSM 8.0 version requirements and uses the label-based monitoring mechanism, no action is required. Skip to the "Ingress Controller" section.

If you need to upgrade, it is recommended to install the new WKO into a new namespace. As individual namespaces are deregistered from old WKO and registered to new WKO, eventually, the old WKO will no longer be managing any namespaces. It can then be safely uninstalled.

• Refer to Weblogic Kubernetes Operator documentation for operator installation at: https://oracle.github.io/weblogic-kubernetes-operator/managing-operators/
• During installation, it is strongly recommended to choose a specific label that this version of WKO must look for, rather than relying on the default label. Relying on the default label causes problems when multiple WKOs need to be installed for any reason (such as during a phased WKO upgrade).
• To provide the custom label, use the domainNamespaceLabelSelector parameter as shown below.
• It is recommended that the custom label you select is descriptive and unique to this WKO. In the example below, this is achieved by specifying the full WKO version.

  # Eg: For installing WKO version 4.1.2 with a custom label wlsko412=enabled, use below after configuring the right helm repo
  $ helm install $WLSKO_NS \
   weblogic-operator/weblogic-operator \
   --namespace $WLSKO_NS \
   --version 4.1.2 \
   --set "domainNamespaceLabelSelector=wlsko412\=enabled"

Unregistering and Registering the Namespace with Weblogic Operator

Using your older CNTK, unregister your namespace forcefully with operator using "unregister-namespace.sh" script

# Set required variables with earlier CN values (before upgrade)
$ export OSM_CNTK=<osm_7.4.1/7.5.0_cntk>
$ export WLSKO_NS=<operator_namespace_before_upgrade>
$ export WLSKO_HOME=<path_for_operator_home>
$ $OSM_CNTK/scripts/unregister-namespace.sh -p $PROJECT -t wlsko -f

Install/upgrade your operator by following weblogic operator documentation.

Register OSM namespace with recent weblogic operator using "register-namespace.sh" script.

# Set required variables with OSM CN 8.0 values
$ export OSM_CNTK=<osm_8.0_cntk>
$ export WLSKO_NS=<operator_namespace_before_upgrade>
$ $OSM_CNTK/scripts/register-namespace.sh -p $PROJECT -t wlsko -l <custom-label>

Ingress Controller

OSM cloud native supports any annotations based ingress controller.

Support for Traefik (deprecated earlier) has been discontinued in favour of ingress controllers that support the generic Kubernetes ingress API. See "Working with Ingress, Ingress Controller, and External Load Balancer" for more details. If you were using Traefik earlier, and you wish to transition to such an Ingress Controller, you must delete ingresses, deregister the OSM project namespace from traefik, install the new ingress controller and, if required, register the OSM project namespace with the new controller. This will result in new ingresses being created. Further, it is recommended to uninstall the Traefik chart and delete the Traefik namespace before beginning the upgrade process. As part of upgrading, add the required annotations for the Ingress object to your 8.0 specification files, ensure Ingress objects in the namespace will be monitored by the replacement Ingress Controller, and create the new Ingress with the 8.0 toolkit using the create-ingress.sh script.

Updating Specification Files

This section focuses on the delta in the specification files between OSM 7.4.1, OSM 7.5.0 and new OSM 8.0 releases.

It is strongly recommended that you begin by copying over the sample specification files from the OSM 8.0 CNTK. You can then re-make the customizations from your current specification files one by one.

Updating the Project Specification

This section describes the sections that you update in the project specification:

Upgrading from 7.4.1 to 8.0

• OSM cloud native 8.0 adds new capabilities (such as OSMGW, RTUX and OCA microservices) for which there are corresponding sections in the project and instance specification files. See Creating OSM Cloud Native Images for more information.
• The default value for Ingress Controller has been set to GENERIC.
• If you use GENERIC ingress controller, the required annotations need to be added to the ingress section. In the project specification file, ingress.annotations lists the required annotations if you are using NGINX ingress controller. Adjust the annotations based on the ingress controller that you use.
• crossDomainTrustUsers section has been added which lists users for the CrossDomainConnectors group, required for cross-domain security and credential mapping. If you were using global trust before, it is recommended to switch to cross-domain trust for finer grained control.
• fluentdLogging section has been added that enables Fluentd sidecar logging by specifying image options for WebLogic and DB Installer pods.
• In OSM 7.4.1 cloud native, there is a separate step to deploy WME WAR file to the instance as a custom archive. In 8.0 cloud native, WebLogic Monitoring Exporter can be enabled and disabled by changing the value of enabled in the instance.yaml. By default, it is set to true, which creates a sidecar container for WME using the default image for the version of WebLogic Operator in use. If you want to use a different image than the one that WKO uses, you have a provision to do that by updating WME in the project specification file.
• openldap authentication is deprecated; use ldap authentication instead. Refer to the authentication section.
• There are new capabilities added for deploying cartridges. OSM cloud native cartridge deployment supports container image built with par file. Either the url or the Image field should be populated for a particular cartridge but not both. See Cartridge par Sources for more details.
• A new section named capibilitiesCartridgeList has been added to the list capabilties cartridge. See About Dynamic Cartridge Assembly (Cloud Native Only) in OSM Concepts.
• Defining osmWLSTargetNodes restricts all OSM cloud native WebLogic pods and DB Installer pods to worker nodes, that match the label conditions and for these pods and it will take precedence over osmcnTargetNodes.
• Update the requiredTargetSystems section if your cartridge needs to integrate with external systems using REST API.
• Declarations for secretNames , customFiles, safDestinationConfig, cartridges, certificates, certificates.secrets, uniformDistributedQueues, and uniformDistributedTopics have been changed from {} (map) to [] (list).

Upgrading from 7.5.0 to 8.0

• OSM cloud native 8.0 adds the OCA microservice for which there are corresponding sections in the project and instance specification files. See Creating OSM Cloud Native Images for more information.
• You can update ingress annotations related to oca in ingress.oca.annotations based on your configuration.
• A new section named capibilitiesCartridgeList has been added to the list capabilties cartridge. See About Dynamic Cartridge Assembly (Cloud Native Only) in OSM Concepts.
• Double quotes have been removed from numeric values such as threshold, duration, and resetDuration, so they are now specified as integers instead of strings.
• Declarations for safDestinationConfig, cartridgeUsers, uniformDistributedTopics, uniformDistributedQueues have been changed from {} (map) to [] (list).

Updating the Instance Specification

This section describes the sections that you update in the instance specification:

Upgrading from 7.4.1 to 8.0

• The serverStartPolicy parameter now requires use of the correct case-sensitive value depending on your deployment version. Refer to the section in 8.0 instance.yaml file.
• If you are planning to use log masking functionality, uncomment and use the logMaskingCustomRegexes section.
• OSM supports fine grain control of the log level for OSM core using the instance specification file. Uncomment and use the log section for overriding the default log level.
• WebLogic Monitoring Exporter can be enabled and disabled by changing the value of weblogicMonitoringExporter.enabled in the instance.yaml file. By default, this is set to true.
• You can use FluentD as a sidecar container for reading the logs from OSM Servers and DB Installer. Enable the fluentdLogging section for making use of this functionality. See "Configuring Fluentd Logging" for more details.
• The storage volume must specify pvc to be used for persistent storage or as emptydir to share with a custom sidecar which is enabled via sidecar.enabled in instance specification file.

  storageVolume:
    enabled: true
    type: pvc # Acceptable values are pvc and emptydir
    volumeName: storage-volume
    pvc: storage-pvc # Specify this only in case type is pvc

• If you want to enable a sidecar other than FluentD, make use of the section sidecar.enabled.
• In earlier versions, to enable global trust, the configuration you needed to use was domainTrust.enabled. While this is still supported for backward compatibility, it is deprecated in favor of the configuration domainTrust.globalEnabled. Oracle recommends that if you are using an instance specification from older versions, you should update it with this new configuration.
• A valid Custom Identity keystore must be provided. The use of DemoIdentity is no longer supported.

  identity:
      name: keystore # Identity store filename without extension (keystore.jks)

• For enabling SSL using generic Ingress, add the annotations required for OSM. The ssl.ingress.annotations section in the instance specification file lists the sample annotations required for OSM if you are using NGINX ingress controller. Add the annotations based on the ingress controller that you choose.
• The db section now adds a required field type (STANDARD or ADB) to specify your database type. Fields like serviceName, datasourcesSecondary, and rcuDb are now included for expanded configuration. Refer to the db section for more details.
• In the 7.4.1 cloud native, the instance.yaml file contains the loadBalancerPort section. In OSM 8.0 cloud native, it is renamed to inboundGateway. As part of the inboundGateway section, you need to provide the host and port details of the actual ingress point or the load balancer.
• Using Host Aliases, you can achieve hostname resolution inside the pods. Refer to the hostAliases element and its comments in the instance specification for more details.
• Update the osm-gateway, osmRuntimeUX and osm-cartridge-assembler sections as per "Configuring Target Systems for Events and System Interactions".
• If you have defined targetSystems in the project specification file, provide the details of the respective target system in the instance specification file.
• The serviceName field in the db section and the host field under the db.datasourcesPrimary section has been made mandatory.

  db:
    type: "STANDARD" # Acceptable values are STANDARD and ADB
    #serviceName: dbserver-servicename
    # This DB protocol is all applicable for all database connections.
    # Default value is TCP, Uncomment and change it to TCPS when required.
    # If TCPS is selected, the dbwallet "<project>-<instance>-db-ssl-wallet" secret must exist
    #protocol: TCP
    # datasourcesPrimary section is applicable only for STANDARD DB. For ADB, values will be used from Autonomous Database Serverless  secrets+configMap.
    datasourcesPrimary:
      port: 1521
      # Provide the DB server hostname/IP address
      #host: dbserver-ip

• Declarations for secretNames and safConnectionConfig has been changed from {} (map) to [] (list).

Upgrading from 7.5.0 to 8.0

• A valid Custom Identity keystore must be provided. The use of DemoIdentity is no longer supported.

  identity:
      name: keystore # Identity store filename without extension (keystore.jks)

• The serviceName field in the db section and the host field under the db.datasourcesPrimary section has been made mandatory.

  db:
    type: "STANDARD" # Acceptable values are STANDARD and ADB
    #serviceName: dbserver-servicename
    # This DB protocol is all applicable for all database connections.
    # Default value is TCP, Uncomment and change it to TCPS when required.
    # If TCPS is selected, the dbwallet "<project>-<instance>-db-ssl-wallet" secret must exist
    #protocol: TCP
    # datasourcesPrimary section is applicable only for STANDARD DB. For ADB, values will be used from Autonomous Database Serverless  secrets+configMap.
    datasourcesPrimary:
      port: 1521
      # Provide the DB server hostname/IP address
      #host: dbserver-ip

• A new portConfig section has been introduced under inboundGateway.port to enable detailed configuration of individual service ports when using the port access style at the ingress point. Refer to the instance specification file for more details.
• A new osm-cartridge-assembler section has been introduced. Refer to the instance specification file for more details.
• Declarations for logMaskingCustomRegexes have been changed from {} (map) to [] (list) and log has been changed from [] (list) to {} (map).

Copy the remaining sections for which there are no changes from your existing OSM cloud native version without making any changes to OSM cloud native 8.0 based on your usage of the respective functionalities in OSM cloud native 8.0.

Updating Shape Specification

If you have a custom shape specification, identify the standard shape on which your custom shape is based. Make a copy of this standard shape from the OSM 8.0 cloud native toolkit and make your customization one-by-one to it.

Upgrading to OSM Cloud Native 8.0

Oracle recommends that you upgrade the existing OSM 7.4.1 or 7.5.0 cloud native application to 8.0 cloud native using the same project namespace and the instance name.

Prerequisites for the Upgrade

Check OSM Compatibility Matrix for details about the required and supported versions of the pre-requisite software, before proceeding with the upgrade.

Note:

Perform the following steps using the existing 7.4.1 or 7.5 Cloud Native toolkit specification files.

Delete the Instance

To delete an instance, run the following:

$OSM_CNTK/scripts/delete-instance.sh -p project -i instance

Delete the Ingress

To delete an ingress, run the following:

$OSM_CNTK/scripts/delete-ingress.sh -p project -i instance

If you are switching from Traefik as part of this upgrade, to stop Traefik from monitoring this namespace, run the following:

$OSM_CNTK/scripts/unregister-namespace.sh -t traefik -p project

Preparation Steps for the Upgrade

This section lists preparatory steps for the upgrade:

• Take a backup of the working OSM 7.4.1 or 7.5.0 project and instance specification files or rely on your source code management system to maintain versioning.
• Take the OSM 7.4.1 or 7.5.0 DB schema backup before the upgrade.
• Take a backup of the following secrets and also of any custom OSM secrets created for 7.4.1 or 7.5.0 cloud native before the upgrade.

  <project>-<instance>-database-credentials
  <project>-<instance>-embedded-ldap-credentials
  <project>-<instance>-weblogic-credentials
  <project>-<instance>-rcudb-credentials
  <project>-<instance>-opss-wallet-password-secret
  <project>-<instance>-runtime-encryption-secret
  <project>-<instance>-osmcn-cred-<user> (If you have this secret created. Ignore, if the secret do not exist.)
  <project>-<instance>-saf-<remote-system> (If you have this secret created. Ignore, if the secret do not exist.)

  Note:

  If you are upgrading from an older version of OSM cloud native and have OpenLDAP, note that it was deprecated in 7.5 and is now discontinued. You need to switch to Generic LDAP.
• Run the following command on each secret which would copy the output to the secret.yaml file. Maintain a separate file for each secret.

  kubectl get secret -n project secretName -o yaml > secret.yaml

• Set the $OSM_CNTK environment variable to point to the 8.0 cloud native toolkit.
• Copy the specification files updated in your earlier steps to your $SPEC_PATH location and rename them to match the same as existing 7.4.1 or 7.5.0 cloud native specification files (to match the existing project name and the instance name).
• (Optional) If you use a newer version of WKO, install the newer version WKO in a new namespace and register the project namespace with it.
• (Optional) If you replace the existing ingress with another ingress controller, install the new ingress controller.

Note:

You need to provide the following GRANT on the dba user which has been used in the secrets created for osmdb and rcudb. This is required for RCU Schema creation.

GRANT execute on dbms_lob to _replace_this_text_with_admin_name_ with grant option;

OSM no longer requires the sysdba role for its DB schema owner. This role can be safely rescinded. To rescind the role, run the following command:

REVOKE sysdba from _replace_this_text_with_admin_name_;

Updating the Secrets

Note:

Starting with the below steps, you will use the OSM 8.0 cloud native toolkit and specification files.

Update Existing Secrets

As a pre-requisite to using the toolkit for upgrading the OSM 7.4.1 or 7.5.0 cloud native environment, you must update the below secrets for access to the following:

• OSM database
• RCU DB
• OSM system users

Secrets can be updated using manage-instance-credentials as below:

$OSM_CNTK/scripts/manage-instance-credentials.sh -p project -i instance -s spec_path update osmdb,rcudb,osmldap

In OSM 8.0 cloud native:

• Database host, port and service name have been moved from secrets to instance specifications. OSM and RCU schemas will be created on the same database.
• When you choose osmldap, the script prompts the password for a new user "osm-gateway-internal", which is used to access the OSM Gateway microservices.

Creating New Secrets

This section describes how to create new secrets and credentials.

Note:

For more details about these secrets, see Reference of Secrets Created by the Scripts.

Creating Secrets for an OSM Upgrade

If you are upgrading from a release prior to OSM 7.5.0, you should create the following secrets. If the upgrade is from OSM 7.5.0, you can update these existing secrets.

To create a new secret, run the following command:

$OSM_CNTK/scripts/manage-instance-credentials.sh -p project -i instance -s spec_path create secret

To update an existing secret, run the following command:

$OSM_CNTK/scripts/manage-instance-credentials.sh -p project -i instance -s spec_path update secret

Clean Up OIDC Secret

If you are upgrading from a release prior to OSM 7.5.0, you need to create OIDC secrets. If you are upgrading from OSM 7.5, you need to delete the existing OIDC secret, project-instance-oidc-credentials, using the OSM 7.5 cloud native toolkit or manually. You can use the following command to delete the existing OIDC Secret using the OSM 7.5 cloud native toolkit:

$OSM_CNTK/scripts/manage-instance-credentials.sh -p project -i instance -s spec_path delete oidc
 
or,
 
kubectl delete secret -n project project-instance-oidc-credentials

After deleting the existing secret, you need to re-create the required OIDC secrets using the OSM 8.0 CNTK. You need to create the new OIDC secrets to the APIs exposed by the Gateway, RTUX and OCA microservices. For more information on how to create the new secrets, see Creating OSM Secrets for OIDC in Configuring OpenID Connect for OSM Microservices

Creating TLS Secrets

If you have SSL turned on for incoming connections (by setting ssl.incoming to true in the specification files), then you must create wlstls and gatewaytls secrets as below to provide the required certificate information. This information is provided securely to the ingress controller.

This secret carries the application or ingress TLS credentials for OSM.

Note:

If you have the secrets project-instance-osm-tls-cert, project-instance-admin-tls-cert and project-instance-t3-tls-cert already created as part of 7.4.1, skip this step and proceed to gatewaytls secret creation. In 7.5.0, if you have already created the gatewaytls secret, you do not need to perform the gatewaytls secret creation step again.

$OSM_CNTK/scripts/manage-instance-credentials.sh -p project -i instance -s spec_path create wlstls

This carries the application and ingress TLS credentials for microservices.

$OSM_CNTK/scripts/manage-instance-credentials.sh -p project -i instance -s spec_path create gatewaytls

Creating FluentD Credentials

If FluentD is enabled as a sidecar, then create the credentials for its connection to Elastic Search server.

$OSM_CNTK/scripts/manage-instance-credentials.sh -p project -i instance -s spec_path create fluentd

Creating Target Systems Credentials

If security schemes for the target systems are enabled, then create secrets for each target system separately.

$OSM_CNTK/scripts/manage-target-system-credentials.sh -p project -i instance -n securitySchemeName -t authenticationType create

Creating SAML SSO Secret

If SAML SSO is enabled, you should create a secret that carries the archive file that is needed by the OSM Weblogic domain to support SAML2 SSO.

$OSM_CNTK/scripts/manage-instance-credentials.sh -p project -i instance -s spec_path create samlsso

Creating DB Wallet Secret

If the db.protocol is TCPS, you need to create a secret to carry the wallet that is needed for OSM to make the connection with the database.

$OSM_CNTK/scripts/manage-instance-credentials.sh -p project -i instance -s spec_path create dbwallet

Upgrading the OSM DB Schema

Make sure you have the updated specification files for OSM 8.0 in your $SPEC_PATH.

Run the following command to upgrade the existing OSM schema:

$OSM_CNTK/scripts/install-osmdb.sh -p project -i instance -s $SPEC_PATH -c 1

Back-out procedure

Perform this procedure if the DB Schema upgrade does not work as expected. Use the 8.0 cloud native toolkit and specification files for this procedure.

To perform back-out procedure:

1. Drop the OSM Schema using DB installer code 8.

   $OSM_CNTK/scripts/install-osmdb.sh -p project -i instance -c 8

2. Delete all the secrets that were created while updating the secrets (including fluentd, gatewaytls, target systems).

   $OSM_CNTK/scripts/manage-instance-credentials.sh -p project -i instance -s spec_path \ delete \ oidc,osmdb,rcudb,wlsadmin,osmldap,opssWP,wlsRTE,fluentd,gatewaytls
   
   $OSM_CNTK/scripts/manage-target-system-credentials.sh -p project -i instance -n security_scheme_name delete (optional)

3. Restore the OSM 7.4.1 or 7.5.0 installation schema backup that was taken during the "Preparation Steps for the Upgrade" task.
4. Copy the 7.4.1 or 7.5.0 cloud native backup specification files to $SPEC_PATH location and set the $OSM_CNTK variable to point to the cloud native toolkit of your 7.4.1 or 7.5.0 OSM cloud native (before upgrade) environment.

5. Import all the secrets from the backup files taken before the upgrade. Run the following command using each secret file to import the secret.

   kubectl apply -f secret.yaml

6. Create the instance.

   $OSM_CNTK/scripts/create-instance.sh -p project -i instance -s $SPEC_PATH

7. Create the ingress.

   $OSM_CNTK/scripts/create-ingress.sh -p project -i instance -s $SPEC_PATH

OSM Application Upgrade

Make sure you have the updated specification files for OSM 8.0 in your $SPEC_PATH.

Run $OSM_CNTK/scripts/create-instance.sh to create an instance with OSM 8.0 cloud native artifacts.

$OSM_CNTK/scripts/create-instance.sh -p project -i instance -s $SPEC_PATH

Once the create-instance.sh script completes running, admin and managed server pods will be in the running state.

Create the ingress.

$OSM_CNTK/scripts/create-ingress.sh -p project -i instance -s $SPEC_PATH

Back-out procedure (in case OSM application upgrade didn't work as expected):

With 8.0 cloud native toolkit and specification files, follow below steps:

1. Delete the OSM cloud native instance.

   $OSM_CNTK/scripts/delete-instance.sh -p project -i instance

2. Perform the DB backout procedure as described above.