12 Upgrading your OSM Cloud Native Deployment
This chapter provides instructions for upgrading OSM cloud native deployment release 7.4.1 to OSM cloud native 7.5.
Overview of the Upgrade Steps
This sections outlines the upgrade steps:
- Download the OSM 7.5.0 CNTK version.
- Install or upgrade WebLogic Kubernetes Operator (WKO). See "Installing WebLogic Kubernetes Operator".
- Decide on the Ingress Controller. See "Ingress Controller".
- Build the OSM, DB Installer, OSMGW, and RTUX images, using the same OSM cloud native version as the toolkit. See "Creating OSM Cloud Native Images" for building the images.
- Updating the specification files. See "Updating Specification Files".
- Upgrading to OSM cloud native 7.5.0. See "Upgrading to OSM Cloud Native 7.5.0".
Installing WebLogic Kubernetes Operator
This section provides information about installing WebLogic Kubernetes Operator.
WKO Monitoring Mechanism
Once WKO is installed in a Kubernetes cluster, it needs to know which namespaces will have OSM instances. Of interest are two mechanisms:
- List: WKO is given an explicit list of namespaces. This is done by upgrading the WKO helm chart to put in a new value for the list (one that takes the existing value and either adds a new namespace or removes one of the namespaces). This mechanism is supported in WKO 3.x and 4.x and is the mechanism in OSM 7.4.1 CNTK as well (upto 7.4.1.0.13). With this, the user needs RBAC access to the WKO namespace, which is likely administratively problematic in a large shared environment.
- Label: When WKO is installed, it is given a label to look for in namespaces. Any namespace in the cluster that has that label is monitored. Adding or removing that label from a namespace has the effect of adding or removing that namespace from WKO's monitoring. This mechanism is supported in WKO 4.x. With this, the user needs RBAC access only to the OSM project namespace and not to WKO namespace.
For OSM cloud native, the recommendation is to use the Label mechanism for WKO 4.x and especially for WKO 4.0.8 and newer.
Operator Installation
Before starting with the procedure in this section, refer to OSM Compatibility Matrix for release 7.5.0 for the list of supported WKO versions. It is mandatory to have WKO installed with label-based monitoring mechanism for OSM 7.5.0.
If your existing WebLogic Kubernetes Operator satisfies the OSM 7.5.0 version requirements and uses the label-based monitoring mechanism, no action is required. Skip to the "Ingress Controller" section.
If you need to upgrade, it is recommended to install the new WKO into a new namespace. As individual namespaces are deregistered from old WKO and registered to new WKO, eventually, the old WKO will no longer be managing any namespaces. It can then be safely uninstalled.
- Refer to Weblogic Kubernetes Operator documentation for operator installation at: https://oracle.github.io/weblogic-kubernetes-operator/managing-operators/
- During installation, it is strongly recommended to choose a specific label that this version of WKO must look for, rather than relying on the default label. Relying on the default label causes problems when multiple WKOs need to be installed for any reason (such as during a phased WKO upgrade).
- To provide the custom label, use the
domainNamespaceLabelSelector
parameter as shown below. - It is recommended that the custom label you select is descriptive and
unique to this WKO. In the example below, this is achieved by specifying the full
WKO
version.
# Eg: For installing WKO version 4.1.2 with a custom label wlsko412=enabled, use below after configuring the right helm repo $ helm install $WLSKO_NS \ weblogic-operator/weblogic-operator \ --namespace $WLSKO_NS \ --version 4.1.2 \ --set "domainNamespaceLabelSelector=wlsko412\=enabled"
Unregistering and Registering the Namespace with Weblogic Operator
Using your older CNTK, unregister your namespace forcefully with operator using "unregister-namespace.sh" script
# Set required variables with earlier CN values (before upgrade)
$ export OSM_CNTK=<osm_7.4.1_cntk>
$ export WLSKO_NS=<operator_namespace_before_upgrade>
$ export WLSKO_HOME=<path_for_operator_home>
$ $OSM_CNTK/scripts/unregister-namespace.sh -p $PROJECT -t wlsko -f
Install/upgrade your operator by following weblogic operator documentation.
Register OSM namespace with recent weblogic operator using "register-namespace.sh" script.
# Set required variables with OSM CN 7.5.0 values
$ export OSM_CNTK=<osm_7.5.0_cntk>
$ export WLSKO_NS=<operator_namespace_before_upgrade>
$ $OSM_CNTK/scripts/register-namespace.sh -p $PROJECT -t wlsko -l <custom-label>
Ingress Controller
OSM cloud native supports any annotations based ingress controller.
If you were using Traefik ingress controller earlier, you can continue to use it. However, support for Traefik is deprecated in favour of ingress controllers that support the generic Kubernetes ingress API. See "Working with Ingress, Ingress Controller, and External Load Balancer" for more details. If you wish to transition to such an Ingress Controller, it is recommended to remove the existing Traefik ingress objects (delete-ingress.sh) and unregister the namespace from Traefik (unregister-namespace.sh -t traefik) before beginning the upgrade process. As part of upgrading, add the required annotations for the Ingress object to your 7.5 specification files, ensure Ingress objects in the namespace will be monitored by the replacement Ingress Controller and create the new Ingress (create-ingress.sh with the 7.5 toolkit.
Updating Specification Files
This section focuses on the delta in the specification files between OSM 7.4.1 and OSM 7.5.0 releases.
It is strongly recommended that you begin by copying over the sample specification files from the OSM 7.5.0 CNTK. You can then re-make the customizations from your current specification files one by one.
Updating the Project Specification
This section describes the sections that you update in the project specification:
- OSM cloud native 7.5.0 adds new capabilities (such as OSMGW and RTUX microservices) for which there are corresponding sections in the project and instance specification files. See to "Creating OSM Cloud Native Images" for details.
- The default value for Ingress Controller has been set to GENERIC. If you wish to retain Traefik, change this to TRAEFIK.
- If you use GENERIC ingress controller, the required annotations need to
be added to the "ingress" section. In the project specification file,
ingress.annotations
lists the required annotations if you are using Nginx ingress controller. Adjust the annotations based on the ingress controller that you use. - In OSM 7.4.1 cloud native, there is a separate step to deploy WME WAR file to the instance as a custom archive. In 7.5.0 cloud native, WebLogic Monitoring Exporter can be enabled and disabled by changing the value of enabled in the instance.yaml. By default, it is set to true, which creates a sidecar container for WME using the default image for the version of WebLogic Operator in use. If you want to use a different image than the one that WKO uses, you have a provision to do that by updating WME in the project specification file.
- There are new capabilities added for deploying cartridges. OSM cloud native cartridge deployment supports container image built with par file. Either the url or the Image field should be populated for a particular cartridge but not both. See "Cartridge par Sources" for more details.
- Update the requiredTargetSystems section if your cartridge needs to integrate with external systems using REST API.
- Defining osmWLSTargetNodes restricts all OSM cloud native WebLogic pods and DB Installer pods to worker nodes, that match the label conditions and for these pods and it will take precedence over osmcnTargetNodes.
- The remaining sections for which there are no changes from OSM cloud native 7.4.1 need to be copied as-is to 7.5.0 based on your usage of the respective functionality in OSM cloud native 7.5.0.
Updating the Instance Specification
This section describes the sections that you update in the instance specification:
- If you are planning to use log masking functionality, uncomment and use
the
logMaskingCustomRegexes
section. - OSM 7.5.0 supports fine graining the log level for OSM core using the instance specification file. Uncomment and use the log section for overriding the default log level.
- WebLogic Monitoring Exporter can be enabled and disabled by changing
the value of
weblogicMonitoringExporter.enabled
in the instance.yaml file. By default, this is set to true. - OSM 7.5.0 introduces support for FluentD as a sidecar container for
reading the logs from OSM Servers and DB Installer. Enable the
fluentdLogging
section for making use of this functionality. See "Configuring Fluentd Logging" for more details. - The storage volume must specify "pvc" to be used for persistent storage or as
"emptydir" to share with a custom sidecar which is enabled via "sidecar.enabled" in
instance specification
file.
storageVolume: enabled: true type: pvc # Acceptable values are pvc and emptydir volumeName: storage-volume pvc: storage-pvc # Specify this only in case type is pvc
- If you want to enable a sidecar other than FluentD, make use of the section
sidecar.enabled
. - For enabling SSL using generic Ingress, add the annotations required
for OSM. The
ssl.ingress.annotations
section in the instance specification file lists the sample annotations required for OSM if you are using Nginx ingress controller. Add the annotations based on the ingress controller that you choose. - In the 7.4.1 cloud native, instance.yaml file contains the loadBalancerPort section. In OSM 7.5.0 cloud native, it is renamed to inboundGateway. As part of the inboundGateway section, provide the host and port details of the actual ingress point or the loadbalancer.
- OSM cloud native 7.5.0 introduces support for Host Aliases with which
you can achieve hostname resolution inside the pods. Refer to the
hostAliases
element and its comments in the instance specification for more details. - Update the
osm-gateway
andosmRuntimeUX
sections as per "Configuring Target Systems for Events and System Interactions". - If you have defined
targetSystems
in the project specification file, provide the details of the respective target system in the instance specification file. - Copy the remaining sections for which there are no changes from OSM cloud native 7.4.1 as-is to 7.5.0 based on your usage of the respective functionality in OSM cloud native 7.5.0.
Upgrading to OSM Cloud Native 7.5.0
Oracle recommends that upgrade the existing OSM 7.4.1 cloud native application to 7.5.0 cloud native using the same project namespace and the instance name.
Prerequisites for the Upgrade
Check OSM Compatibility Matrix for details about the required and supported versions of the pre-requisite software, before proceeding with the upgrade.
Note:
Perform the following steps using the existing 7.4.1 Cloud Native toolkit specification files.Delete the Instance
To delete an instance, run the following:
$OSM_CNTK/scripts/delete-instance.sh -p project -i instance
Delete the Ingress
To delete an ingress, run the following:
$OSM_CNTK/scripts/delete-ingress.sh -p project -i instance
If you are switching from Traefik as part of this upgrade, to stop Traefik from monitoring this namespace, run the following:
$OSM_CNTK/scripts/unregister-namespace.sh -t traefik -p project
Preparation Steps for the Upgrade
This section lists preparatory steps for the upgrade:
- Take a backup of the working OSM 7.4.1 project and instance specification files or rely on your source code management system to maintain versioning.
- Take the OSM 7.4.1 DB schema backup before the upgrade.
- Take a backup of the following secrets and also if there are any custom
OSM secrets created for 7.4.1 cloud native before the
upgrade.
<project>-<instance>-database-credentials <project>-<instance>-embedded-ldap-credentials <project>-<instance>-weblogic-credentials <project>-<instance>-rcudb-credentials <project>-<instance>-opss-wallet-password-secret <project>-<instance>-runtime-encryption-secret <project>-<instance>-openldap-credentials <project>-<instance>-osmcn-cred-<user> (If you have this secret created. Ignore, if the secret do not exist.) <project>-<instance>-saf-<remote-system> (If you have this secret created. Ignore, if the secret do not exist.)
- Run the following command on each secret which would copy the output to
the secret.yaml file. Maintain a separate file for each
secret.
kubectl get secret -n project secretName -o yaml > secret.yaml
- Set the $OSM_CNTK environment variable to point to the 7.5.0 cloud native toolkit.
- Copy the specification files updated in your earlier steps to your $SPEC_PATH location and rename them to match the same as existing 7.4.1 cloud native specification files (to match the existing project name and the instance name).
- (Optional) If you use a newer version of WKO, install the newer version WKO in a new namespace and register the project namespace with it.
- (Optional) If you replace Traefik with another ingress controller, install the new ingress controller.
Updating the Secrets
Note:
Starting with the below steps, you will use the OSM 7.5.0 cloud native toolkit and specification files.Update Existing Secrets
As a pre-requisite to using the toolkit for upgrading the OSM 7.4.1 cloud native environment, you must update the below secrets for access to the following:
- OSM database
- RCU DB
- OSM system users
Secrets can be updated using manage-instance-credentials as below:
$OSM_CNTK/scripts/manage-instance-credentials.sh -p project -i instance update osmdb,rcudb,osmldap
In OSM 7.5.0 cloud native, when osmldap is chosen, the script prompts the password for a new user "osm-gateway-internal", which is used to access the OSM Gateway microservices.
Creating New Secrets
Creating OIDC Secret for OSMGW and RTUX Microservices
This section describes how to create new secrets and credentials.
Note:
For more details about these secrets, see "Reference of Secrets Created by the Scripts".Creating Secrets for an OSM Upgrade
If you are upgrading from a release prior to OSM 7.5.0, you should create the following secrets. If the upgrade is from OSM 7.5.0, you can update these existing secrets.
$OSM_CNTK/scripts/manage-instance-credentials.sh -p project -i instance create secret
$OSM_CNTK/scripts/manage-instance-credentials.sh -p project -i instance update secret
You must create the OSMGW and RTUX secrets to access OSM Gateway and RTUX microservices which are secured with OIDC (OpenID connect) by default.
Run the following script to create the required secrets for access to OSM Gateway services. Make sure you have the required credentials available before running this script.
$OSM_CNTK/scripts/manage-instance-credentials.sh -p project -i instance create oidc
where oidc
specifies the details of the OIDC IDP that
will be used to authenticate access to the gateway API.
This command creates secret
project-instance-oidc-credentials
.
Creating TLS Secrets
If you have SSL turned on for incoming connections (by setting ssl.incoming to true in the specification files), then you must create wlstls and gatewaytls secrets as below to provide the required certificate information. This information is provided securely to the ingress controller.
Note:
If you have thesecrets
project-instance-osm-tls-cert
,
project-instance-admin-tls-cert
and
project-instance-t3-tls-cert
already created as
part of 7.4.1, skip this step and proceed to gatewaytls secret
creation.
$OSM_CNTK/scripts/manage-instance-credentials.sh -p project -i instance create wlstls
This carries the application and ingress TLS credentials for microservices.
$OSM_CNTK/scripts/manage-instance-credentials.sh -p project -i instance create gatewaytls
Creating FluentD Credentials
If FluentD is enabled as a sidecar, then create the credentials for its connection to Elastic Search server.
$OSM_CNTK/scripts/manage-instance-credentials.sh -p project -i instance create fluentd
Creating Target Systems Credentials
If security schemes for the target systems are enabled, then create secrets for each target system separately.
$OSM_CNTK/scripts/manage-target-system-credentials.sh -p project -i instance -n securitySchemeName -t authenticationType create
Creating SAML SSO Secret
If SAML SSO is enabled, you should create a secret that carries the archive file that is needed by the OSM Weblogic domain to support SAML2 SSO.
$OSM_CNTK/scripts/manage-instance-credentials.sh -p project -i instance create samlsso
Upgrading the OSM DB Schema
Make sure you have the updated specification files for OSM 7.5.0 in your $SPEC_PATH.
Run the following command to upgrade the existing OSM schema:
$OSM_CNTK/scripts/install-osmdb.sh -p project -i instance -s $SPEC_PATH -c 1
Back-out procedure
Perform this procedure if the DB Schema upgrade does not work as expected. Use the 7.5.0 cloud native toolkit and specification files for this procedure.
To perform back-out procedure:
-
Drop the OSM Schema using DB installer code 8.
$OSM_CNTK/scripts/install-osmdb.sh -p project -i instance -c 8
-
Delete all the secrets that were created while updating the secrets (including fluentd, gatewaytls, target systems).
$OSM_CNTK/scripts/manage-instance-credentials.sh -p project -i instance delete oidc,osmdb,rcudb,wlsadmin,osmldap,opssWP,wlsRTE,fluentd,gatewaytls $OSM_CNTK/scripts/manage-target-system-credentials.sh -p project -i instance -n security_scheme_name delete (optional)
- Restore the OSM 7.4.1 schema backup that was taken during the "Preparation Steps for the Upgrade" task.
- Copy the 7.4.1 cloud native backup specification files to $SPEC_PATH location and set the $OSM_CNTK variable to point to the 7.4.1 CNTK.
-
Import all the secrets from the backup files taken before the upgrade. Run the following command using each secret file to import the secret.
kubectl apply -f secret.yaml
-
Create the instance.
$OSM_CNTK/scripts/create-instance.sh -p project -i instance -s $SPEC_PATH
-
Create the ingress.
$OSM_CNTK/scripts/create-ingress.sh -p project -i instance -s $SPEC_PATH
OSM Application Upgrade
Make sure you have the updated specification files for OSM 7.5.0 in your $SPEC_PATH.
Run $OSM_CNTK/scripts/create-instance.sh to create an instance with OSM 7.5.0 cloud native artifacts.
$OSM_CNTK/scripts/create-instance.sh -p project -i instance -s $SPEC_PATH
Once the create-instance.sh script execution is completed, admin and managed server pods will be in the running state.
Create the ingress.
$OSM_CNTK/scripts/create-ingress.sh -p project -i instance -s $SPEC_PATH
Back-out procedure (in case OSM application upgrade didn't work as expected):
With 7.5.0 cloud native toolkit and specification files, follow below steps:
-
Delete the OSM cloud native instance.
$OSM_CNTK/scripts/delete-instance.sh -p project -i instance
- Perform the DB backout procedure as described above.