10 Upgrading the ASAP Cloud Native Environment

This chapter describes the tasks you perform in order to apply a change or upgrade to a component in the cloud native environment.

ASAP Cloud Native Upgrade Procedures

The upgrade procedure includes upgrading the ASAP cloud native image and deploying the ASAP image in the instance from ASAP 7.3.0 or 7.4.x.y to ASAP 8.0.

To upgrade the ASAP image:

  1. Download the latest Cloud Native 8.0 toolkit. It consists of the following files:
    • asap-img-builder.zip: Contains tools for building images.
    • asap-cntk.zip: Contains tools for deploying images.

  2. Delete the running instance using the delete–instance.sh script.

  3. Copy the required installers to the $asap-img-builder/installers directory with the following files and folders:
    • ASAP Redhat Package Manager (RPM) Installer, for example asap-installer-8.0.0.0.0-Bxxx.x86_64.rpm
    • WebLogic Server 14.1.2 installer zip file, for example V1045131-01.zip
    • Oracle Database Client golden base version 19.3.0.0 for Linux 64 bit LINUX.X64_193000_client.zip.This only applies to 7.4.0..
    • OPatch Utility. This only applies to 7.4.0.
    • JDK 21 (jdk-21_linux-x64_bin.tar.gz)
    • Oracle Database Release Updates. This only applies to 7.4.0.
    • ANT installer tar file (apache-ant-1.10.15-bin.tar.gz). This only applies to 7.4.0

  4. Copy the required cartridges to the $asap-img-builder/cartridges directory.

  5. Run the following command to copy installers and cartridges to the volume:
    $ASAP_IMG_BUILDER/upgrade-asap-image.sh
  6. Create a new container using the previous version of the ASAP image using the following command:
    podman run --name asap-c -dit -h $podman_HOSTNAME -p $WEBLOGIC_PORT -v $ASAP_VOLUME:/$ASAP_VOLUME ASAP-BASE-IMAGE

    The container will be created with asap-c.

  7. Enter the ASAP container using the following command:
    podman exec -it asap-c bash

    You have entered the ASAP container.

    Note:

    Steps 8 to 12 are only applicable when you are upgrading from ASAP 7.4.0.
  8. Run the following command to copy the ANT and DB utility scripts to the container:
    podman cp asap-img-builder/scripts/extract-ant.sh asap-c:/scripts
podman cp asap-img-builder/scripts/upgrade-db-client.sh asap-c:/scripts
  9. Within the container, run the following commands:
    chmod +x /scripts/extract-ant.sh
chmod +x /scripts/upgrade-db-client.sh

    Note:

    Before running the ANT script, you must:
    • Edit the ANT_FILE variable to match the file in $ASAP_VOLUME/installers.
    • Edit line 29 to specify the correct ANT version in the -name field.
  10. Extract the ANT file using the following command:
    /scripts/extract-ant.sh $ASAP_BASE/ant

    where $ASAP_BASE/ant is the location where ANT is installed.

    Note:

    Before running the upgrade-db-client.sh script as the next step, you must:
    • Identify the Release Update patch number from the README.html by extracting the Database Release Updates file.
    • Modify the directory structure in line 102 of upgrade-db-client.sh to include the Release Update patch number. For example, if the RU file name is p35319490_190000_Linux-x86-64 and the patch number identified is 35320081 then the directory structure shall be 35319490/35320081.
    • Update DB_OPATCH_FILE, DB_RU_FILE, and DB_CLIENT_FILE variables in upgrade-db-client.sh to point to the respective files present in the $asap-img-builder/installers directory..
  11. Run the following command to upgrade the Database Client:
    /scripts/upgrade-db-client.sh

  12. To upgrade Java to JDK 21, extract JDK 21 using the following command:

    tar -xvf /$ASAP_VOLUME/installers/jdk-21_linux-x64_bin.tar.gz -C /usr/lib/jvm/java/
  13. Set JAVA_HOME and update PATH by running the following commands:
    
export JAVA_HOME=/usr/lib/jvm/java/jdk-21.0.9 
export PATH=$JAVA_HOME/bin:$PATH 
# Persist for future shells: 
echo 'export JAVA_HOME=/usr/lib/jvm/java/jdk-21.0.8' >> ~/.bashrc 
echo 'export PATH=$JAVA_HOME/bin:$PATH' >> ~/.bashrc

  14. Remove the existing JDK 8. For example, if jdk1.8.0_441 is present, run the following commands: 

    cd /usr/lib/jvm/java
rm -rf jdk1.8.0_441
  15. Create WebLogic home and set permissions by running the following commands:
    mkdir -p /home/oracle/wls1412
chown -R oracle:oinstall /home/oracle/wls1412
chmod -R 775 /home/oracle/wls1412
  16. Copy the response and inventory files by running the following commands:
    cd /scripts
cat oraInst.loc > /home/oracle/wls1412/oraInst.loc
cat install.file > /home/oracle/wls1412/install.file

  17. Edit /home/oracle/wls1412/oraInst.loc and ensure that it contains: 

    inventory_loc=/home/oracle/wls1412
inst_group=oinstall
  18. Unzip and install WebLogic using JDK 21 by running the following commands:
    export FMW_FILE=V1045131-01.zip
unzip -o /$ASAP_VOLUME/installers/$FMW_FILE -d /$ASAP_VOLUME/installers/
export FMW_JAR=/$ASAP_VOLUME/installers/fmw_14.1.2.0.0_wls.jar
chmod +xr "$FMW_JAR"
  19. Run the installer as the oracle user by using the following command:
    su oracle -c "$JAVA_HOME/bin/java -jar $FMW_JAR
-invPtrLoc /home/oracle/wls1412/oraInst.loc
-silent -responseFile /home/oracle/wls1412/install.file
-jreLoc $JAVA_HOME -ignoreSysPrereqs -force -novalidation
ORACLE_HOME=/home/oracle/wls1412 INSTALL_TYPE="WebLogic Server""
  20. Ensure that both the WebLogic and ASAP servers are stopped.
  21. Create a WebLogic Scripting Tool (WLST) script (for example, reconfigwls.py) with the following commands to upgrade the existing domain:
    # Silent domain re-config for WebLogic 14.1.2
readDomainForUpgrade('/u01/oracle/user_projects/domain/asapDomain')  
updateDomain()
closeDomain()
  22. Run the WLST from the new WebLogic installation by using the following command:
    /home/oracle/wls1412/oracle_common/common/bin/wlst.sh reconfigwls.py
  23. Export the variables WLS_HOME and WL_HOME by using the following commands:
    echo 'export WLS_HOME=/home/oracle/wls1412/wlserver' >> ~/.bashrc
echo 'export PATH=$WLS_HOME/bin:$PATH' >> ~/.bashrc
echo 'export WL_HOME=/home/oracle/wls1412' >> ~/.bashrc
echo 'export PATH=$WL_HOME/bin:$PATH' >> ~/.bashrc
source ~/.bashrc
  24. Verify the WebLogic domain version using the following command:
    cat $WEBLOGIC_DOMAIN/config/config.xml | grep domain-version
  14.1.2.0.0
  25. Insert the following line into $WEBLOGIC_DOMAIN/bin/setDomainEnv.sh before if [ "${WLS_STDERR_LOG}" != "" ] ; then:
     export JAVA_OPTIONS="${JAVA_OPTIONS}
-Djavax.xml.parsers.DocumentBuilderFactory=weblogic.xml.jaxp.RegistryDocumentBuilderFactory
-Djavax.xml.transform.TransformerFactory=weblogic.xml.jaxp.RegistrySAXTransformerFactory
-Djavax.xml.parsers.SAXParserFactory=weblogic.xml.jaxp.RegistrySAXParserFactory
-Dweblogic.oif.serialFilter=java.net.URL 
--add-exports=java.xml/com.sun.org.apache.xalan.internal.res=ALL-UNNAMED
--add-exports=java.xml/com.sun.org.apache.xpath.internal=ALL-UNNAMED
--add-exports=java.xml/com.sun.org.apache.xml.internal.dtm=ALL-UNNAMED
--add-exports=java.xml/com.sun.org.apache.xml.internal.utils=ALL-UNNAMED
--add-exports=java.xml/com.sun.org.apache.xpath.internal.objects=ALL-UNNAMED
--add-exports=java.xml/com.sun.org.apache.xml.internal.dtm.ref=ALL-UNNAMED
--add-exports=java.xml/com.sun.org.apache.xalan.internal.extensions=ALL-UNNAMED
--add-exports=java.xml/com.sun.org.apache.xml.internal.serialize=ALL-UNNAMED
--add-exports=java.xml/com.sun.org.apache.xerces.internal.parsers=ALL-UNNAMED
--add-exports=java.xml/com.sun.org.apache.xerces.internal.dom=ALL-UNNAMED
--add-exports=java.xml/com.sun.org.apache.xalan.internal.xsltc.trax=ALL-UNNAMED
--add-opens=java.xml/com.sun.org.apache.xerces.internal.util=ALL-UNNAMED"
  26. Create a backup archive of the current ASAP installation. For example: ASAP_backup.tar.gz.
  27. Upgrade the ASAP RPM and specify the RPM file name that is copied to the installers directory by using the following command:
    rpm -U --prefix $ASAP_BASE /$ASAP_VOLUME/installers/asap-installer-8.0.0.0.0-26.x86_64.rpm
  28. Update the environment profile by doing the following:
    • Remove legacy JAVA_HOME_ASAP entries.
    • Set JAVA_HOME to the new JDK 21 location.
  29. Start WebLogic in the background in a new shell by running the following commands:
    podman exec -it asap-c bash
cd $WEBLOGIC_DOMAIN/bin/
nohup ./startWebLogic.sh &
  30. Run the following script to update the WebLogic resources after sourcing the Environment_Profile:
    cd $ASAP_BASE
source Environment_Profile
cd $ASAP_BASE/scripts
./asap800.wl.resource.update.sh

    Note:

    WL_HOME should be set to the location of the WebLogic Server 14.1.2 installation.

    Restart WebLogic and verify the JNDI configuration from the 14.1.2 Admin Console.

  31. To upgrade ASAP, in a new shell, update the sample properties by running the following command:
    podman exec -it asap-c bash
vi $ASAP_BASE/config/sampleUpgradeConfiguration.properties
  32. Run the upgrade script:
    cd $ASAP_BASE/scripts
./upgradeASAP -properties ../config/sampleUpgradeConfiguration.properties
  33. Reload the WebLogic which was started earlier in different shell in step 29.
  34. If you are upgrading from 7.4.0 to 8.0 and have the NORTEL_DMS cartridge or any other state table cartridge installed in 7.4.0, you must uninstall it and then install it again. To uninstall and install the cartridge, run the following commands:
    asapd -start -d -host $HOSTNAME -port <DAEMON_PORT>
uninstallCartridge $ASAP_BASE/activationModels/Nortel_DMS_POTS.sar
installCartridge $ASAP_BASE/activationModels/Nortel_DMS_POTS.sar
asapd -stop -d -url $HOSTNAME:<DAEMON_PORT>

    The DAEMON_PORT value is present in $ASAP_BASE/ASAP.properties file.

  35. In a new shell, start ASAP by running the following command:
    cd $ASAP_BASE
source Environment_profile
# Start ASAP 
start_asap_sys -d

    Verify the ASAP server status using the status command.

    Note:

    Ensure that there are a minimum of 5 processes in the output of status command. You should see one process each for Control server, Daemon server, SARM, NEP and JNEP. In ASAP 8.0.0 and 7.4.1, on successful startup, the minimum process count is 5 as compared to 8 processes in 7.3.0 and 7.4.0.
  36. Exit all the shells started in the above steps. Commit the container to image by running the following command:
    podman commit asap-c imagename:version

    For example, podman commit $ASAP_CONTAINER asap-upgraded:8.0.0.0.

  37. Stop and remove the container using the following commands:
    podman stop asap-c 
podman rm asap-c
Upgrading cartridges

To upgrade cartridges, uninstall the previous cartridges and install the new cartridges.

To uninstall and install cartridges:

  1. Repeat the steps 1 to 6 to create the staging container.

    Cartridges are present in the /podmanhost_volume/cartridges container.

  2. Start ASAP and WebLogic Server using the startALL.sh script.
  3. Navigate to the ASAP installation directory using cd $ASAP_BASE.
  4. Source the environment profile using the source Environment_Profile script.
  5. Verify the ASAP server status using the status command.
  6. Uninstall the cartridges. For more information, see "Uninstalling a Cartridge" in ASAP Installation Guide.
  7. Install the cartridges present in the /podmanhost_volume/cartridges directory. For more information, see "Installing a Cartridge" in ASAP Installation Guide.

To upgrade Java, WebLogic Server, and database client, see "About Upgrading ASAP" in ASAP Installation Guide.

Creating an Image from the Staging Container

The staging container is deployed with all the required updates to provision network elements. Save this container as an ASAP image to deploy in the Kubernetes cluster.

To create an image from the staging container:

  1. Run the following command to create an image from the staging container:
    podman commit asap-c imagename:version

    Where version is the version of the ASAP image. This version should be higher than the previous version.

  2. To deploy a new ASAP image in the Kubernetes cluster, the image should be available in the configured registry or on all worker nodes. To push the ASAP image to the Kubernetes registry, run the following commands:
    podman push <imageid> <image-name_repository:tag>
podman tag <imageid> <image-name_repository:tag>
  3. Stop and remove the containers using the following commands:
    podman stop asap-c
podman rm asap-c
  4. Update the ASAP image in the $ASAP_CNTK/charts/asap/values.yaml file.
  5. Create the ASAP instance using the following command:
    $ASAP_CNTK/scripts/create-instance.sh -p sr -i quick

Now the ASAP instance is upgraded successfully.

Order Balancer Cloud Native Upgrade Procedures

To upgrade the Order Balancer Image:
  1. The Cloud Native toolkit includes all the necessary scripts to facilitate the image building and deployment process. Download the latest Cloud Native Toolkit:
    • asap-img-builder.zip: Contains tools for image building.
    • ob-cntk.zip: Contains tools for image deployment.
  2. Delete the running instance using the cleanup script ($OB_CNTK/scripts/delete-instance.sh):
    ./delete-instance.sh
  3. Copy the following installers to $ASAP_IMG_BUILDER/installers:
    • WebLogic Server 14.1.2 installer zip file (for example, V1045131-01.zip)
    • JDK 21 (for example, jdk-21_linux-x64_bin.tar.gz)
    • OB tar file (for example, ASAP.R7_4_0_P4.B4.ob.tar)
  4. Run the following command to copy installers and cartridges to the volume:
    $ASAP_IMG_BUILDER/upgrade-asap-image.sh
  5. Create a new container from the previous Order Balancer base image and mount the volume by running the following command:
    podman run --name ob-c -dit -h <PODMAN_HOSTNAME> -p <WEBLOGIC_PORT> -v <OB_VOLUME>:/<OB_VOLUME> OB-BASE-IMAGE
  6. Run the following command to access the container shell:
    podman exec -it ob-c bash
  7. Run the following command to extract JDK 21:
    tar -xvf /$OB_VOLUME/installers/jdk-21_linux-x64_bin.tar.gz -C /usr/lib/jvm/java/
  8. Set JAVA_HOME and update PATH by running the following commands:
    export JAVA_HOME=/usr/lib/jvm/java/jdk-21.0.9
export PATH=$JAVA_HOME/bin:$PATH
  9. Run the following commands to remove the existing JDK 8 (For example, jdk1.8.0_441):
    cd /usr/lib/jvm/java
rm -rf jdk1.8.0_441
  10. Create the WebLogic Server 14.1.2 home directory and set permissions by running the following commands:
    mkdir -p /home/oracle/wls1412
chown -R oracle:oinstall /home/oracle/wls1412
chmod -R 775 /home/oracle/wls1412
  11. Copy the response and inventory pointer files (as prepared in /scripts) by running the following commands:
    cd /scripts
cat oraInst.loc > /home/oracle/wls1412/oraInst.loc
cat install.file > /home/oracle/wls1412/install.file
  12. Edit /home/oracle/wls1412/oraInst.loc to point to the new location:
    vi /home/oracle/wls1412/oraInst.loc
    Ensure the contents are as follows:
    inventory_loc=/home/oracle/wls1412
inst_group=oinstall
  13. Run the following commands to unpack and install WebLogic Server 14.1.2 using JDK 21:
    export FMW_FILE=V1045131-01.zip
unzip -o /$OB_VOLUME/installers/$FMW_FILE -d /$OB_VOLUME/installers/
export FMW_JAR=/$OB_VOLUME/installers/fmw_14.1.2.0.0_wls.jar
chmod +xr "$FMW_JAR"
su oracle -c "$JAVA_HOME/bin/java -jar /$OB_VOLUME/installers/fmw_14.1.2.0.0_wls.jar \
-invPtrLoc /home/oracle/wls1412/oraInst.loc \
-silent -responseFile /home/oracle/wls1412/install.file \
-jreLoc $JAVA_HOME -ignoreSysPrereqs -force -novalidation \
ORACLE_HOME=/home/oracle/wls1412 INSTALL_TYPE=\"WebLogic Server\""
  14. Ensure that the WebLogic Server and Order Balancer are stopped.
  15. Create a WebLogic Scripting Tool script (for example, reconfigwls.py) by using the following commands to upgrade the existing domain:
    # Silent domain re-config for WebLogic 14.1.2
readDomainForUpgrade('/u01/oracle/user_projects/domains/ob70')
updateDomain()
closeDomain()
  16. Run the WebLogic Scripting Tool from the WebLogic 14.1.2 installation:
    /home/oracle/wls1412/oracle_common/common/bin/wlst.sh reconfigwls.py
  17. Run the following command to point WEBLOGIC_HOME to the WebLogic 14.1.2 installation directory and update your shell:
    echo 'export WEBLOGIC_HOME=/home/oracle/wls1412' >> ~/.bashrc
echo 'export PATH=$WEBLOGIC_HOME/bin:$PATH' >> ~/.bashrc
echo 'export JAVA_HOME=/usr/lib/jvm/java/jdk-21.0.9' >> ~/.bashrc
echo 'export PATH=$JAVA_HOME/bin:$PATH' >> ~/.bashrc
source ~/.bashrc
  18. Run the following commands to verify the WebLogic domain version:
    cat $WEBLOGIC_DOMAIN/config/config.xml | grep domain-version
14.1.2.0.0
  19. Rename the existing OB directory by using the following command:
    mv /scripts/ob ob_backup
  20. Extract the Order Balancer tar from the location provided:
    tar -xvf /$OB_VOLUME/installers/ASAP.R8_0_0.B92.ob.tar -C /scripts
  21. Install necessary dependencies by using the following commands:
    yum install ksh zip

# WebLogic dependencies
yum install -y \
  binutils \
  gcc gcc-c++ glibc glibc-devel \
  libaio libaio-devel \
  libcap \
  libgcc libstdc++ libstdc++-devel \
  libnsl \
  make motif motif-devel \
  openssl-libs \
  sysstat
  22. Navigate to the extracted Order Balancer directory by using the following command:
    cd <ob_installer>/ob
# where 'ob_installer' is the directory that contains the extracted OB files
  23. Uncomment the code listed below the statement "uncomment below variable assignment for non-SSL" in ob/plan/OBPlan.xml.
  24. Start WebLogic in the background by using the following command:
    nohup $WEBLOGIC_DOMAIN/startWebLogic.sh > nohup.out 2>&1 &
  25. Run the upgrade command:
    installOB -properties /$OB_VOLUME/installers/ob/config.properties -d
  26. Commit the container to the image by running the following command:
    podman commit --change 'ENV WEBLOGIC_HOME=/home/oracle/wls1412' ob-c obcn:8.0.0.0.0

    Set WEBLOGIC_HOME to the new 14.1.2 WebLogic Server home.

  27. Stop and remove the container by using the following commands:
    podman stop <OB_CONTAINER>
podman rm <OB_CONTAINER>

    Note:

    • Paths and file names (especially installer versions) may need to be adjusted based on your environment and available artifacts.
    • Replace domain paths and variable values (for example, $WEBLOGIC_DOMAIN, $ASAP_BASE, $ASAP_VOLUME) according to your setup.
    • Set environment parameters like WEBLOGIC_HOME to the new 14.1.2 WebLogic Server home to maintain consistency.

Creating an Image from the Staging Container

The staging container is deployed with all the required updates to route work orders to ASAP instances. Save this container as a podman image to deploy in the Kubernetes cluster.

To create an image from the staging container:

  1. Run the following command to create an image from the staging container:
    podman commit ob-c imagename:version

    Where version is the version of the Order Balancer image. This version should be higher than the previous version.

  2. To deploy the new Order Balancer image in the Kubernetes cluster, the image should be available in the configured registry or on all worker nodes. To push the image to the Kubernetes registry, run the following commands:
    podman images | grep image name 
podman push <imageid> <image-name_repository:tag>
podman tag <imageid> <image-name_repository:tag>
  3. Stop and remove the containers using the following commands:
    podman stop ob-c
podman rm ob-c
  4. Update the Order Balancer image in the $OB_CNTK/charts/ob/values.yaml file.
  5. Create the Order Balancer instance using the following command:
    $OB_CNTK/scripts/create-instance.sh -p sr -i quick

Now the Order Balancer instance is upgraded successfully.

Upgrades to Infrastructure

From the point of view of ASAP instances, upgrades to the cloud infrastructure fall into two categories:
  • Rolling upgrades
  • One-time upgrades

Note:

All infrastructure upgrades must continue to meet the supported types and versions listed in the ASAP documentation's certification statement.

Rolling upgrades are where, with proper high-availability planning (like anti-affinity rules), the instance as a whole remains available as parts of it undergo temporary outages. Examples of this are Kubernetes worker node OS upgrades, Kubernetes version upgrades and Podman version upgrades.

One-time upgrades affect a given instance all at once. The instance as a whole suffers either an operational outage or a control outage. Examples of this is Ingress controller upgrade.

Kubernetes and Podman Infrastructure Upgrades

Follow standard Kubernetes and Podman practices to upgrade these components. The impact at any point should be limited to one node - master (Kubernetes and OS) or worker (Kubernetes, OS, and Podman). If a worker node is going to be upgraded, drain and cordon the node first. This will result in all pods moving away to other worker nodes. This is assuming your cluster has the capacity for this - you may have to temporarily add a worker node or two. For ASAP instances, any pods on the cordoned worker will suffer an outage until they come up on other workers. However, their messages and orders are redistributed to surviving pods and processing continues at a reduced capacity until the affected pods relocate and initialize. As each worker undergoes this process in turn, pods continue to terminate and start up elsewhere, but as long as the instance has pods in both affected and unaffected nodes, it will continue to process orders.

Ingress Controller Upgrade

Follow the documentation of your chosen Ingress Controller to perform an upgrade. Depending on the Ingress Controller used and its deployment in your Kubernetes environment, the ASAP instances it serves may see a wide set of impacts, ranging from no impact at all (if the Ingress Controller supports a clustered approach and can be upgraded that way) to a complete outage.

The new Traefik can be installed into a new name space, and one-by-one, projects can be unregistered from the old Traefik and registered with the new Traefik.

export TRAEFIK_NS=old-namespace $ASAP_CNTK/scripts/unregister-namespace -p project -t traefik 
export TRAEFIK_NS=new-namespace $ASAP_CNTK/scripts/register-namespace -p project -t traefik

During this transition, there will be an outage in terms of the outside world interacting with ASAP. Any data that flows through the ingress will be blocked until the new Traefik takes over. This includes GUI traffic, order injection, API queries, and SAF responses from external systems. This outage will affect all the instances in the project being transitioned.

Miscellaneous Upgrade Procedures

This section describes miscellaneous upgrade scenarios.

Network File System (NFS)

If an instance is created successfully, but a change to the NFS configuration is required, then the change cannot be made to a running ASAP instance. In this case, the procedure is as follows:

  1. Delete the ASAP instance.
  2. Update the nfs details in the pv.yaml and pvc.yaml files.
  3. Start the instance.