Verrazzano Enterprise Container Platform – Upgrade Verrazzano

Docs: Prepare to Upgrade Verrazzano

Mon, 01 Jan 0001 00:00:00 +0000

A Verrazzano installation consists of a stack of components, such as cert-manager, where each component has a specific release version that may be different from the overall Verrazzano version. The Verrazzano platform operator knows the versions of each component associated with the Verrazzano version. When you perform the initial Verrazzano installation, the appropriate version of each component is installed by the platform operator. Post installation, it may be necessary to update one or more of the component images or Helm charts. This update is also handled by the platform operator and is called an upgrade. Currently, Verrazzano does only patch-level upgrades, where a helm upgrade command can be issued for the component. Typically, patch-level upgrades simply replace component images with newer versions.

Application and system pod restarts

If Verrazzano has a new version of Istio, then all the pods with Istio proxy sidecars need to be restarted. This is done so that the new version of the proxy sidecar can be injected into the pods. All Verrazzano system pods containing Istio proxy sidecars will be restarted. Additionally, OAM application workloads with old versions of the Istio proxy sidecar will be restarted as well. Verrazzano will perform a rolling restart of all OAM workloads, except WebLogic workloads with an Istio proxy sidecar that is out of date by two minor versions or more. For these workloads, Verrazzano will stop the WebLogic domain prior to upgrade and restart it after the upgrade has been completed. This may result in downtime during upgrade for these WebLogic workloads. There is no user involvement related to restarting OAM applications; it is done automatically during upgrade.

Docs: Upgrade Verrazzano

Mon, 01 Jan 0001 00:00:00 +0000

It is important to distinguish between updating the Verrazzano platform operator versus upgrading the Verrazzano installation. The platform operator contains the newer component charts and image versions, so it must be updated prior to upgrading the installation. Updating the platform operator has no effect on an existing installation until you initiate the Verrazzano installation upgrade. Currently, there is no way to roll back either the platform operator update or the Verrazzano installation upgrade.

NOTE: For upgrading to Verrazzano v1.7 or later, before initiating an upgrade, we recommend that you back up your existing OpenSearch cluster using Velero. For the backup process, follow the Backup documentation for your existing Verrazzano version, not the version to which you are upgrading. Also, before you start the upgrade, we advise that you make sure that the OpenSearch cluster health is green.

You can upgrade Verrazzano using the Verrazzano CLI or with kubectl. See the following respective sections.

NOTE

For optimal functionality, be sure to install or upgrade the CLI version to match the Verrazzano version to which you are upgrading.

Upgrade Verrazzano using the CLI

In one simple step, you can upgrade to a specified version of Verrazzano using this command.

Update the Verrazzano resource to the desired version.

To update to the latest version (default):
```
$ vz upgrade
```
To update to a specific version, where <version> is the desired version:
```
$ vz upgrade --version <version>
```
Wait for the upgrade to complete. Upgrade logs will be streamed to the command window until the upgrade has completed or until the default timeout (30m) has been reached.

Upgrade using `kubectl`

Upgrading an existing Verrazzano installation using kubectl is a two-step process:

Update the Verrazzano platform operator to the Verrazzano release version to which you want to upgrade.
Upgrade the Verrazzano installation.

Update the Verrazzano platform operator

In order to upgrade an existing Verrazzano installation, you must first update the Verrazzano platform operator.

Update the Verrazzano platform operator.

NOTE: If you are using a private container registry, then to update the platform operator, follow the instructions at Install Verrazzano in a Disconnected Environment.

To update to the latest version:
```
$ kubectl apply -f https://github.com/verrazzano/verrazzano/releases/download/v1.7.2/verrazzano-platform-operator.yaml
```
To update to a specific version, where <version> is the desired version:
```
# To update to the desired version:
$ kubectl apply -f https://github.com/verrazzano/verrazzano/releases/download/<version>/verrazzano-platform-operator.yaml
```
Wait for the deployment to complete.
```
$ kubectl -n verrazzano-install rollout status deployment/verrazzano-platform-operator
```
```
# Expected response
deployment "verrazzano-platform-operator" successfully rolled out
```
Confirm that the operator pod is correctly defined and running.
```
$ kubectl -n verrazzano-install get pods
```
```
# Sample output
NAME                                            READY   STATUS    RESTARTS   AGE
verrazzano-platform-operator-59d5c585fd-lwhsx   1/1     Running   0          114s
```

Upgrade Verrazzano

To upgrade the Verrazzano installation, you need to change the version of your installed Verrazzano resource to the version supported by the Verrazzano platform operator.

NOTE: You may only change the version field during an upgrade; changes to other fields or component configurations are not supported at this time.

In one simple step, you can upgrade to a specified version of Verrazzano using this command.

$ kubectl patch vz example-verrazzano -p '{"spec":{"version":"v1.7.2"}}' --type=merge

Alternatively, you can upgrade the Verrazzano installation using the following steps.

Update the Verrazzano resource to the desired version.

To upgrade the Verrazzano components, you must update the version field in your Verrazzano resource specification to match the version supported by the platform operator to which you upgraded and apply it to the cluster.

The value of the version field in the resource specification must be a Semantic Versioning value corresponding to a valid Verrazzano release version.

To update the resource, do one of the following:

a. Edit the YAML file you used to install Verrazzano and set the version field to the latest version.

For example, to upgrade to v1.7.2, your YAML file should be edited to add or update the version field.
```
apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
   profile: dev
   version: v1.7.2
```
Then, apply the resource to the cluster (if you have not edited the resource in-place using kubectl edit).
```
$ kubectl apply -f example-verrazzano.yaml
```
b. Edit the Verrazzano resource directly using kubectl and set the version field directly, for example:
```
$ kubectl edit verrazzano example-verrazzano
# In the resource editor, add or update the version field to "version: v1.7.2", then save.
```
Wait for the upgrade to complete.
```
$ kubectl wait \
    --timeout=30m \
    --for=condition=UpgradeComplete verrazzano/example-verrazzano
```

Docs: Verify the Upgrade

Mon, 01 Jan 0001 00:00:00 +0000

To see detailed progress of the upgrade, view the logs with the following command.

$ kubectl logs -n verrazzano-install \
    -f $(kubectl get pod \
    -n verrazzano-install \
    -l app=verrazzano-platform-operator \
    -o jsonpath="{.items[0].metadata.name}") | grep '^{.*}$' \
    | jq -r '."@timestamp" as $timestamp | "\($timestamp) \(.level) \(.message)"'

Check that all the pods in the verrazzano-system namespace are in the Running state. While the upgrade is in progress, you may see some pods terminating and restarting as newer versions of components are applied, for example:

$ kubectl get pods -n verrazzano-system

# Sample output
coherence-operator-866798c99d-r69xt                1/1     Running   1          43m
fluentd-f9fbv                                      2/2     Running   0          38m
fluentd-n79c4                                      2/2     Running   0          38m
fluentd-xslzw                                      2/2     Running   0          38m
oam-kubernetes-runtime-56cdb56c98-wn2mb            1/1     Running   0          43m
verrazzano-application-operator-7c95ddd5b5-7xzmn   1/1     Running   0          42m
verrazzano-authproxy-594d8c8dcd-llmlr              2/2     Running   0          38m
verrazzano-console-74dbf97fdf-zxvvn                2/2     Running   0          38m
verrazzano-monitoring-operator-6fcf8484fd-gfkhs    1/1     Running   0          38m
verrazzano-operator-66c8566f95-8lbs6               1/1     Running   0          38m
vmi-system-grafana-799d79648d-wsdp4                2/2     Running   0          38m
vmi-system-kiali-574c6dd94d-f49jv                  2/2     Running   0          41m
weblogic-operator-7b447fdb47-wlw64                 2/2     Running   0          42m

Check that the pods in your application namespaces are ready, for example:

$ kubectl get pods -n todo-list

# Sample output
NAME                     READY   STATUS    RESTARTS   AGE
mysql-67575d8954-d4vkm   2/2     Running   0          39h
tododomain-adminserver   4/4     Running   0          39h

Docs: Upgrade Verrazzano in a Multicluster Environment

Mon, 01 Jan 0001 00:00:00 +0000

Each cluster of a multicluster environment is upgraded separately. Follow the Upgrade Verrazzano instructions, starting with the admin cluster and then for each managed cluster.

Verify the upgrade of each managed cluster

For each managed cluster, follow the instructions in each of the following sections:

Docs: Upgrade Kubernetes With Verrazzano Installed

Mon, 01 Jan 0001 00:00:00 +0000

After Verrazzano is installed in a cluster, you may want to upgrade the Kubernetes version of that cluster. For information on the Kubernetes versions that Verrazzano supports, see the Prerequisites.

The following lists documentation sources for Kubernetes cluster updates. If yours does not appear on this list, then consult your cluster provider’s upgrade documentation.

Upgrade a multinode cluster

For a typical multinode Kubernetes cluster, we recommend keeping one or more nodes present and available while upgrading nodes. This allows the cordoned nodes to distribute the pods to available nodes, which eliminates downtime during an upgrade. Your cluster provider can provide information on maintaining node availability for an in-place upgrade.

Upgrade a single-node cluster

For a single-node cluster upgrade, there will be downtime in the cluster to allow the node to cordon while it is upgraded. For this reason, there are a few manual workaround steps that you may need to perform to be able to fully drain the Kubernetes node.

Disable the MySQL pod disruption budget

The MySQL Operator deploys a Pod Disruption Budget for the MySQL database pods. This Pod Disruption Budget will prevent the node from being drained. To change this, you can patch the Pod Disruption budget to allow the MySQL replicas to be drained from the node.

$ kubectl patch poddisruptionbudgets.policy -n keycloak mysql-pdb -p '{"spec":{"minAvailable":0, "maxUnavailable":null}}' --type=merge

Remove the MySQL pod finalizers

It is possible that the MySQL pods will be stuck in the Terminating state while the node is being drained. If you find that the MySQL pod will not complete termination, then you can remove the finalizers to manually terminate these pods.

$ kubectl patch pod -n keycloak mysql-0 -p '{"metadata":{"finalizers":null}}' --type=merge

Delete the Rancher Helm pods

Rancher spins up Helm pods for cluster operations. Because these pods are not managed by any parent resources, they can prevent the node from being drained. If this is the case, then you can delete these pods with the following command.

$ kubectl get pods --no-headers=true -n cattle-system | awk '{print $1}' | grep helm | xargs kubectl delete pod --ignore-not-found -n cattle-system

Verrazzano Enterprise Container Platform – Upgrade Verrazzano

Docs: Prepare to Upgrade Verrazzano

Application and system pod restarts

Docs: Upgrade Verrazzano

NOTE

Upgrade Verrazzano using the CLI

Upgrade using kubectl

Update the Verrazzano platform operator

Upgrade Verrazzano

Docs: Verify the Upgrade

Docs: Upgrade Verrazzano in a Multicluster Environment

Verify the upgrade of each managed cluster

Docs: Upgrade Kubernetes With Verrazzano Installed

Upgrade a multinode cluster

Upgrade a single-node cluster

Disable the MySQL pod disruption budget

Remove the MySQL pod finalizers

Delete the Rancher Helm pods

Upgrade using `kubectl`