3.5.1 Updating the High Availability cluster

Important

The kubeadm-ha-setup update command is only supported for errata release updates on existing High Availability clusters.

A kubeadm-ha-setup upgrade command for larger upgrades will be provided in a future release. Major release upgrades are not supported at this time.

Errata Release Update Steps

  1. Create a backup for your High Availability cluster before proceeding before proceeding by following the instructions in Section 4.3, “Cluster Backup and Restore”.

  2. On each master node in the cluster, update the kubeadm-ha-setup package:

    # yum update kubeadm-ha-setup
  3. On the master node from which you intend to run the cluster update from, update the required prerequisite packages:

    # yum update kubeadm
  4. If you are using the Oracle Container Registry to obtain images, log in.

    Follow the instructions in Section 3.2.5, “Oracle Container Registry Requirements”. Note that if images are updated on the Oracle Container Registry, you may be required to accept the Oracle Standard Terms and Restrictions again before you are able to perform the update. If you are using one of the Oracle Container Registry mirrors, see Section 3.2.5.1, “Using an Oracle Container Registry Mirror” for more information. If you have configured a local registry, you may need to set the KUBE_REPO_PREFIX environment variable to point to the appropriate registry. You may also need to update your local registry with the most current images for the version that you are upgrading to. See Section 3.2.5.2, “Setting Up an Optional Local Registry” for more information.

  5. Verify that the currently reported node versions match those of the previous package:

    # kubectl get nodes
    NAME                  STATUS   ROLES    AGE     VERSION
    master1.example.com   Ready    master   4m8s    v1.12.5+2.1.1.el7
    master2.example.com   Ready    master   2m25s   v1.12.5+2.1.1.el7
    master3.example.com   Ready    master   2m12s   v1.12.5+2.1.1.el7
    worker1.example.com   Ready    <none>   25s     v1.12.5+2.1.1.el7
  6. Start the scripted update process by using the kubeadm-ha-setup tool:

    # kubeadm-ha-setup update
    [WARNING] This action will update this cluster to the latest version(1.12.7).
    [WARNING] You must take a backup before updating the cluster, as the update may fail. 
    [PROMPT] Do you want to continue updating your cluster?
    Please type Yes/y to confirm or No/n to abort(Case insensitive):
    Y
    Kubernetes Cluster Version: v1.12.5
    Kubeadm version:1.12.7-1.1.2, Kueblet version 1.12.5-2.1.1
    Kubeadm version: 1.12.5-2.1.1 Kubelet version: 1.12.7-1.1.2
    Reading configuration file /usr/local/share/kubeadm/run/kubeadm/ha.yaml ...
    Checking repo access
    [preflight] Running pre-flight checks.
    [upgrade] Making sure the cluster is healthy:
    [upgrade/config] Making sure the configuration is correct:
    [upgrade/config] Reading configuration from the cluster...
    [upgrade/config] FYI: You can look at this config file with 
    'kubectl -n kube-system get cm kubeadm-config -oyaml'
    [upgrade/apply] Respecting the --cri-socket flag that is set 
    with higher priority than the config file.
    [upgrade/version] You have chosen to change the cluster version to "v1.12.7"
    [upgrade/versions] Cluster version: v1.12.5+2.1.1.el7
    [upgrade/versions] kubeadm version: v1.12.7+1.1.2.el7
    [upgrade/prepull] Will prepull images for components 
    [kube-apiserver kube-controller-manager kube-scheduler etcd]
    [upgrade/prepull] Prepulling image for component etcd.
    [upgrade/prepull] Prepulling image for component kube-apiserver.
    [upgrade/prepull] Prepulling image for component kube-controller-manager.
    [upgrade/prepull] Prepulling image for component kube-scheduler.
    [apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-etcd
    [apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-controller-manager
    [apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler
    [apiclient] Found 3 Pods for label selector k8s-app=upgrade-prepull-kube-apiserver
    [apiclient] Found 3 Pods for label selector k8s-app=upgrade-prepull-etcd
    [apiclient] Found 3 Pods for label selector k8s-app=upgrade-prepull-kube-controller-manager
    [apiclient] Found 3 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler
    [upgrade/prepull] Prepulled image for component kube-apiserver.
    [upgrade/prepull] Prepulled image for component kube-controller-manager.
    [upgrade/prepull] Prepulled image for component kube-scheduler.
    [upgrade/prepull] Prepulled image for component etcd.
    [upgrade/prepull] Successfully prepulled the images for all the control plane components
    [upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.12.7"...
    Static pod: kube-apiserver-master1.example.com hash: 
    f9004e982ed918c6303596943cef5493
    Static pod: kube-controller-manager-master1.example.com hash: 
    9590101be574fc0a237ca3f029f03ea2
    Static pod: kube-scheduler-master1.example.com hash: 
    22961405d099beb7721c7598daaa73d6
    [upgrade/staticpods] Writing new Static Pod manifests to 
    "/etc/kubernetes/tmp/kubeadm-upgraded-manifests867609756"
    [controlplane] wrote Static Pod manifest for component kube-apiserver to 
    "/etc/kubernetes/tmp/kubeadm-upgraded-manifests867609756/kube-apiserver.yaml"
    [controlplane] wrote Static Pod manifest for component kube-controller-manager to 
    "/etc/kubernetes/tmp/kubeadm-upgraded-manifests867609756/kube-controller-manager.yaml"
    [controlplane] wrote Static Pod manifest for component kube-scheduler to 
    "/etc/kubernetes/tmp/kubeadm-upgraded-manifests867609756/kube-scheduler.yaml"
    [upgrade/staticpods] Moved new manifest to 
    "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to 
    "/etc/kubernetes/tmp/kubeadm-backup-manifests-2019-04-08-14-28-11/kube-apiserver.yaml"
    [upgrade/staticpods] Waiting for the kubelet to restart the component
    [upgrade/staticpods] This might take a minute or longer depending on 
    the component/version gap (timeout 5m0s
    Static pod: kube-apiserver-master1.example.com hash: f9004e982ed918c6303596943cef5493
    Static pod: kube-apiserver-master1.example.com hash: f9004e982ed918c6303596943cef5493
    Static pod: kube-apiserver-master1.example.com hash: f9004e982ed918c6303596943cef5493
    Static pod: kube-apiserver-master1.example.com hash: a692b9726292a4c2a89e2cdcd8301035
    [apiclient] Found 3 Pods for label selector component=kube-apiserver
    [upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
    [upgrade/staticpods] Moved new manifest to 
    "/etc/kubernetes/manifests/kube-controller-manager.yaml" and 
    backed up old manifest to 
    "/etc/kubernetes/tmp/kubeadm-backup-manifests-2019-04-08-14-28-11/
    kube-controller-manager.yaml"
    [upgrade/staticpods] Waiting for the kubelet to restart the component
    [upgrade/staticpods] This might take a minute or longer depending on 
    the component/version gap (timeout 5m0s
    Static pod: kube-controller-manager-master1.example.com hash: 
    9590101be574fc0a237ca3f029f03ea2
    Static pod: kube-controller-manager-master1.example.com hash: 
    7dbb816a4ac17a9522e761017dcd444c
    [apiclient] Found 3 Pods for label selector component=kube-controller-manager
    [upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
    [upgrade/staticpods] Moved new manifest to 
    "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to 
    "/etc/kubernetes/tmp/kubeadm-backup-manifests-2019-04-08-14-28-11/kube-scheduler.yaml"
    [upgrade/staticpods] Waiting for the kubelet to restart the component
    [upgrade/staticpods] This might take a minute or longer depending on 
    the component/version gap (timeout 5m0s
    Static pod: kube-scheduler-master1.example.com hash: 22961405d099beb7721c7598daaa73d6
    Static pod: kube-scheduler-master1.example.com hash: 980091350a77a7fbcff570589689adc2
    [apiclient] Found 3 Pods for label selector component=kube-scheduler
    [upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
    [uploadconfig] storing the configuration used in 
    ConfigMap "kubeadm-config" in the "kube-system" Namespace
    [kubelet] Creating a ConfigMap "kubelet-config-1.12" in namespace kube-system 
    with the configuration for the kubelets in the cluster
    [kubelet] Downloading configuration for the kubelet from 
    the "kubelet-config-1.12" ConfigMap in the kube-system namespace
    [kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
    [patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to 
    the Node API object "master1.example.com" as an annotation
    [bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to 
    post CSRs in order for nodes to get long term certificate credentials
    [bootstraptoken] configured RBAC rules to allow the csrapprover controller 
    automatically approve CSRs from a Node Bootstrap Token
    [bootstraptoken] configured RBAC rules to allow certificate rotation for 
    all node client certificates in the cluster
    [addons] Applied essential addon: CoreDNS
    [addons] Applied essential addon: kube-proxy
    
    [upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.12.7". Enjoy!
    
    [upgrade/kubelet] Now that your control plane is upgraded, please proceed with 
    upgrading your kubelets if you haven't already done so.
    Loaded plugins: langpacks, ulninfo
    Resolving Dependencies
    --> Running transaction check
    ---> Package kubelet.x86_64 0:1.12.5-2.1.1.el7 will be updated
    ---> Package kubelet.x86_64 0:1.12.7-1.1.2.el7 will be an update
    --> Processing Dependency: conntrack for package: kubelet-1.12.7-1.1.2.el7.x86_64
    --> Running transaction check
    ---> Package conntrack-tools.x86_64 0:1.4.4-4.el7 will be installed
    --> Processing Dependency: libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.1)(64bit) 
    for package: conntrack-tools-1.4.4-4.el7.x86_64
    --> Processing Dependency: libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.0)(64bit) 
    for package: conntrack-tools-1.4.4-4.el7.x86_64
    --> Processing Dependency: libnetfilter_cthelper.so.0(LIBNETFILTER_CTHELPER_1.0)(64bit) 
    for package: conntrack-tools-1.4.4-4.el7.x86_64
    --> Processing Dependency: libnetfilter_cttimeout.so.1()(64bit) 
    for package: conntrack-tools-1.4.4-4.el7.x86_64
    --> Processing Dependency: libnetfilter_cthelper.so.0()(64bit) 
    for package: conntrack-tools-1.4.4-4.el7.x86_64
    --> Processing Dependency: libnetfilter_queue.so.1()(64bit) 
    for package: conntrack-tools-1.4.4-4.el7.x86_64
    --> Running transaction check
    ---> Package libnetfilter_cthelper.x86_64 0:1.0.0-9.el7 will be installed
    ---> Package libnetfilter_cttimeout.x86_64 0:1.0.0-6.el7 will be installed
    ---> Package libnetfilter_queue.x86_64 0:1.0.2-2.el7_2 will be installed
    --> Finished Dependency Resolution
    
    Dependencies Resolved
    
    ================================================================================
     Package                   Arch      Version                Repository     Size
    ================================================================================
    Updating:
     kubelet                   x86_64    1.12.7-1.1.2.el7       ol7_addons        19 M
    Installing for dependencies:
     conntrack-tools           x86_64    1.4.4-4.el7            ol7_latest    186 k
     libnetfilter_cthelper     x86_64    1.0.0-9.el7            ol7_latest     17 k
     libnetfilter_cttimeout    x86_64    1.0.0-6.el7            ol7_latest     17 k
     libnetfilter_queue        x86_64    1.0.2-2.el7_2          ol7_latest     22 k
    
    Transaction Summary
    ================================================================================
    Install             ( 4 Dependent packages)
    Upgrade  1 Package
    
    Total download size: 19 M
    Downloading packages:
    Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
    --------------------------------------------------------------------------------
    Total                                              5.2 MB/s |  19 MB  00:03     
    Running transaction check
    Running transaction test
    Transaction test succeeded
    Running transaction
      Installing : libnetfilter_cthelper-1.0.0-9.el7.x86_64                     1/6 
      Installing : libnetfilter_cttimeout-1.0.0-6.el7.x86_64                    2/6 
      Installing : libnetfilter_queue-1.0.2-2.el7_2.x86_64                      3/6 
      Installing : conntrack-tools-1.4.4-4.el7.x86_64                           4/6 
      Updating   : kubelet-1.12.7-1.1.2.el7.x86_64                              5/6 
      Cleanup    : kubelet-1.12.5-2.1.1.el7.x86_64                              6/6 
      Verifying  : libnetfilter_queue-1.0.2-2.el7_2.x86_64                      1/6 
      Verifying  : libnetfilter_cttimeout-1.0.0-6.el7.x86_64                    2/6 
      Verifying  : kubelet-1.12.7-1.1.2.el7.x86_64                              3/6 
      Verifying  : libnetfilter_cthelper-1.0.0-9.el7.x86_64                     4/6 
      Verifying  : conntrack-tools-1.4.4-4.el7.x86_64                           5/6 
      Verifying  : kubelet-1.12.5-2.1.1.el7.x86_64                              6/6 
    
    Dependency Installed:
      conntrack-tools.x86_64 0:1.4.4-4.el7                                          
      libnetfilter_cthelper.x86_64 0:1.0.0-9.el7                                    
      libnetfilter_cttimeout.x86_64 0:1.0.0-6.el7                                   
      libnetfilter_queue.x86_64 0:1.0.2-2.el7_2                                     
    
    Updated:
      kubelet.x86_64 0:1.12.7-1.1.2.el7                                             
    
    Complete!
    Loaded plugins: langpacks, ulninfo
    Resolving Dependencies
    --> Running transaction check
    ---> Package kubectl.x86_64 0:1.12.5-2.1.1.el7 will be updated
    ---> Package kubectl.x86_64 0:1.12.7-1.1.2.el7 will be an update
    --> Finished Dependency Resolution
    
    Dependencies Resolved
    
    ================================================================================
     Package         Arch           Version                  Repository        Size
    ================================================================================
    Updating:
     kubectl         x86_64         1.12.7-1.1.2.el7         ol7_addons          7.7 M
    
    Transaction Summary
    ================================================================================
    Upgrade  1 Package
    
    Total download size: 7.7 M
    Downloading packages:
    Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
    Running transaction check
    Running transaction test
    Transaction test succeeded
    Running transaction
      Updating   : kubectl-1.12.7-1.1.2.el7.x86_64                              1/2 
      Cleanup    : kubectl-1.12.5-2.1.1.el7.x86_64                              2/2 
      Verifying  : kubectl-1.12.7-1.1.2.el7.x86_64                              1/2 
      Verifying  : kubectl-1.12.5-2.1.1.el7.x86_64                              2/2 
    
    Updated:
      kubectl.x86_64 0:1.12.7-1.1.2.el7                                             
    
    Complete!
    Waiting for the cluster to become healthy
    .Updating remote master nodes
    CreateSSH /root/.ssh/id_rsa root
    Updating the master node:  master2.example.com
    Successfully updated the master node:  master2.example.com
    Updating the master node:  master3.example.com
    Successfully updated the master node:  master3.example.com
    The cluster has been updated successfully
    Please update the worker nodes in your cluster and do the following:
         1. On Master: kubectl drain worker1.example.com --ignore-daemonsets
         2. On Worker1: yum install -y \
    kubeadm-1.12.7-1.1.2.el7 kubelet-1.12.7-1.1.2.el7 \
    kubectl-1.12.7-1.1.2.el7 kubeadm-ha-setup-0.0.2-1.0.21.el7
         3. On Worker1: systemctl restart kubelet
         4. On Master: kubectl uncordon worker1.example.com
         5. Verify the update on master node: kubectl get nodes

    Optionally, you can override the default container registry choice during the errata release update by specifying the --registry option:

    # kubeadm-ha-setup update --registry container-registry-phx.oracle.com
  7. Verify that your master nodes have been updated correctly before proceeding to update the worker nodes:

    # kubectl get nodes
    NAME                  STATUS   ROLES    AGE   VERSION
    master1.example.com   Ready    master   17m   v1.12.7+1.1.2.el7
    master2.example.com   Ready    master   15m   v1.12.7+1.1.2.el7
    master3.example.com   Ready    master   15m   v1.12.7+1.1.2.el7
    worker1.example.com   Ready    <none>   13m   v1.12.5+2.1.1.el7
  8. Use the kubectl tool to drain each of your worker nodes from the cluster:

    # kubectl drain worker1.example.com --ignore-daemonsets
    node/worker1.example.com cordoned

    Check that the worker nodes are unable to accept any further scheduling or new pods:

    # kubectl get nodes

    Note that a node that has been drained should have its status set to SchedulingDisabled.

  9. On each of the worker nodes, upgrade the required packages to the latest versions and restart the kubelet service:

    # yum update kubeadm kubelet kubectl kubeadm-ha-setup
    # systemctl restart kubelet
  10. Now that the upgrades are complete for each worker node, uncordon them using the kubectl tool from the master cluster:

    # kubectl uncordon worker1.example.com
    node/worker1.example.com uncordoned

    Check that the worker nodes are now able to accept new schedules and pods:

    # kubectl get nodes
    NAME                  STATUS   ROLES    AGE   VERSION
    master1.example.com   Ready    master   17m   v1.12.7+1.1.2.el7
    master2.example.com   Ready    master   15m   v1.12.7+1.1.2.el7
    master3.example.com   Ready    master   15m   v1.12.7+1.1.2.el7
    worker1.example.com   Ready    <none>   13m   v1.12.7+1.1.2.el7

Recover from Errata Release Update Failures

If the update fails to complete successfully, you will need to do a full cluster restore from backup. Note that the cluster will not be responsive to new commands until the restore process is complete.

Recovery Steps

  1. Check which of the required packages were updated on each node:

    # yum list installed kubeadm kubelet kubectl
  2. Downgrade each of the individual packages that has already been updated to the previous errata version. For example, to downgrade the kubeadm package:

    # yum downgrade kubeadm
    Note

    Do not downgrade the kubeadm-ha-setup package on your master nodes, as the latest version is always designed to support errata release update recovery.

  3. Follow the restore steps in Section 4.3, “Cluster Backup and Restore”, but add the --force flag to override any version checks:

    # kubeadm-ha-setup restore /backups/master-backup-v1.12.5-2-1544442719.tar --force
  4. When recovery is complete, you may re-attempt the High Availability cluster update.