Replacing Boot Volumes of Worker Nodes

If you have added a self-managed node to a cluster and you subsequently rotate the cluster's credentials, replacing the self-managed node's boot volume requires extra steps (in addition to those described in Replacing Boot Volumes of Worker Nodes).The extra steps are required because the cluster's CA certificate is referenced in the self-managed nodes cloud-init script.

Assume that you have already rotated the CA certificate of the cluster to which a self-managed node has been added, by following the instructions in Rotating Cluster Credentials. If you now want to replace the boot volume of the self-managed node, you have to first modify the self-managed node's cloud-init script. You have to replace the cluster's previous CA certificate referenced in the script with the cluster's current CA certificate. Follow these steps:

1. Obtain details of the compute instance hosting the self-managed node, by entering:

   oci compute instance get --instance-id <instance-ocid>

   For example:

   oci compute instance get --instance-id ocid1.instance.oc1.phx.any______blra

   Details about the compute instance are returned.

2. Locate, and make a note of, the boot volume OCID (shown in source-details in the output of the oci compute instance get command), by entering:

   oci compute instance get --instance-id <instance-ocid> --query 'data."source-details"."boot-volume-id"' --raw-output
   

   For example:

   oci compute instance get --instance-id ocid1.instance.oc1.phx.any______blra --query 'data."source-details"."boot-volume-id"' --raw-output
   

   Example output:

   ocid1.bootvolume.oc1.phx.aaaa______g4a

   You use the boot volume OCID in a later step.

3. Locate the metadata values shown in the output as follows:

   "metadata": {
   "ssh_authorized_keys": "<key-value>",
   "user_data": "<existing-base64-encoded-init-script>"
   }

   Note that ssh_authorized_keys is present when SSH access to the node has been set up (which is usually, but not always, the case).

   For example:

   "metadata": {
   "ssh_authorized_keys": "ssh-rsa AAAA_____ example@acme.com",
   "user_data": "IyEv___1234___PSIK"
   }

   The value of user_data is the cloud-init script that was originally used to add the compute instance to the cluster as a self-managed node, in base64-encoded format. The cloud-init script specifies the cluster's Kubernetes API private endpoint, and the cluster's previous base64-encoded CA certificate.
4. Copy the ssh_authorized_keys and user_data values and save them in a new file named metadata.json, by entering:

   oci compute instance get --instance-id <instance-ocid> --query 'data.metadata' --raw-output > metadata.json

   Example metadata.json file:

   {
   "ssh_authorized_keys": "ssh-rsa AAAA_____ user@acme.com",
   "user_data": "IyEv___1234___PSIK"
   }

5. Copy the value of user_data (the original base64-encoded cloud-init script), and decode it by entering:

   echo "<existing-base64-encoded-init-script>" | base64 --decode

   For example:

   echo "IyEv___1234___PSIK" | base64 --decode

   The decoded output is the original cloud-init script. For example:

   #!/usr/bin/env bash
   bash /etc/oke/oke-install.sh \
     --apiserver-endpoint "10.0.0.12" \
     --kubelet-ca-cert "LS0txxxx______Cg=="

   The value of --kubelet-ca-cert is the cluster's previous CA certificate, in base64-encoded format.

6. Save the decoded output as a text file named my-cloud-init-file.
7. Obtain the cluster's current CA certificate (that is, the cluster's CA certificate after rotation) by entering:

   oci ce cluster create-kubeconfig --cluster-id <cluster-ocid> --region <region-identifier> --file - | grep -oE "LS0t.*"

   The cluster's current base64-encoded CA certificate is returned as a long alphanumeric string, starting with the characters LS0t. For example:

   LS0tyyyy______aP==

8. In the cloud-init script that you saved in my-cloud-init-file, replace the original value of --kubelet-ca-cert with the current base64-encoded CA certificate, and save the file.

   For example:

   #!/usr/bin/env bash
   bash /etc/oke/oke-install.sh \
     --apiserver-endpoint "10.0.0.12" \
     --kubelet-ca-cert "LS0tyyyy______aP=="

9. Base64 encode the updated cloud-init script in my-cloud-init-file. For example, by entering:

   base64 my-cloud-init-file

   The updated cloud-init script is base64-encoded and returned as a long alphanumeric string. For example:

   IyEv___5678___PSIK

10. Update the metadata.json file you created earlier, as follows:

    • If present, keep the value of ssh_authorized_keys unchanged.
    • Change the value of user_data to the base64-encoded alphanumeric string you just created from the updated cloud-init script.

    For example:

    {
          "ssh_authorized_keys": "ssh-rsa AAAA_____ user@acme.com",
          "user_data": "IyEv___5678___PSIK"
    }

11. Use the boot volume OCID that you made a note of earlier, to get details of the existing boot volume, by entering:

    oci bv boot-volume get --boot-volume-id <boot-volume-ocid> --query 'data.{ "image_id": "image-id", "kms_key_id": "kms-key-id", "boot-volume-size-in-gbs": "boot-volume-size-in-gbs" }' --raw-output

    For example:

    oci bv boot-volume get --boot-volume-id ocid1.bootvolume.oc1.phx.aaaa______g4a --query 'data.{ "image_id": "image-id", "kms_key_id": "kms-key-id", "boot-volume-size-in-gbs": "boot-volume-size-in-gbs" }' --raw-output

    Example output:

    {
      "image_id": "ocid1.image.oc1.phx.aaaa____cra",
      "boot-volume-size-in-gbs": 100,
      "kms_key_id": null
    }

12. Use the output to specify the same values for the compute instance source-details properties in a new json file named my-instance-source-details.json, in the following format:

    {
    "image-id": "<returned-bv-image-ocid>",
    "source-type": "image",
    "boot-volume-size-in-gbs": <returned-bv-size>,
    "kms_key_id": <returned-bv-kms-key-ocid>
    "isPreserveBootVolumeEnabled": true
    }

    Note that you do not have to include boot volume properties in the my-instance-source-details.json that are shown as having a null value in the output of the oci bv boot-volume get command. For example, if the boot volume is not encrypted with a user-managed key, the value of kms_key_id is shown as null.

    For example:

    {
    "image-id": "ocid1.image.oc1.phx.aaaa____cra",
    "source-type": "image",
    "boot-volume-size-in-gbs": 100,
    "isPreserveBootVolumeEnabled": true
    }

    Note that you must provide a value for image-id in order to update user_data properties (specifically, to update the cloud-init script with the new CA certificate).

13. Update the details of the compute instance hosting the self-managed node, by entering

    oci compute instance update --instance-id ocid1.instance.oc1.phx.any______blra --metadata file://metadata.json --source-details file://my-instance-source-details.json
     

14. Verify that the compute instance hosting the self-managed node has the updated value for user_data, by entering:

    oci compute instance get --instance-id <instance-ocid>

    For example:

    oci compute instance get --instance-id ocid1.instance.oc1.phx.any______blra

    Updated details about the compute instance hosting the self-managed node are returned.

    {
      "data": {
        ...
        "metadata": {
          "ssh_authorized_keys": "ssh-rsa AAAA_____ user@acme.com",
          "user_data": "IyEv___5678___PSIK"
        ...

    Having updated the user_data value to the base64-encoded cloud-init file containing the new cluster CA certificate, you can now replace the boot volume of the self-managed node.

15. Replace the boot volume of the self-managed node by entering:

    oci ce cluster replace-boot-volume-cluster-node --cluster-id <cluster-ocid> --node-id <instance-ocid> [OPTIONS]

    For example:

    oci ce cluster replace-boot-volume-cluster-node --cluster-id ocid1.cluster.oc1.phx.aaaa______xoq --node-id ocid1.instance.oc1.phx.any______blra --node-eviction-settings "{\"evictionGraceDuration\": \"PT0M\",\"isForceActionAfterGraceDuration\": true}"

    For more information, see Replacing Boot Volumes of Worker Nodes.