Encrypting Kubernetes Secrets at Rest in Etcd

The control plane nodes in a Kubernetes cluster store sensitive configuration data (such as authentication tokens, passwords, and SSH keys) as Kubernetes secret objects in etcd. Etcd is an open source distributed key-value store that Kubernetes uses for cluster coordination and state management. In the Kubernetes clusters created by Container Engine for Kubernetes, etcd writes and reads data to and from block storage volumes in the Oracle Cloud Infrastructure Block Volume service. Although the data in block storage volumes is encrypted, Kubernetes secrets at rest in etcd itself are not encrypted by default.

For additional security, when you create a new cluster you can specify that Kubernetes secrets at rest in etcd are to be encrypted using the Oracle Cloud Infrastructure Vault service (see Overview of Vault). Before you can create a cluster where Kubernetes secrets are encrypted in the etcd key-value store, you have to:

  • know the name and OCID of a suitable master encryption key in Vault
  • create a dynamic group that includes all clusters in the compartment in which you are going to create the new cluster
  • create a policy authorizing the dynamic group to use the master encryption key

Having created the cluster and specified that you want Kubernetes secrets at rest in the etcd key-value store to be encrypted, you can optionally restrict the use of the master encryption key by modifying the dynamic group to include just that cluster.

Note the following:

  • You can only select the option to encrypt the Kubernetes secrets in the cluster's etcd key-value store when creating a new cluster in the 'Custom Create' workflow. You cannot encrypt Kubernetes secrets in the cluster's etcd key-value store when creating a new cluster in the 'Quick Create' workflow. And you cannot encrypt Kubernetes secrets in the etcd key-value stores of clusters you previously created in the 'Custom Create' workflow.
  • You can only select the option to encrypt Kubernetes secrets in the cluster's etcd key-value store if you specify Kubernetes version 1.13.x or later as the version of Kubernetes to run on the control plane nodes of the cluster.
  • Policies must have been defined to authorize Container Engine for Kubernetes to use the master encryption key, and to authorize users to delegate key usage to Container Engine for Kubernetes in the first place. For more information, see Let a user group delegate key usage in a compartment and Let Block Volume, Object Storage, File Storage, and Container Engine for Kubernetes services encrypt and decrypt volumes, buckets, file systems, and Kubernetes secrets in Common Policies.
  • After you've specified a master encryption key for a new cluster and created the cluster, do not subsequently delete the master encryption key in the Vault service. As soon as you schedule a key for deletion in Vault, the Kubernetes secrets stored for the cluster in etcd become inaccessible. If you have already scheduled the key for deletion, it might still be in the Pending Deletion state. If that is the case, cancel the scheduled key deletion (see To cancel the deletion of a key) to restore access to the Kubernetes secrets. If you allow the scheduled key deletion operation to complete and the master encryption key to be deleted, the Kubernetes secrets stored for the cluster in etcd are permanently inaccessible. As a result, cluster upgrades will fail. In this situation, you have no choice but to delete and recreate the cluster.

Master Encryption Keys in Other Tenancies

You can create a cluster in one tenancy that uses a master encryption key in a different tenancy. In this case, you have to write cross-tenancy policies to enable the cluster in its tenancy to access the master encryption key in the Vault service's tenancy. Note that if you want to create a cluster and specify a master encryption key that's in a different tenancy, you cannot use the Console to create the cluster.

For example, assume the cluster is in the ClusterTenancy, and the master encryption key is in the KeyTenancy. Users belonging to a group (OKEAdminGroup) in the ClusterTenancy have permissions to create clusters. A dynamic group (OKEAdminDynGroup) has been created in the cluster, with the rule ALL {resource.type = 'cluster', resource.compartment.id = 'ocid1.compartment.oc1..<unique_ID>'}, so all clusters created in the ClusterTenancy belong to the dynamic group.

In the root compartment of the KeyTenancy, the following policies:

  • use the ClusterTenancy's OCID to map ClusterTenancy to the alias OKE_Tenancy
  • use the OCIDs of OKEAdminGroup and OKEAdminDynGroup to map them to the aliases RemoteOKEAdminGroup and RemoteOKEClusterDynGroup respectively
  • give RemoteOKEAdminGroup and RemoteOKEClusterDynGroup the ability to list, view, and perform cryptographic operations with a particular master key in the KeyTenancy
Define tenancy OKE_Tenancy as ocid1.tenancy.oc1..<unique_ID>
Define dynamic-group RemoteOKEClusterDynGroup as ocid1.dynamicgroup.oc1..<unique_ID>
Define group RemoteOKEAdminGroup as ocid1.group.oc1..<unique_ID>
Admit dynamic-group RemoteOKEClusterDynGroup of tenancy ClusterTenancy to use keys in tenancy where target.key.id = 'ocid1.key.oc1..<unique_ID>'
Admit group RemoteOKEAdminGroup of tenancy ClusterTenancy to use keys in tenancy where target.key.id = 'ocid1.key.oc1..<unique_ID>'

In the root compartment of the ClusterTenancy, the following policies:

  • use the KeyTenancy's OCID to map KeyTenancy to the alias KMS_Tenancy
  • give OKEAdminGroup and OKEAdminDynGroup the ability to use master keys in the KeyTenancy
  • allow OKEAdminDynGroup to use a specific master key obtained from the KeyTenancy in the ClusterTenancy
Define tenancy KMS_Tenancy as ocid1.tenancy.oc1..<unique_ID>
Endorse group OKEAdminGroup to use keys in tenancy KMS_Tenancy
Endorse dynamic-group OKEAdminDynGroup to use keys in tenancy KMS_Tenancy
Allow dynamic-group OKEAdminDynGroup to use keys in tenancy where target.key.id = 'ocid1.key.oc1..<unique_ID>'

See Accessing Object Storage Resources Across Tenancies for more examples of writing cross-tenancy policies.

Having entered the policies, you can now run a command similar to the following to create a cluster in the ClusterTenancy that uses the master key obtained from the KeyTenancy:
oci ce cluster create --name oke-with-cross-kms --kubernetes-version v1.16.8 --vcn-id ocid1.vcn.oc1.iad.<unique_ID> --service-lb-subnet-ids '["ocid1.subnet.oc1.iad.<unique_ID>"]' --compartment-id ocid1.compartment.oc1..<unique_ID> --kms-key-id ocid1.key.oc1.iad.<unique_ID>

Using the Console

To create a new cluster in the 'Custom Create' workflow where Kubernetes secrets are encrypted in the cluster's etcd key-value store:

  1. Log in to the Console.
  2. If you know the OCID of the master encryption key to use to encrypt Kubernetes secrets, go straight to the next step. Otherwise:

    • If a suitable master encryption key already exists in Vault but you're not sure of its OCID, follow the instructions in To view key details and make a note of the master encryption key's OCID.
    • If a suitable master encryption key does not already exist in Vault, follow the instructions in To create a new master encryption key to create one. Having created a new master encryption key, make a note of its OCID.
  3. Create a new dynamic group containing all the clusters in the compartment in which you intend to create the new cluster:

    1. Open the navigation menu and click Identity & Security. Under Identity, click Dynamic Groups.
    2. Follow the instructions in To create a dynamic group, and give the dynamic group a name (for example, acme-oke-kms-dyn-grp).
    3. Enter a rule that includes all clusters in the compartment in the format:

      ALL {resource.type = 'cluster', resource.compartment.id = '<compartment-ocid>'}

      where <compartment-ocid> is the OCID of the compartment in which you intend to create the new cluster.

      For example:

      ALL {resource.type = 'cluster', resource.compartment.id = 'ocid1.compartment.oc1..aaaaaaaa23______smwa'}
    4. Click Create Dynamic Group.

    Having created a dynamic group that includes all clusters in the compartment, you can now create a policy to give the dynamic group access to the master encryption key in Vault.

  4. Create a new policy to enable use of the master encryption key:

    1. Open the navigation menu and click Identity & Security. Under Identity, click Policies.
    2. Follow the instructions in To create a policy, and give the policy a name (for example, acme-oke-kms-dyn-grp-policy).
    3. Enter a policy statement to give the dynamic group access to the master encryption key, in the format:

      Allow dynamic-group <dynamic-group-name> to use keys in compartment <compartment-name> where target.key.id = '<key-OCID>'

      where:

      • <dynamic-group-name> is the name of the dynamic group you created earlier.
      • <compartment-name> is the name of the compartment containing the master encryption key.
      • <key-OCID> is the OCID of the master encryption key in Vault.

      For example:

      Allow dynamic-group <acme-oke-kms-dyn-grp> to use keys in compartment acme-kms-key-compartment where target.key.id = 'ocid1.key.oc1.iad.annrl______trfg'
    4. Click Create to create the new policy.
  5. Follow the instructions to create a new cluster in Using the Console to create a Cluster with Explicitly Defined Settings in the 'Custom Create' workflow, select the Encrypt Using Customer-Managed Keys option, and select:

    • Choose a Vault in <compartment-name>: The vault that contains the master encryption key, from the list of vaults in the specified compartment. By default, <compartment-name> is the compartment in which you are creating the cluster, but you can select a different compartment by clicking Change Compartment.
    • Choose a Key in <compartment-name>: The name of the master encryption key, from the list of keys in the specified compartment. By default, <compartment-name> is the compartment in which you are creating the cluster, but you can select a different compartment by clicking Change Compartment. Note that you cannot change the master encryption key after the cluster has been created.
  6. (Optional) Having created the cluster, for additional security:

    1. Make a note of the OCID of the new cluster you just created.
    2. Restrict the use of the master encryption key by modifying the dynamic group rule you created earlier to explicitly specify the OCID of the new cluster, rather than all clusters in the compartment. For example:

      resource.id = 'ocid1.cluster.oc1.iad.aaaaaaaaaf______yg5q'