Note:

Deploy Monitoring Solution to Private OKE Clusters Using Oracle Cloud Infrastructure Resource Manager

Introduction

Oracle Cloud offers a complementary monitoring solution for your Oracle Cloud Infrastructure Container Engine for Kubernetes (OKE) implementations, using Oracle Cloud Infrastructure (OCI) Logging Analytics. This solution can be found in the documentation and can be implemented as a Resource Manager Terraform deployment available on GitHub. However, this solution is addressed to OKE clusters with public API endpoints.

If you attempt the same deployment on a private OKE cluster, the deployment fails because of network connectivity issues, as OCI Resource Manager jobs are executing on a service tenancy. We can use OCI Resource Manager Private endpoints in this situation, but we need to make a few changes to the Terraform code so the Helm deployment can run.

Working with private endpoints

Objectives

Assuming you have already created a private OKE cluster (all cluster components located in private subnets), at this point deploying the above Logging Analytics solution through the OCI Resource Manager will fail with the following error:

Error: Kubernetes cluster unreachable: Get "https://192.0.2.1:6443/version": dial tcp 192.0.2.1:6443: i/o timeout
  with module.helm_release.helm_release.oci-kubernetes-monitoring[0],
  on modules/helm/helm.tf line 11, in resource "helm_release" "oci-kubernetes-monitoring"
  11: resource "helm_release" "oci-kubernetes-monitoring"

In this case, the IP 192.0.2.1 is the IP of the OKE private endpoint, which is unreachable to the OCI Resource Manager.

To overcome this problem, the following supplementary steps need to be done:

Prerequisites

Task 1: Create Resource Manager Private Endpoint

For this task, you can follow these steps, to create a new Private Endpoint for the OCI Resource Manager from the OCI Console. Make sure to select the VCN and subnet of the OKE cluster private API endpoint.

This way, we map the OKE cluster to this OCI Resource Manager private endpoint, but it is still not reachable from the outside.

OCI Resource Manager Private Endpoint

Task 2: Add an ingress rule in the private subnet security list

SL ingress rule

Add an ingress rule to the security list of the private API endpoint subnet, in order to allow traffic from the OCI Resource Manager private endpoint. The reachable IP of the OCI Resource Manager private endpoint is a class E IP, meaning that the CIDR 240.0.0.0/4 will cover this.

Make sure to check the rest of the rules, ingress and egress, comply with the OCI recommendations.

Task 3: Make changes in the Terraform code

In order to use this new reachable IP of the private endpoint created in the previous steps, you need to modify the file /modules/helm/provider.tf from the stack.

Include the below data source block which gives the alternate reachable IP address of the private resource. This IP will be used by the OCI Resource Manager Service to connect to the private resource.

data "oci_resourcemanager_private_endpoint_reachable_ip" "test_private_endpoint_reachable_ip" {
    #Required
    private_endpoint_id = "ocid1.ormprivateendpoint.oc1....."
    private_ip = "192.0.2.1"
}

locals {
  // following locals are set as "place-holder" when user opts out of helm release
  cluster_endpoint       = "https://${data.oci_resourcemanager_private_endpoint_reachable_ip.test_private_endpoint_reachable_ip.ip_address}:6443"  
  ...
}

Task 4: Resolve certificates error

After the above changes, the code apply will progress, but will subsequently fail because of the TLS certificate of the API endpoint, since the OKE private endpoint reachable IP is not included in the OKE certificate.

module.helm_release.helm_release.oci-kubernetes-monitoring[0]: Creating...
Error: Kubernetes cluster unreachable: Get "https://255.x.y.0:6443/version": x509: certificate is valid for 192.0.2.1, 192.0.2.2, 192.0.2.3, 127.0.0.1, not 255.x.y.0

This can be avoided by eliminating the certificate verification from the code with the below two settings:

Acknowledgments

More Learning Resources

Explore other labs on docs.oracle.com/learn or access more free learning content on the Oracle Learning YouTube channel. Additionally, visit education.oracle.com/learning-explorer to become an Oracle Learning Explorer.

For product documentation, visit Oracle Help Center.