Note:
- This tutorial requires access to Oracle Cloud. To sign up for a free account, see Get started with Oracle Cloud Infrastructure Free Tier.
- It uses example values for Oracle Cloud Infrastructure credentials, tenancy, and compartments. When completing your lab, substitute these values with ones specific to your cloud environment.
Deploy Monitoring Solution to Private OKE Clusters Using Oracle Cloud Infrastructure Resource Manager
Introduction
Oracle Cloud offers a complementary monitoring solution for your Oracle Cloud Infrastructure Container Engine for Kubernetes (OKE) implementations, using Oracle Cloud Infrastructure (OCI) Logging Analytics. This solution can be found in the documentation and can be implemented as a Resource Manager Terraform deployment available on GitHub. However, this solution is addressed to OKE clusters with public API endpoints.
If you attempt the same deployment on a private OKE cluster, the deployment fails because of network connectivity issues, as OCI Resource Manager jobs are executing on a service tenancy. We can use OCI Resource Manager Private endpoints in this situation, but we need to make a few changes to the Terraform code so the Helm deployment can run.
Objectives
Assuming you have already created a private OKE cluster (all cluster components located in private subnets), at this point deploying the above Logging Analytics solution through the OCI Resource Manager will fail with the following error:
Error: Kubernetes cluster unreachable: Get "https://192.0.2.1:6443/version": dial tcp 192.0.2.1:6443: i/o timeout
with module.helm_release.helm_release.oci-kubernetes-monitoring[0],
on modules/helm/helm.tf line 11, in resource "helm_release" "oci-kubernetes-monitoring"
11: resource "helm_release" "oci-kubernetes-monitoring"
In this case, the IP 192.0.2.1 is the IP of the OKE private endpoint, which is unreachable to the OCI Resource Manager.
To overcome this problem, the following supplementary steps need to be done:
- Create OCI Resource Manager private endpoint. This private endpoint will create the connection to the cluster.
- Add an ingress rule for the class E IP specific to the OCI Resource Manager private endpoint to the Security List of the private subnet of the OKE API endpoint.
- Use that endpoint in the Terraform code in the OCI Resource Manager.
- Resolve subsequent certificates error that appears as a side effect of these changes
Prerequisites
- Basic knowledge of OKE, OCI Resource Manager, and OCI Networking.
- Understanding of Terraform.
- User privileges to be able to create and access OCI Resource Manager and Networking resources.
Task 1: Create Resource Manager Private Endpoint
For this task, you can follow these steps, to create a new Private Endpoint for the OCI Resource Manager from the OCI Console. Make sure to select the VCN and subnet of the OKE cluster private API endpoint.
This way, we map the OKE cluster to this OCI Resource Manager private endpoint, but it is still not reachable from the outside.
Task 2: Add an ingress rule in the private subnet security list
Add an ingress rule to the security list of the private API endpoint subnet, in order to allow traffic from the OCI Resource Manager private endpoint. The reachable IP of the OCI Resource Manager private endpoint is a class E IP, meaning that the CIDR 240.0.0.0/4 will cover this.
Make sure to check the rest of the rules, ingress and egress, comply with the OCI recommendations.
Task 3: Make changes in the Terraform code
In order to use this new reachable IP of the private endpoint created in the previous steps, you need to modify the file /modules/helm/provider.tf from the stack.
Include the below data source block which gives the alternate reachable IP address of the private resource. This IP will be used by the OCI Resource Manager Service to connect to the private resource.
data "oci_resourcemanager_private_endpoint_reachable_ip" "test_private_endpoint_reachable_ip" {
#Required
private_endpoint_id = "ocid1.ormprivateendpoint.oc1....."
private_ip = "192.0.2.1"
}
locals {
// following locals are set as "place-holder" when user opts out of helm release
cluster_endpoint = "https://${data.oci_resourcemanager_private_endpoint_reachable_ip.test_private_endpoint_reachable_ip.ip_address}:6443"
...
}
Task 4: Resolve certificates error
After the above changes, the code apply will progress, but will subsequently fail because of the TLS certificate of the API endpoint, since the OKE private endpoint reachable IP is not included in the OKE certificate.
module.helm_release.helm_release.oci-kubernetes-monitoring[0]: Creating...
Error: Kubernetes cluster unreachable: Get "https://255.x.y.0:6443/version": x509: certificate is valid for 192.0.2.1, 192.0.2.2, 192.0.2.3, 127.0.0.1, not 255.x.y.0
This can be avoided by eliminating the certificate verification from the code with the below two settings:
-
Comment or delete or set the below parameter to empty in the same provider.tf file:
# cluster_ca_certificate = local.cluster_ca_certificate
-
Add the following line:
insecure = true
Related Links
-
Monitor Kubernetes and OKE clusters with OCI Logging Analytics
-
Network Resource Configuration for Cluster Creation and Deployment
Acknowledgments
- Adina Nicolescu (Senior Cloud Engineer), Andrei Ilas (Principal Cloud Architect)
More Learning Resources
Explore other labs on docs.oracle.com/learn or access more free learning content on the Oracle Learning YouTube channel. Additionally, visit education.oracle.com/learning-explorer to become an Oracle Learning Explorer.
For product documentation, visit Oracle Help Center.
Deploy Monitoring Solution to Private OKE Clusters Using Oracle Cloud Infrastructure Resource Manager
F83051-01
July 2023
Copyright © 2023, Oracle and/or its affiliates.