11 Configuring CNC Console to support ASM and OSO

Introduction

Oracle CNCC leverages the Istio or Envoy service mesh (Aspen Service Mesh) for all internal and external communication. The service mesh integration provides inter-NF communication and allows API gateway co-working with service mesh. The service mesh integration supports the services by deploying a special sidecar proxy in the environment to intercept all network and external communication between microservices.

Note:

For ASM installation and configuration, refer Official Aspen Service Mesh website for details.

Predeployment Configuration

Following are the prerequisites to install CNCC with support for ASM:

• Enabling Auto sidecar Injection for Namespace
• Set the connectivity to database (DB) service
• Set the mTLS Connection from Client Browser to ASM
• Create Service Account, Role and Role bindings with ASM annotations
  • Single Cluster Deployment
  • Multi Cluster Deployment

Enabling Auto sidecar Injection for Namespace

This section explains how to enable auto sidecar injection for namespace.

1. Run the following command to enable auto sidecar injection to automatically add the sidecars in all of the pods spawned in CNCC namespace:

   $ kubectl label ns <cncc-namespace> istio-injection=enabled

   Example:

   $ kubectl label ns cncc istio-injection=enabled

Set the Connectivity to Database (DB) Service

Note:

Optional Step:

Creation of Destination-Rule and Service-Entry is applicable only if ASM is not enabled for database. That is, Istio-sidecar is not injected to the database pods.

Following are the steps for VM based DB deployment:

1. Run the following command to create a Headless service for DB connectivity in CNCC namespace:

   $ kubectl apply -f db-connectivity.yaml

   Sample db-connectivity.yaml

   # db-connectivity.yaml
   apiVersion: v1
   kind: Endpoints
   metadata:
     name: cncc-db-connectivity-service-headless
     namespace: <db-namespace>
   subsets:
   - addresses:
     - ip: <10.7x.2xx.xx> # IP Endpoint of DB service.
     ports:
     - port: 3306
       protocol: TCP
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: cncc-db-connectivity-service-headless
     namespace: <db-namespace>
   spec:
     clusterIP: None
     ports:
     - port: 3306
       protocol: TCP
       targetPort: 3306
     sessionAffinity: None
     type: ClusterIP
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: cncc-db-connectivity-service
     namespace: <cncc-namespace>
   spec:
     externalName: cncc-db-connectivity-service-headless.<db-namespace>.svc.<domain>
     sessionAffinity: None
     type: ExternalName

2. Run the following command to create ServiceEntry and DestinationRule for DB connectivity service:

   $ kubectl apply -f db-se-dr.yaml -n <cncc-namespace>

   Sample db-se-dr.yaml

   apiVersion: networking.istio.io/v1alpha3
   kind: ServiceEntry
   metadata:
     name: cncc-db-external-se
     namespace: <cncc-namespace>
   spec:
     exportTo:
     - "."
     hosts:
     - cncc-db-connectivity-service-headless.<db-namespace>.svc.<domain>
     ports:
     - number: 3306
       name: mysql
       protocol: MySQL
     location: MESH_EXTERNAL
     resolution: NONE
   ---
   apiVersion: networking.istio.io/v1alpha3
   kind: DestinationRule
   metadata:
     name: cncc-db-external-dr
     namespace: <cncc-namespace>
   spec:
     exportTo:
     - "."
     host: cncc-db-connectivity-service-headless.<db-namespace>.svc.<domain>
     trafficPolicy:
       tls:
         mode: DISABLE

Set the mTLS Connection from Client Browser to ASM

Prerequisites

Enable certificateCustomFields in ASM values.yaml

Note:

Ensure that ASM is deployed with certificateCustomFields enabled.

ASM values.yaml

global:
 certificateCustomFields: true

Using ASM self-signed CA (Default)

1. ASM creates istio-ca secrets (ca-certs, ca-key) in istio-namespace which contains CA public and private key.
   1. Run the following command to verify if the certificate is created :

      $ kubectl get secrets -n istio-system -o yaml istio-ca-secret

      Note:

      Export the ca-cert.pem, ca-key.pem from the secret istio-ca-secret to your local machine where browser is installed.

      ca-cert.pem → Istio CA public

      ca-key.pem → CA private key

   2. Run the following commands to get ASM Istio CA certificate with base64 decoded and copy the output to a file in you local machine:

   
   kubectl get secret istio-ca-secret -n istio-system -o go-template='{{ index .data "ca-cert.pem" | base64decode}}'
   kubectl get secret istio-ca-secret -n istio-system -o go-template='{{ index .data "ca-key.pem" | base64decode}}'

2. Create client certificate using Openssl commands using ca-cert.pem and ca-key.pem obtained in Step1 and import it to the browser. Refer to your browser specific documentation on how to import certificate and key.
3. Update the browser configuration to trust the CA certificate (ca-cert.pem) obtained from Step 1. Refer to your browser specific documentation on how to trust the CA certificate.

Existing Organization CA

1. Create client certificate using Openssl commands using Organization CA public and private key and import it to the browser. Refer to your browser specific documentation on how to import certificate and key.
2. Update the browser configuration to trust the Organization CA. Refer to your browser specific documentation on how to trust the CA certificate.

Create Service Account, Role and Role bindings with ASM annotations

While creating service account for M-CNCC IAM, M-CNCC Core and A-CNCC Core you need to provide following ASM annotations in the given format:

certificate.aspenmesh.io/customFields:'{"SAN":{"DNS":["<helm-release-name>-ingress-gateway.<cncc_namespace>.svc.<cluster_domain>"]}}'

Sample ASM annotations for M-CNCC-IAM, M-CNCC and A-CNCC

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    certificate.aspenmesh.io/customFields: '{ "SAN": { "DNS": [ "cncc-iam-ingress-gateway.cncc.svc.cluster.local", "cncc-acore-ingress-gateway.cncc.svc.cluster.local", "cncc-mcore-ingress-gateway.cncc.svc.cluster.local"] } }'

Single Cluster Deployment

For Single cluster deployment, where M-CNCC IAM, M-CNCC Core and A-CNCC Core are deployed in same cluster or site can share same service account, role, and rolebinding.

Sample example for M-CNCC IAM, M-CNCC Core and A-CNCC Core| cncc-sa-role-rolebinding.yaml

kubectl apply -n cncc -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cncc-serviceaccount
  labels:
    app.kubernetes.io/component: internal
  annotations:
    sidecar.istio.io/inject: "false"
    "certificate.aspenmesh.io/customFields": '{ "SAN": { "DNS": [ "cncc-iam-ingress-gateway.cncc.svc.cluster.local", "cncc-acore-ingress-gateway.cncc.svc.cluster.local", "cncc-mcore-ingress-gateway.cncc.svc.cluster.local"] } }'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: cncc-role
  labels:
    app.kubernetes.io/component: internal
  annotations:
    sidecar.istio.io/inject: "false"
rules:
- apiGroups:
  - "" # "" indicates the core API group
  resources:
  - services
  - configmaps
  - pods
  - secrets
  - endpoints
  - persistentvolumeclaims
  verbs:
  - get
  - watch
  - list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: cncc-rolebinding
  labels:
    app.kubernetes.io/component: internal
  annotations:
    sidecar.istio.io/inject: "false"
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: cncc-role
subjects:
- kind: ServiceAccount
  name: cncc-serviceaccount
---
EOF  

Multi Cluster Deployment

For Multi cluster deployment,

1. Where M-CNCC IAM, M-CNCC Core and A-CNCC Core are deployed in same site/cluster can share same service account, role and rolebinding.
   See the above single cluster service account example.

   Note:

   In multi cluster deployment A-CNCC Core is an optional component in manager cluster, make sure to take out "cncc-ccore-ingress-gateway.cncc.svc.cluster.local" from "certificate.aspenmesh.io/customFields" in case A-CNCC Core is not deployed in manager cluster
2. Where M-CNCC IAM and M-CNCC Core which are in same cluster can still share same service account, role and rolebinding and A-CNCC deployed in different cluster, a separate service account, role and rolebinding needs to be created.
   Example for M-CNCC IAM, M-CNCC Core | cncc-sa-role-rolebinding.yaml:

   kubectl apply -n cncc -f - <<EOF
   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: cncc-serviceaccount
     labels:
       app.kubernetes.io/component: internal
     annotations:
       sidecar.istio.io/inject: "false"
       "certificate.aspenmesh.io/customFields": '{ "SAN": { "DNS": [ "cncc-iam-ingress-gateway.cncc.svc.cluster.local", "cncc-mcore-ingress-gateway.cncc.svc.cluster.local"] } }'
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: cncc-role
     labels:
       app.kubernetes.io/component: internal
     annotations:
       sidecar.istio.io/inject: "false"
   rules:
   - apiGroups:
     - "" # "" indicates the core API group
     resources:
     - services
     - configmaps
     - pods
     - secrets
     - endpoints
     - persistentvolumeclaims
     verbs:
     - get
     - watch
     - list
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: cncc-rolebinding
     labels:
       app.kubernetes.io/component: internal
     annotations:
       sidecar.istio.io/inject: "false"
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: Role
     name: cncc-role
   subjects:
   - kind: ServiceAccount
     name: cncc-serviceaccount
   ---
   EOF

   Example for A-CNCC Core | cncc-sa-role-rolebinding.yaml

   kubectl apply -n cncc -f - <<EOF
   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: cncc-serviceaccount
     labels:
       app.kubernetes.io/component: internal
     annotations:
       sidecar.istio.io/inject: "false"
       "certificate.aspenmesh.io/customFields": '{ "SAN": { "DNS": [ "cncc-acore-ingress-gateway.cncc.svc.cluster.local"] } }'
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: cncc-role
     labels:
       app.kubernetes.io/component: internal
     annotations:
       sidecar.istio.io/inject: "false"
   rules:
   - apiGroups:
     - "" # "" indicates the core API group
     resources:
     - services
     - configmaps
     - pods
     - secrets
     - endpoints
     - persistentvolumeclaims
     verbs:
     - get
     - watch
     - list
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: cncc-rolebinding
     labels:
       app.kubernetes.io/component: internal
     annotations:
       sidecar.istio.io/inject: "false"
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: Role
     name: cncc-role
   subjects:
   - kind: ServiceAccount
     name: cncc-serviceaccount
   ---
   EOF

M-CNCC IAM, M-CNCC Core and A-CNCC Core configuration for ASM

This section explains about the CNCC IAM deployment configuration for ASM.

Update occncc_custom_values_<version>.yaml as follows:

1. Add Required Annotation:

   global:
     # ********  Sub-Section Start: Common Global Parameters *************
     # *******************************************************************
    
       nonlbStatefulSets:
         labels: {}
         annotations:
           sidecar.istio.io/rewriteAppHTTPProbers: "true"
     
     # ********  Sub-Section End: Common Global Parameters *******************
     # ***********************************************************************

2. Provide the service account name:

   
   global:
     # *****  Sub-Section Start: Ingress Gateway Global Parameters *****
     serviceAccountName: &serviceAccountName <cncc-serviceaccount-name>

3. Enable Service Mesh Flag :

   global: 
    # Mandatory: This flag needs to set it "true" if Service Mesh would be present where CNCC will be deployed  
             serviceMeshCheck: true

4. If ASM is deployed with mTLS disabled, then set serviceMeshHttpsEnabled flag to false:

   global:  
                serviceMeshHttpsEnabled:false

M-CNCC IAM, M-CNCC Core, and A-CNCC Core Configuration for OSO

Add Annotation oracle.com/cnc: "true" under global.customExtention.lbDeployments.annotations section in occncc_custom_values_<version>.yaml file to indicate OSO to scrape metrics from ingress pod.

global:
  # ****  Sub-Section Start: Common Global Parameters *****
  
   
  customExtension:
    lbDeployments:
      labels: {}
      annotations:
        oracle.com/cnc: "true"
 
  # ****  Sub-Section End: Common Global Parameters *******