1. Cloud Native Core Console Installation and Upgrade Guide
  2. CNC Console Debug Tools

12 CNC Console Debug Tools

Overview

The Debug Tools provides third party troubleshooting tools for debugging the runtime issues for both lab and production environment. Following are the available tools:

  • tcpdump
  • ip
  • netstat
  • curl
  • ping
  • dig

Prerequisites

This section explains the prerequisites for using debug tool.

Configurations in CNE

The following configurations must be performed in the Bastion Host.

Note:

These steps are needed only when you have PSP admission controller enabled in your kubernetes environment.

PodSecurityPolicy (PSP) Creation

  1. Log in to the Bastion Host.
  2. Create a new PSP by running the following command. The parameters readOnlyRootFileSystem, allowPrivilegeEscalation, allowedCapabilities are needed by debug container.

    Note:

    Other parameters are mandatory for PSP creation and can be customized as per the CNE environment. Default values are recommended.
    .
PodSecurityPolicy
kubectl apply -f - <<EOF
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: debug-tool-psp
spec:
  readOnlyRootFilesystem: false
  allowPrivilegeEscalation: true
  allowedCapabilities:
  - NET_ADMIN
  - NET_RAW
  fsGroup:
    ranges:
    - max: 65535
      min: 1
    rule: MustRunAs
  runAsUser:
    rule: MustRunAsNonRoot
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  volumes:
  - configMap
  - downwardAPI
  - emptyDir
  - persistentVolumeClaim
  - projected
  - secret
EOF

Table 12-1 PodSecurityPolicy

Parameter Description
apiVersion APIVersion defines the versioned schema of this representation of an object.
kind Kind is a string value representing the REST resource this object represents.
metadata Standard object's metadata.
metadata.name Name must be unique within a namespace.
spec spec defines the policy enforced.
spec.readOnlyRootFilesystem Controls whether the containers run with a read-only root filesystem (that is no writable layer).
spec.allowPrivilegeEscalation Gates whether or not a user is allowed to set the security context of a container to allowPrivilegeEscalation=true.
spec.allowedCapabilities Provides a list of capabilities that are allowed to be added to a container.
spec.fsGroup Controls the supplemental group applied to some volumes. RunAsAny allows any fsGroup ID to be specified.
spec.runAsUser Controls which user ID the containers are run with. RunAsAny allows any runAsUser to be specified.
spec.seLinux RunAsAny allows any seLinuxOptions to be specified.
spec.supplementalGroups Controls which group IDs containers add. RunAsAny allows any supplementalGroups to be specified.
spec.volumes Provides a list of allowed volume types. The allowable values correspond to the volume sources that are defined when creating a volume.

Role Creation

Create a role for the PSP by executing the following commands:

Role
kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: debug-tool-role
  namespace: cncc
rules:
- apiGroups:
  - policy
  resources:
  - podsecuritypolicies
  verbs:
  - use
  resourceNames:
  - debug-tool-psp
EOF

Table 12-2 Role

Parameter Description
apiGroups APIGroups is the name of the APIGroup that contains the resources.
apiVersion APIVersion defines the versioned schema of this representation of an object.
kind Kind is a string value representing the REST resource this object represents.
metadata Standard object's metadata.
metadata.name Name must be unique within a namespace.
metadata.namespace Namespace defines the space within which each name must be unique.
rules Rules holds all the PolicyRules for this Role
rules.resourceNames ResourceNames is an optional white list of names that the rule applies to.
rules.resources Resources is a list of resources this rule applies to.
rules.verbs Verbs is a list of Verbs that apply to ALL the ResourceKinds and AttributeRestrictions contained in this rule.

RoleBinding Creation

Run the following command to attach the service account for your namespace with the role created for the tool PSP:

RoleBinding
kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: debug-tool-rolebinding
  namespace: cncc
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: debug-tool-role
subjects:
- kind: Group
  apiGroup: rbac.authorization.k8s.io
  name: system:serviceaccounts
EOF

Table 12-3 RoleBinding

Parameter Description
apiVersion APIVersion defines the versioned schema of this representation of an object.
kind Kind is a string value representing the REST resource this object represents.
metadata Standard object's metadata.
metadata.name Name must be unique within a namespace.
metadata.namespace Namespace defines the space within which each name must be unique.
roleRef RoleRef can reference a Role in the current namespace or a ClusterRole in the global namespace.
roleRef.apiGroup APIGroup is the group for the resource being referenced
roleRef.kind Kind is the type of resource being referenced
roleRef.name Name is the name of resource being referenced
subjects Subjects holds references to the objects the role applies to.
subjects.kind Kind of object being referenced. Values defined by this API group are "User", "Group", and "ServiceAccount".
subjects.apiGroup APIGroup holds the API group of the referenced subject.
subjects.name Name of the object being referenced.

Configuration changes in CNCC Helm Charts

To enable debug tools container, make changes to occncc_custom_values_<version>.yaml file at Global Level by setting extraContainers: ENABLED.

global:
# Allowed Values: DISABLED, ENABLED
extraContainers: ENABLED
extraContainersTpl: |
  - command:
      - /bin/sleep
      - infinity
    image: {{ .Values.global.dockerRegistry }}/cncc/debug_tools:22.4.1
    imagePullPolicy: Always
    name: {{ printf "%s-tools-%s" (include "getprefix" .) (include "getsuffix" .) | trunc 63 | trimPrefix "-" | trimSuffix "-"  }}
    resources:
      limits:
        ephemeral-storage: "4Gi"
        cpu: "1"
        memory: "2Gi"
      requests:
        ephemeral-storage: "2Gi"
        cpu: "0.5"
        memory: "1Gi"
    securityContext:
      allowPrivilegeEscalation: true
      capabilities:
        drop:
        - ALL
        add:
        - NET_RAW
        - NET_ADMIN
      readOnlyRootFilesystem: false

To enable debug tools at service level, make changes to occncc_custom_values_<version>.yaml file at service level by setting extraContainers: USE_GLOBAL_VALUE

occncc_custom_values_<version>.yaml


cncc-iam:
kc:
 
 # Allowed Values: DISABLED, ENABLED, USE_GLOBAL_VALUE
 extraContainers: USE_GLOBAL_VALUE
 
 
ingress-gateway:
 
 # Allowed Values: DISABLED, ENABLED, USE_GLOBAL_VALUE
 extraContainers: USE_GLOBAL_VALUE

mcncc-core:
  cmservice:
    # Allowed Values: DISABLED, ENABLED, USE_GLOBAL_VALUE
    extraContainers: USE_GLOBAL_VALUE
 
  ingress-gateway:
    # Allowed Values: DISABLED, ENABLED, USE_GLOBAL_VALUE
    extraContainers: USE_GLOBAL_VALUE
 
 
 
acncc-core:
  cmservice:
    # Allowed Values: DISABLED, ENABLED, USE_GLOBAL_VALUE
    extraContainers: USE_GLOBAL_VALUE
 
  ingress-gateway:
    # Allowed Values: DISABLED, ENABLED, USE_GLOBAL_VALUE
    extraContainers: USE_GLOBAL_VALUE

Note:

User ID: `runAsUser`

Debug Tool Container comes up with the default user ID - 7000. If the operator wants to override this default value, it can be done using the `runAsUser` field, otherwise the field can be skipped.

Default value: uid=7000(debugtool) gid=7000(debugtool) groups=7000(debugtool)

To override runAsUser add this line under securityContext in extraContainersTpl

runAsUser: <user-id>

Configuration Options

Table 12-4 Configuration Options

Parameter Description
command String array used for container command.
image Docker image name
imagePullPolicy Image Pull Policy
name Name of the container
resources Compute Resources required by this container
resources.limits Limits describes the maximum amount of compute resources allowed
resources.requests Requests describes the minimum amount of compute resources required
resources.limits.cpu CPU limits
resources.limits.memory Memory limits
resources.limits.ephemeral-storage Ephemeral Storage limits
resources.requests.cpu CPU requests
resources.requests.memory Memory requests
resources.requests.ephemeral-storage Ephemeral Storage requests
securityContext Security options the container should run with.
securityContext.allowPrivilegeEscalation AllowPrivilegeEscalation controls whether a process can gain more privileges than its parent process. This boolen directly controls if the no_new_privs flag will be set on the container process
secuirtyContext.readOnlyRootFilesystem Whether this container has a read-only root filesystem. Default is false.
securityContext.capabilities The capabilities to add or drop when running containers. Defaults to the default set of capabilities granted by the container runtime.
securityContext.capabilities.drop Removed capabilities
secuirtyContext.capabilities.add Added capabilities
securityContext.runAsUser The UID to run the entrypoint of the container process.

Debug Tool Usage

Following is the procedure to run Debug Tool:

  1. Run the following command to retrieve the POD details:
    $ kubectl get pods -n <k8s namespace>

    After installation the debug-tool container will get injected into the pods, sample get pod output is here :

     [root@master ~]# kubectl get po -n cncc
NAME                                         READY   STATUS   RESTARTS   AGE
cncc-acore-cmservice-947cf4c89-76vq6          2/2     Running   0         19m
cncc-acore-ingress-gateway-764f7f5f77-qnr5p   2/2     Running   0         19m
cncc-iam-ingress-gateway-55987f7dc9-x5nt2     2/2     Running   0         147m
cncc-iam-kc-0                                 2/2     Running   0         147m
cncc-mcore-cmservice-947cf4c89-76vq6          2/2     Running   0         19m
cncc-mcore-ingress-gateway-764f7f5f77-qnr5p   2/2     Running   0         19m
  2. Run the following command to enter Debug Tools Container: 
    kubectl exec -it <pod name> -c <debug_container name> -n <namespace> bash
    Example:
    kubectl exec -it cncc-mcore-ingress-gateway-599d858867-x9pvz -c tools -n cncc bash
  3. Run the debug tools:
    bash -4.2$ <debug_tools>
    Example:
    bash -4.2$ tcpdump
  4. Run the following command to copy output files from container to host:
    $ kubectl cp -c <debug_container name>  <pod name>:<file location in container> -n <namespace>  <destination location>
    Example:
    $ kubectl cp -c tools -n cncc cncc-mcore-ingress-gateway-764f7f5f77-qnr5p:/tmp/capture.pcap /tmp/

Steps to Enable Debug Tools Container

Debug tools container can be enabled or disabled for CNCC by using helm install or helm upgrade command.

CNC Console

Run the following command to enable or disable CNC Console IAM after updating occncc_custom_values_<version>.yaml file on a installed setup: 

$ helm upgrade <release_name> -f occncc_custom_values_<version>.yaml <helm-repo> --version <helm_version>

Example :

$ helm upgrade cncc -f occncc_custom_values_<version>.yaml ocspf-helm-repo/cncc --version 22.4.1