3 Debug Tool

Overview

Note:

CNC Console IAM is not applicable for OCI.
The Debug Tools provides third-party troubleshooting tools for debugging the runtime issues for the lab environment. Following are the available tools:
  • tcpdump
  • ip
  • netstat
  • curl
  • ping
  • dig

Note:

Debug Tool is only applicable for lab setup, and not recommended for production use.

Note:

While testing in OCCNE environment, check the Kyverno policies and make sure to exclude the namespace in the disallow-capabilities.

Prerequisites

This section explains the prerequisites for using debug tool.

  • For CNE 23.2.0 and later versions, follow step a of Configuration in CNE.
  • For CNE versions prior to 23.2.0, follow step b of Configuration in CNE.
  1. Configuration in CNE

    The following configurations must be performed in the Bastion Host.

    1. When CNC Console is installed on CNE version 23.2.0 or above

      Note:

      • You need admin privileges to edit or patch the clusterpolicies that are mentioned in following steps.
      • In CNE v23.2.0 onwards, kyverno policy, disallow-capabilities does not allow NET_ADMIN and NET_RAW capabilities required for debug tool.

      Adding Namespace to the Empty Resource

      If the current disallow-capabilities cluster policy has no namespace in it, for example:
      $ kubectl get clusterpolicies disallow-capabilities -oyaml
      Sample output
      apiVersion: kyverno.io/v1
kind: ClusterPolicy
...
...
spec:
  rules:
  - exclude:
      resources: {}

      Then, run the following command to add the namespace: 

      $ kubectl patch clusterpolicy disallow-capabilities --type=json \
  -p='[{"op": "add", "path": "/spec/rules/0/exclude/any/0/resources", "value": {"namespaces":["<namespace>"]} }]'
      Example:
      $ kubectl patch clusterpolicy disallow-capabilities --type=json \
  -p='[{"op": "add", "path": "/spec/rules/0/exclude/any/0/resources", "value": {"namespaces":["cncc"]} }]'
      Sample output
      apiVersion: kyverno.io/v1
kind: ClusterPolicy
...
...
spec:
  rules:
  - exclude:
      resources:
        namespaces:
        - cncc
      To remove the namespace, run the following:
      $ kubectl patch clusterpolicy disallow-capabilities --type=json \
  -p='[{"op": "replace", "path": "/spec/rules/0/exclude/any/0/resources", "value": {} }]'

      Sample output

      
apiVersion: kyverno.io/v1
kind: ClusterPolicy
...
...
spec:
  rules:
  - exclude:
      resources: {}

      Adding a Namespace to the Existing Namespace List

      If the current disallow-capabilities cluster policy has already namespaces added in it, for example:
      $ kubectl get clusterpolicies disallow-capabilities -oyaml
      Output:
      
apiVersion: kyverno.io/v1
kind: ClusterPolicy
...
...
spec:
  rules:
  - exclude:
      resources:
        namespaces:
        - namespace1
        - namespace2
        - namespace3
      Then, run the following command to add your namespace:
      $ kubectl patch clusterpolicy disallow-capabilities --type=json \
  -p='[{"op": "add", "path": "/spec/rules/0/exclude/any/0/resources/namespaces/-", "value": "<namespace>" }]'
      Example:
      $ kubectl patch clusterpolicy disallow-capabilities --type=json \
  -p='[{"op": "add", "path": "/spec/rules/0/exclude/any/0/resources/namespaces/-", "value": "cncc" }]'
      Sample Output:
      
 apiVersion: kyverno.io/v1
kind: ClusterPolicy
...
...
spec:
  rules:
  - exclude:
      resources:
        namespaces:
        - namespace1
        - namespace2
        - namespace3
        - cncc
      To remove the namespace, run the following command:
      $ kubectl patch clusterpolicy disallow-capabilities --type=json \
  -p='[{"op": "remove", "path": "/spec/rules/0/exclude/any/0/resources/namespaces/<index>"}]'
      Example:
      
$ kubectl patch clusterpolicy disallow-capabilities --type=json \
  -p='[{"op": "remove", "path": "/spec/rules/0/exclude/any/0/resources/namespaces/3"}]'
      Output:
      apiVersion: kyverno.io/v1
kind: ClusterPolicy
...
...
spec:
  rules:
  - exclude:
      resources:
        namespaces:
        - namespace1
        - namespace2
        - namespace3
    2. Configurations in CNE Versions Prior to 23.2.0

      The following configurations must be performed in the Bastion Host.

      Note:

      These steps are needed only when you have PSP admission controller enabled in your kubernetes environment.

      PodSecurityPolicy (PSP) Creation

      1. Log in to the Bastion Host.
      2. Create a new PSP by running the following command. The parameters readOnlyRootFileSystem, allowPrivilegeEscalation, allowedCapabilities are needed by debug container.

        Note:

        Other parameters are mandatory for PSP creation and can be customized as per the CNE environment. Default values are recommended.
      PodSecurityPolicy
      kubectl apply -f - <<EOF
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: debug-tool-psp
spec:
  readOnlyRootFilesystem: false
  allowPrivilegeEscalation: true
  allowedCapabilities:
  - NET_ADMIN
  - NET_RAW
  fsGroup:
    ranges:
    - max: 65535
      min: 1
    rule: MustRunAs
  runAsUser:
    rule: MustRunAsNonRoot
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  volumes:
  - configMap
  - downwardAPI
  - emptyDir
  - persistentVolumeClaim
  - projected
  - secret
EOF

      Table 3-1 PodSecurityPolicy

      Parameter Description
      apiVersion APIVersion defines the versioned schema of this representation of an object.
      kind Kind is a string value representing the REST resource this object represents.
      metadata Standard object's metadata.
      metadata.name Name must be unique within a namespace.
      spec spec defines the policy enforced.
      spec.readOnlyRootFilesystem Controls whether the containers run with a read-only root filesystem (that is no writable layer).
      spec.allowPrivilegeEscalation Gates whether or not a user is allowed to set the security context of a container to allowPrivilegeEscalation=true.
      spec.allowedCapabilities Provides a list of capabilities that are allowed to be added to a container.
      spec.fsGroup Controls the supplemental group applied to some volumes. RunAsAny allows any fsGroup ID to be specified.
      spec.runAsUser Controls which user ID the containers are run with. RunAsAny allows any runAsUser to be specified.
      spec.seLinux RunAsAny allows any seLinuxOptions to be specified.
      spec.supplementalGroups Controls which group IDs containers add. RunAsAny allows any supplementalGroups to be specified.
      spec.volumes Provides a list of allowed volume types. The allowable values correspond to the volume sources that are defined when creating a volume.

    Role Creation

    Create a role for the PSP by running the following commands:

    Role
    kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: debug-tool-role
  namespace: cncc
rules:
- apiGroups:
  - policy
  resources:
  - podsecuritypolicies
  verbs:
  - use
  resourceNames:
  - debug-tool-psp
EOF

    Table 3-2 Role

    Parameter Description
    apiGroups APIGroups is the name of the APIGroup that contains the resources.
    apiVersion APIVersion defines the versioned schema of this representation of an object.
    kind Kind is a string value representing the REST resource this object represents.
    metadata Standard object's metadata.
    metadata.name Name must be unique within a namespace.
    metadata.namespace Namespace defines the space within which each name must be unique.
    rules Rules holds all the PolicyRules for this Role
    rules.resourceNames ResourceNames is an optional white list of names that the rule applies to.
    rules.resources Resources is a list of resources this rule applies to.
    rules.verbs Verbs is a list of Verbs that apply to ALL the ResourceKinds and AttributeRestrictions contained in this rule.

    RoleBinding Creation

    Run the following command to attach the service account for your namespace with the role created for the tool PSP:

    RoleBinding
    kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: debug-tool-rolebinding
  namespace: cncc
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: debug-tool-role
subjects:
- kind: Group
  apiGroup: rbac.authorization.k8s.io
  name: system:serviceaccounts
EOF

    Table 3-3 RoleBinding

    Parameter Description
    apiVersion APIVersion defines the versioned schema of this representation of an object.
    kind Kind is a string value representing the REST resource this object represents.
    metadata Standard object's metadata.
    metadata.name Name must be unique within a namespace.
    metadata.namespace Namespace defines the space within which each name must be unique.
    roleRef RoleRef can reference a Role in the current namespace or a ClusterRole in the global namespace.
    roleRef.apiGroup APIGroup is the group for the resource being referenced
    roleRef.kind Kind is the type of resource being referenced
    roleRef.name Name is the name of resource being referenced
    subjects Subjects holds references to the objects the role applies to.
    subjects.kind Kind of object being referenced. Values defined by this API group are "User", "Group", and "ServiceAccount".
    subjects.apiGroup APIGroup holds the API group of the referenced subject.
    subjects.name Name of the object being referenced.

Configuration Changes in CNCC Helm Charts

To enable debug tools container, make changes to occncc_custom_values_<version>.yaml file at Global level by setting extraContainers: ENABLED.

global:
  # Allowed Values: DISABLED, ENABLED
  # Preference is to set  "resources" request and limit to same values to avoid HPA issues.
  extraContainers: ENABLED
  debugToolContainerMemoryLimit: 4Gi
  extraContainersVolumesTpl: |
    - name: debug-tools-dir
      emptyDir:
        medium: Memory
        sizeLimit: {{ .Values.global.debugToolContainerMemoryLimit | quote }}
  extraContainersTpl: |
    - command:
        - /bin/sleep
        - infinity
      image: {{ .Values.global.dockerRegistry }}/occncc/ocdebug-tools:<debugtool_version
      imagePullPolicy: IfNotPresent
      name: {{ printf "%s-tools-%s" (include "getprefix" .) (include "getsuffix" .) | trunc 63 | trimPrefix "-" | trimSuffix "-"  }}
      resources:
        limits:
          ephemeral-storage: "512Mi"
          cpu: "0.5"
          memory: {{ .Values.global.debugToolContainerMemoryLimit | quote }}
        requests:
          ephemeral-storage: "512Mi"
          cpu: "0.5"
          memory: {{ .Values.global.debugToolContainerMemoryLimit | quote }}
      securityContext:
        allowPrivilegeEscalation: true
        capabilities:
          drop:
          - ALL
          add:
          - NET_RAW
          - NET_ADMIN
        #runAsUser: <user-id>
      volumeMounts:
      - mountPath: /tmp/tools
        name: debug-tools-dir

To enable debug tools at service level, make changes to occncc_custom_values_<version>.yaml file at service level by setting extraContainers: USE_GLOBAL_VALUE

occncc_custom_values_<version>.yaml

cncc-iam:
  kc:
    # Allowed Values: DISABLED, ENABLED, USE_GLOBAL_VALUE
    extraContainers: USE_GLOBAL_VALUE
 
  ingress-gateway:
    # Allowed Values: DISABLED, ENABLED, USE_GLOBAL_VALUE
    extraContainers: USE_GLOBAL_VALUE
 
 
 
 
mcncc-core:
  cmservice:
    # Allowed Values: DISABLED, ENABLED, USE_GLOBAL_VALUE
    extraContainers: USE_GLOBAL_VALUE
 
  ingress-gateway:
    # Allowed Values: DISABLED, ENABLED, USE_GLOBAL_VALUE
    extraContainers: USE_GLOBAL_VALUE
 
 
 
 
acncc-core:
  ingress-gateway:
    # Allowed Values: DISABLED, ENABLED, USE_GLOBAL_VALUE
    extraContainers: USE_GLOBAL_VALUE

Note:

Debug Tool Container comes up with the default user ID - 7000. If the operator wants to override this default value, it can be done using the runAsUser field, otherwise the field can be skipped.

Default value: uid=7000(debugtool) gid=7000(debugtool) groups=7000(debugtool)

To override runAsUser add this line under securityContext in extraContainersTpl

runAsUser: <user-id>

Configuration Options

Table 3-4 Configuration Options

Parameter Description
global.extraContainersTpl Describes the kubernetes container template for the debug-tool.
global.debugToolContainerMemoryLimit This field describes the memory size (request and limit) and emptyDir volume size for the Debug-tool container.
global.extraContainersTpl.command String array used for container command.
global.extraContainersTpl.image Docker image name
global.extraContainersTpl.imagePullPolicy Image Pull Policy
global.extraContainersTpl.name Name of the container
global.extraContainersTpl.resources Compute Resources required by this container
global.extraContainersTpl.resources.limits Limits describes the maximum amount of compute resources allowed
global.extraContainersTpl.resources.requests Requests describes the minimum amount of compute resources required
global.extraContainersTpl.resources.limits.cpu CPU limits
global.extraContainersTpl.resources.limits.memory Memory limits
global.extraContainersTpl.resources.limits.ephemeral-storage Ephemeral Storage limits
global.extraContainersTpl.resources.requests.cpu CPU requests
global.extraContainersTpl.resources.requests.memory Memory requests
global.extraContainersTpl.resources.requests.ephemeral-storage Ephemeral Storage requests
global.extraContainersTpl.securityContext Security options the container should run with.
global.extraContainersTpl.securityContext.allowPrivilegeEscalation AllowPrivilegeEscalation controls whether a process can gain more privileges than its parent process. This boolen directly controls if the no_new_privs flag will be set on the container process
global.extraContainersTpl.securityContext.capabilities The capabilities to add or drop when running containers. Defaults to the default set of capabilities granted by the container runtime.
global.extraContainersTpl.securityContext.capabilities.drop Removed capabilities
global.extraContainersTpl.secuirtyContext.capabilities.add Added capabilities
global.extraContainersTpl.securityContext.runAsUser The UID to run the entrypoint of the container process.
global.extraContainersTpl.volumeMounts.mountPath The path where the emptyDir volume has to be mounted inside the container.
global.extraContainersTpl.volumeMounts.name Name of the emptyDir volume.

Debug Tool Volume Parameters

Table 3-5 Debug Tool Volume Parameters

Parameter Description
global.extraContainersVolumesTpl Describes the kubernetes volume template for the debug-tool container
global.extraContainersVolumesTpl.name Name of the emptyDir volume.
global.extraContainersVolumesTpl.emptyDir.sizeLimit The size of the emptyDir volume.
global.extraContainersVolumesTpl.emptyDir.medium Describes where emptyDir volumes are stored.

Debug Tool Usage

Following is the procedure to run Debug Tool:

  1. Run the following command to retrieve the POD details:
    $ kubectl get pods -n <k8s namespace>
    Example:
    # kubectl get pod -n cncc

    Sample Output:

     NAME                                         READY   STATUS   RESTARTS   AGE
cncc-acore-ingress-gateway-764f7f5f77-qnr5p   2/2     Running   0         19m
cncc-iam-ingress-gateway-55987f7dc9-x5nt2     2/2     Running   0         147m
cncc-iam-kc-0                                 2/2     Running   0         147m
cncc-mcore-cmservice-947cf4c89-76vq6          2/2     Running   0         19m
cncc-mcore-ingress-gateway-764f7f5f77-qnr5p   2/2     Running   0         19m
  2. Run the following command to enter Debug Tools Container: 
    kubectl exec -it <pod name> -c <debug_container name> -n <namespace> bash
    Example:
    kubectl exec -it cncc-mcore-ingress-gateway-599d858867-x9pvz -c tools -n cncc bash
  3. Run the debug tools:
    bash -4.2$ <debug_tools>
    Example:
    bash -4.2$ tcpdump
  4. Run the following command to copy output files from container to host:
    $ kubectl cp -c <debug_container name>  <pod name>:<file location in container> -n <namespace>  <destination location>
    Example:
    $ kubectl cp -c tools -n cncc cncc-mcore-ingress-gateway-764f7f5f77-qnr5p:/tmp/capture.pcap /tmp/

Steps to Enable Debug Tools Container

Debug tools container can be enabled or disabled for CNCC components by using helm install or helm upgrade command.

CNC Console

Run the following command to enable or disable CNC Console IAM after updating occncc_custom_values_<version>.yaml file on a installed setup: 

$ helm upgrade <release_name> -f occncc_custom_values_<version>.yaml <helm-repo> --version <helm_version>

Example:

$ helm upgrade cncc -f occncc_custom_values_23.4.0.yaml ocspf_helm-repo/cncc --version 23.4.0