4 Debug Tool

Overview

The Debug Tool provides third-party troubleshooting tools for debugging the runtime issues in a lab environment.

Following are the available tools:

• tcpdump
• ip
• netstat
• curl
• ping
• dig

Preconfiguration Steps

This section explains the preconfiguration steps for using the debug tool:

Note:

• For the CNE 23.2.0 and later versions, follow the Step a of Configuration in CNE to Update the Cluster Policies and Add Namespace.
• For the CNE 23.1.x and previous versions, follow the Step b of Configuration in CNE for PodSecurityPolicy (PSP) Creation, Role Creation, and RoleBinding Creation.

1. Configuration in CNE
   Perform the following configurations in the Bastion Host. You need admin privileges to perform these configurations.

   1. When NEF is installed on CNE version 23.2.0 or above

      Note:

      • In CNE version 23.2.0 or above, the default CNE 23.2.0 Kyverno policy, disallow-capabilities, do not allow NET_ADMIN and NET_RAW capabilities that are required for debug tool.
      • To run Debug tool on CNE 23.2.0 and above, the user must modify the existing Kyverno policy, disallow-capabilities, as below.
      Adding a Namespace to an Empty Resource

      1. Run the following command to verify if the current disallow-capabilities cluster policy has namespace in it.
         Example:

         $ kubectl get clusterpolicies disallow-capabilities -oyaml

         Sample output:

         apiVersion: kyverno.io/v1
         kind: ClusterPolicy
         ...
         ...
         spec:
           rules:
           -exclude:
               any:
               -resources:{}

      2. If there are no namespaces, then patch the policy using the following command to add <namespace> under resources:

         $ kubectl patch clusterpolicy disallow-capabilities --type=json \
           -p='[{"op": "add", "path": "/spec/rules/0/exclude/any/0/resources", "value": {"namespaces":["<namespace>"]} }]'

         Example:

         $ kubectl patch clusterpolicy disallow-capabilities --type=json \
           -p='[{"op": "add", "path": "/spec/rules/0/exclude/any/0/resources", "value": {"namespaces":["ocnef"]} }]'

         Sample output:

         apiVersion: kyverno.io/v1
         kind: ClusterPolicy
         ...
         ...
         spec:
           rules:
           -exclude:
               resources:
                 namespaces:
                 - sepp1
         

      3. If in case it is needed to remove the namespace added in the above step, use the following command:

         $ kubectl patch clusterpolicy disallow-capabilities --type=json \
           -p='[{"op": "replace", "path": "/spec/rules/0/exclude/any/0/resources", "value": {} }]'

         Sample output:

         apiVersion: kyverno.io/v1
         kind: ClusterPolicy
         ...
         ...
         spec:
           rules:
           -exclude:
               any:
               -resources:{}

      Adding a Namespace to an Existing Namespace List

      1. Run the following command to verify if the current disallow-capabilities cluster policy has namespaces in it.

         Example:

         $ kubectl get clusterpolicies disallow-capabilities -oyaml

         Sample output:

         apiVersion: kyverno.io/v1
         kind: ClusterPolicy
         ...
         ...
         spec:
           rules:
           -exclude:
               any:
               -resources:
                   namespaces:
                   -namespace1
                   -namespace2
                   -namespace3

      2. If there are namespaces already added, then patch the policy using the following command to add <namespace> to the existing list:

         $ kubectl patch clusterpolicy disallow-capabilities --type=json \
           -p='[{"op": "add", "path": "/spec/rules/0/exclude/any/0/resources/namespaces/-", "value": "<namespace>" }]'

         Example:

         $ kubectl patch clusterpolicy disallow-capabilities --type=json \
           -p='[{"op": "add", "path": "/spec/rules/0/exclude/any/0/resources/namespaces/-", "value": "seppsvc" }]'

         Sample output:

         apiVersion: kyverno.io/v1
         kind: ClusterPolicy
         ...
         ...
         spec:
           rules:
           -exclude:
               resources:
                 namespaces:
                 -namespace1
                 -namespace2
                 -namespace3
                 - sepp1
         

      3. If in case it is needed to remove the namespace added in the above step, use the following command:

         $ kubectl patch clusterpolicy disallow-capabilities --type=json \
           -p='[{"op": "remove", "path": "/spec/rules/0/exclude/any/0/resources/namespaces/<index>"}]'

         Example:

         $ kubectl patch clusterpolicy disallow-capabilities --type=json \
           -p='[{"op": "remove", "path": "/spec/rules/0/exclude/any/0/resources/namespaces/3"}]'

         Sample output:

         apiVersion: kyverno.io/v1
         kind: ClusterPolicy
         ...
         ...
         spec:
           rules:
           -exclude:
               resources:
                 namespaces:
                 -namespace1
                 -namespace2
                 -namespace3

         Note:

         While removing the namespace, provide the index value for namespace within the array. The index starts from '0'.
   2. When NEF is installed on CNE version prior to 23.2.0

PodSecurityPolicy (PSP) Creation

1. Log in to the Bastion Host.
2. Create a new PSP by running the following command from the bastion host. The parameters readOnlyRootFileSystem, allowPrivilegeEscalation, allowedCapabilities are required by the debug container.

   Note:

   Other parameters are mandatory for PSP creation and can be customized as per the CNE environment. Default values are recommended.

   $ kubectl apply -f - <<EOF
   
   apiVersion: policy/v1beta1
   kind: PodSecurityPolicy
   metadata:
     name: debug-tool-psp
   spec:
     readOnlyRootFilesystem: false
     allowPrivilegeEscalation: true
     allowedCapabilities:
     - NET_ADMIN
     - NET_RAW
     fsGroup:
       ranges:
       - max: 65535
         min: 1
       rule: MustRunAs
     runAsUser:
       rule: MustRunAsNonRoot
     seLinux:
       rule: RunAsAny
     supplementalGroups:
       rule: RunAsAny
     volumes:
     - configMap
     - downwardAPI
     - emptyDir
     - persistentVolumeClaim
     - projected
     - secret
   EOF

Role Creation

Run the following command to create a role for the PSP:

kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: debug-tool-role
  namespace: seppsvc
rules:
- apiGroups:
  - policy
  resources:
  - podsecuritypolicies
  verbs:
  - use
  resourceNames:
  - debug-tool-psp
EOF

RoleBinding Creation

Run the following command to attach the service account for your NF namespace with the role created for the tool PSP:

$ kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: debug-tool-rolebinding
  namespace: seppsvc
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: debug-tool-role
subjects:
- kind: Group
  apiGroup: rbac.authorization.k8s.io
  name: system:serviceaccounts
EOF

For parameter details, see Debug Tool Configuration Parameters.

1. Configuration in NF specific Helm

   Following updates must be performed in custom_values.yaml file.

   1. Log in to the NF server.
   2. Open the custom_values file:

      $ vim <custom_values file>

   3. Under global configuration, add the following:

      # Allowed Values: DISABLED, ENABLED
        podSecurityPolicy: "DISABLED"
        extraContainers: "DISABLED"
        debugToolContainerMemoryLimit: 4Gi
        extraContainersImageDetails:
          image: ocdebugtool/ocdebug-tools
          tag: debug_container_tag
          imagePullPolicy: Always
        extraContainersVolumesTpl: |
          - name: debug-tools-dir
            emptyDir:
              medium: Memory
              sizeLimit: {{ .Values.global.debugToolContainerMemoryLimit | quote }}
        extraContainersTpl: |-
          - command:
              - /bin/sleep
              - infinity
            name: tools
            resources:
              requests:
                ephemeral-storage: "512Mi"
                cpu: "0.5"
                memory: {{ .Values.global.debugToolContainerMemoryLimit | quote }}
              limits:
                ephemeral-storage: "512Mi"
                cpu: "1"
                memory: {{ .Values.global.debugToolContainerMemoryLimit | quote }}
            securityContext:
              allowPrivilegeEscalation: true
              capabilities:
                drop:
                - ALL
                add:
                - NET_RAW
                - NET_ADMIN
              runAsUser: 1012
            volumeMounts:
            - mountPath: /tmp/tools
              name: debug-tools-dir
      

      Note:

      • Debug Tool Container comes up with the default user ID - 7000. If you want to override this default value, use the `runAsUser` field, or else, you can skip the field.

        Default value: uid=7000(debugtool) gid=7000(debugtool) groups=7000(debugtool)

      • In case you want to customize the container name, replace the `name` field in the above values.yaml with the following:

        name: {{ printf "%s-tools-%s" (include "getprefix" .) (include "getsuffix" .) | trunc 63 | trimPrefix "-" | trimSuffix "-"  }}

        This will ensure that the container name is prefixed and suffixed with the necessary values.
   4. Under service specific configurations for which debugging is required, add the following:

      # Allowed Values: DISABLED, ENABLED, USE_GLOBAL_VALUE
      extraContainers: USE_GLOBAL_VALUE

      Note:

      • At the global level, extraContainers flag can be used to enable or disable injecting extra containers globally. This ensures that all the services that use this global value have extra containers enabled or disabled using a single flag.
      • At the service level, extraContainers flag determines whether to use the extra container configuration from the global level or enable or disable injecting extra containers for the specific service.

Run the Debug Tool

Following is the procedure to run Debug Tool.

Run the following command to enter Debug Tool Container:

1. Run the following command to retrieve the POD details:

   $ kubectl get pods -n <k8s namespace>

   Example:

   $ kubectl get pods -n seppsvc

   Sample Output:

   NAME                                                   READY  STATUS   RESTARTS    AGE
   ocsepp-release-appinfo-75894d8d8c-4zzkt                        2/2     Running    0          5m54s
   ocsepp-release-cn32c-svc-5f5cdbfb7f-kspw6                      2/2     Running    0          5m55s
   ocsepp-release-cn32f-svc-5458886cc7-nm7c8                      2/2     Running    0          5m55s
   ocsepp-release-config-mgr-svc-6c94c449f-v8qnv                  2/2     Running    0          5m55s
   ocsepp-release-n32-egress-gateway-55ccbbf46f-bb4tp             3/3     Running    0          5m54s
   ocsepp-release-n32-ingress-gateway-7bd984c9c6-pcpqd            3/3     Running    0          5m54s
   ocsepp-release-ocpm-config-65dd85d96d-59t4w                    2/2     Running    0          5m54s
   ocsepp-release-performance-7456bbd8-2j7dx                      2/2     Running    0          5m54s
   ocsepp-release-plmn-egress-gateway-67b7864664-cmcf8            3/3     Running    0          5m54s
   ocsepp-release-plmn-egress-gateway-67b7864664-lwhxz            3/3     Running    0          4m31s
   ocsepp-release-plmn-ingress-gateway-596c78f967-sc44c           3/3     Running    0          5m53s
   ocsepp-release-pn32c-svc-6498f6dc-lrvtt                        2/2     Running    0          5m53s
   ocsepp-release-pn32f-svc-59bcb4c545-c4pqj                      2/2     Running    0          5m53s
   ocsepp-release-sepp-nrf-client-nfdiscovery-9db8957cb-47j6g     1/1     Running    0          5m54s
   ocsepp-release-sepp-nrf-client-nfmanagement-5ddfd8d754-nbx69   1/1     Running    0          5m54s
   sepp-mysql-54b7c5699d-5nmzc                                    1/1     Running    0          3d23h

2. Run the following command to enter Debug Tool Container:

   $ kubectl exec -it <pod name> -c <debug_container name> -n <namespace> bash

   Example:

   $ kubectl exec -it ocsepp-release-cn32c-svc-5f5cdbfb7f-kspw6 -c tools -n seppsvc bash

3. Run the commands supported by debug tools:

   bash -4.2$ <debug_tools>

   Example:

   bash -4.2$ tcpdump

4. Copy the output files from container to host:

   $ kubectl cp -c <debug_container name> <pod name>:<file location in container> -n <namespace> <destination location>

   Example:

   $ kubectl cp -c tools ocsepp-release-cn32c-svc-5f5cdbfb7f-kspw6:/tmp/capture.pcap -n seppsvc /tmp/

Tools Tested in Debug Container

Following is the list of debugging tools that are tested.

tcpdump

Table 4-1 tcpdump

Options Tested	Description	Output	Capabilities
-D	Print the list of the network interfaces available on the system and on which tcpdump can capture packets.	tcpdump -D eth02. nflog (Linux netfilter log (NFLOG) interface) nfqueue (Linux netfilter queue (NFQUEUE) interface) any (Pseudo-device that captures on all interfaces) lo [Loopback]	NET_ADMIN, NET_RAW
-i	Listen on interface	tcpdump -i eth0 tcpdump: verbose output suppressed, use -v or -vv for full protocol decodelistening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes12:10:37.381199 IP ocsepp-plmn-ingress-gateway-7ffc49bb7f-2kkhc.46519 > kubernetes.default.svc.cluster.local.https: Flags [P.], seq 1986927241:1986927276, ack 1334332290, win 626, options [nop,nop,TS val 849591834 ecr 849561833], length 3512:10:37.381952 IP ocsepp-plmn--ingress-gateway-7ffc49bb7f-2kkhc.45868 > kube-dns.kube-system.svc.cluster.local.domain: 62870+ PTR? 1.0.96.10.in-addr.arpa. (40)	NET_ADMIN, NET_RAW
-w	Write the raw packets to file rather than parsing and printing them out.	tcpdump -w capture.pcap -i eth0	NET_ADMIN, NET_RAW
-r	Read packets from file (which was created with the -w option).	tcpdump -r capture.pcap reading from file /tmp/capture.pcap, link-type EN10MB (Ethernet)12:13:07.381019 IP ocsepp-plmn-ingress-gateway-7ffc49bb7f-2kkhc.46519 > kubernetes.default.svc.cluster.local.https: Flags [P.], seq 1986927416:1986927451, ack 1334332445, win 626, options [nop,nop,TS val 849741834 ecr 849711834], length 3512:13:07.381194 IP kubernetes.default.svc.cluster.local.https > ocsepp-plmn-ingress-gateway-7ffc49bb7f-2kkhc.46519: Flags [P.], seq 1:32, ack 35, win 247, options [nop,nop,TS val 849741834 ecr 849741834], length 3112:13:07.381207 IP ocsepp-plmn-ingress-gateway-7ffc49bb7f-2kkhc.46519 > kubernetes.default.svc.cluster.local.https: Flags [.], ack 32, win 626, options [nop,nop,TS val 849741834 ecr 849741834], length 0	NET_ADMIN, NET_RAW

ip

Table 4-2 ip

Options Tested	Description	Output	Capabilities
addr show	Look at protocol addresses.	ip addr show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group defaultlink/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host lovalid_lft forever preferred_lft forever2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group defaultlink/ipip 0.0.0.0 brd 0.0.0.04: eth0@if190: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group defaultlink/ether aa:5a:27:8d:74:6f brd ff:ff:ff:ff:ff:ff link-netnsid 0inet 192.168.219.112/32 scope global eth0valid_lft forever preferred_lft forever	--
route show	List routes	ip route show default via 169.254.1.1 dev eth0 169.254.1.1 dev eth0 scope link	--
addrlabel list	List address labels	ip addrlabel list prefix ::1/128 label 0 prefix ::/96 label 3 prefix ::ffff:0.0.0.0/96 label 4 prefix 2001::/32 label 6 prefix 2001:10::/28 label 7 prefix 3ffe::/16 label 12 prefix 2002::/16 label 2 prefix fec0::/10 label 11 prefix fc00::/7 label 5 prefix ::/0 label 1	--

netstat

Table 4-3 netstat

Options Tested	Description	Output	Capabilities
-a	Show both listening and non-listening (for TCP, this means established connections) sockets.	netstat -a Active Internet connections (servers and established)Proto Recv-Q Send-Q Local Address Foreign Address Statetcp 0 0 0.0.0.0:tproxy 0.0.0.0:* LISTENtcp 0 0 0.0.0.0:websm 0.0.0.0:* LISTENtcp 0 0 cncc-core-ingress:websm 10-178-254-194.ku:47292 TIME_WAITtcp 0 0 cncc-core-ingress:46519 kubernetes.defaul:https ESTABLISHEDtcp 0 0 cncc-core-ingress:websm 10-178-254-194.ku:47240 TIME_WAITtcp 0 0 cncc-core-ingress:websm 10-178-254-194.ku:47347 TIME_WAITudp 0 0 localhost:59351 localhost:ambit-lm ESTABLISHEDActive UNIX domain sockets (servers and established)Proto RefCnt Flags Type State I-Node Pathunix 2 [ ] STREAM CONNECTED 576064861	--
-l	Show only listening sockets.	netstat -l Active Internet connections (only servers)Proto Recv-Q Send-Q Local Address Foreign Address Statetcp 0 0 0.0.0.0:tproxy 0.0.0.0:* LISTENtcp 0 0 0.0.0.0:websm 0.0.0.0:* LISTENActive UNIX domain sockets (only servers)Proto RefCnt Flags Type State I-Node Path	--
-s	Display summary statistics for each protocol.	netstat -s Ip:4070 total packets received0 forwarded0 incoming packets discarded4070 incoming packets delivered4315 requests sent outIcmp:0 ICMP messages received0 input ICMP message failed.ICMP input histogram:2 ICMP messages sent0 ICMP messages failedICMP output histogram:destination unreachable: 2	--
-i	Display a table of all network interfaces.	netstat -i Kernel Interface tableIface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flgeth0 1440 4131 0 0 0 4355 0 0 0 BMRUlo 65536 0 0 0 0 0 0 0 0 LRU	--

curl

Table 4-4 curl

Options Tested	Description	Output	Capabilities
-o	Write output to <file> instead of stdout.	curl -o file.txt http://abc.com/file.txt	--
-x	Use the specified HTTP proxy.	curl -x proxy.com:8080 -o http://abc.com/file.txt	--

ping

Table 4-5 ping

Options Tested	Description	Output	Capabilities
<ip>	Run a ping test to see whether the target host is reachable or not.	ping 10.178.254.194	NET_ADMIN, NET_RAW
-c	Stop after sending 'c' number of ECHO_REQUEST packets.	ping -c 5 10.178.254.194	NET_ADMIN, NET_RAW
-f (with non zero interval)	Flood ping. For every ECHO_REQUEST sent, a period ''.'' is printed, while for every ECHO_REPLY received a backspace is printed.	ping -f -i 2 10.178.254.194	NET_ADMIN, NET_RAW

dig

Table 4-6 dig

Options Tested	Description	Output	Capabilities
<ip>	It performs DNS lookups and displays the answers that are returned from the name server(s) that were queried.	dig 10.178.254.194 Note: The IP should be reachable from inside the container.	--
-x	Query DNS Reverse lookup.	dig -x 10.178.254.194	--

4.1 Debug Tool Configuration Parameters

Following are the parameters used to configure debug tool.

OCCNE Parameters

Table 4-7 OCCNE Parameters

Parameter	Description
apiVersion	APIVersion defines the version schema of this representation of an object.
kind	Kind is a string value representing the REST resource this object represents.
metadata	Standard object's metadata.
metadata.name	Name must be unique within a namespace.
spec	spec defines the policy enforced.
spec.allowPrivilegeEscalation	Gates whether or not a user is allowed to set the security context of a container to allowPrivilegeEscalation=true.
spec.allowedCapabilities	Provides a list of capabilities that are allowed to be added to a container.
spec.fsGroup	Controls the supplemental group applied to some volumes. RunAsAny allows any fsGroup ID to be specified.
spec.runAsUser	Controls which user ID the containers are run with. RunAsAny allows any runAsUser to be specified.
spec.seLinux	RunAsAny allows any seLinuxOptions to be specified.
spec.supplementalGroups	Controls which group IDs containers add. RunAsAny allows any supplementalGroups to be specified.
spec.volumes	Provides a list of allowed volume types. The allowable values correspond to the volume sources that are defined when creating a volume.

Role Creation Parameters

Table 4-8 Role Creation

Parameter	Description
apiVersion	APIVersion defines the versioned schema of this representation of an object.
kind	Kind is a string value representing the REST resource this object represents.
metadata	Standard object's metadata.
metadata.name	Name must be unique within a namespace.
metadata.namespace	Namespace defines the space within which each name must be unique.
rules	Rules holds all the PolicyRules for this Role
apiGroups	APIGroups is the name of the APIGroup that contains the resources.
rules.resources	Resources is a list of resources this rule applies to.
rules.verbs	Verbs is a list of Verbs that apply to ALL the ResourceKinds and AttributeRestrictions contained in this rule.
rules.resourceNames	ResourceNames is an optional allowed list of names that the rule applies to.

Table 4-9 Role Binding Creation

Parameter	Description
apiVersion	APIVersion defines the versioned schema of this representation of an object.
kind	Kind is a string value representing the REST resource this object represents.
metadata	Standard object's metadata.
metadata.name	Name must be unique within a namespace.
metadata.namespace	Namespace defines the space within which each name must be unique.
roleRef	RoleRef can reference a Role in the current namespace or a ClusterRole in the global namespace.
roleRef.apiGroup	APIGroup is the group for the resource being referenced
roleRef.kind	Kind is the type of resource being referenced
roleRef.name	Name is the name of resource being referenced
subjects	Subjects holds references to the objects the role applies to.
subjects.kind	Kind of object being referenced. Values defined by this API group are "User", "Group", and "ServiceAccount".
subjects.apiGroup	APIGroup holds the API group of the referenced subject.
subjects.name	Name of the object being referenced appended by namespace.

Debug Tool Configuration Parameters

Table 4-10 Debug Tool Configuration Parameters

Parameter	Description
extraContainers	Specifies the spawns debug container along with application container in the pod.
debugToolContainerMemoryLimit	Indicates the memory assigned for the debug tool container.
extraContainersVolumesTpl	Specifies the extra container template for the debug tool volume.
extraContainersVolumesTpl.name	Indicates the name of the volume for debug tool logs storage.
extraContainersVolumesTpl.emptyDir.medium	Indicates the location where emptyDir volume is stored.
extraContainersVolumesTpl.emptyDir.sizeLimit	Indicates the emptyDir volume size.
command	String array used for container command.
image	Docker image name
imagePullPolicy	Image Pull Policy
name	Name of the container
resources	Compute Resources required by this container
resources.limits	Limits describes the maximum amount of compute resources allowed
resources.requests	Requests describes the minimum amount of compute resources required
resources.limits.cpu	CPU limits
resources.limits.memory	Memory limits
resources.limits.ephemeral-storage	Ephemeral Storage limits
resources.requests.cpu	CPU requests
resources.requests.memory	Memory requests
resources.requests.ephemeral-storage	Ephemeral Storage requests
securityContext	Security options the container should run with.
securityContext.allowPrivilegeEscalation	AllowPrivilegeEscalation controls whether a process can gain more privileges than its parent process. This directly controls if the no_new_privs flag will be set on the container process
securityContext.capabilities	The capabilities to add or drop when running containers. Defaults to the default set of capabilities granted by the container runtime.
securityContext.capabilities.drop	Removed capabilities
secuirtyContext.capabilities.add	Added capabilities
securityContext.runAsUser	The UID to run the entry point of the container process.
volumeMounts.mountPath	Indicates the path for volume mount.
volumeMounts.name	Indicates the name of the directory for debug tool logs storage.