Debug Tool

3 Debug Tool

Overview

The Debug Tool provides third-party troubleshooting tools for debugging the runtime issues in a lab environment. Following are the available tools:

tcpdump
ip
netstat
curl
ping
nmap
dig

Prerequisites

This section explains the preconfiguration steps for using the debug tool:

Note:

For CNE 23.2.0 and later versions, follow Step a of Configuration in CNE.
For CNE versions prior to 23.2.0, follow Step b of Configuration in CNE.

Configuration in CNE
Perform the following configurations in the Bastion Host. You need admin privileges to perform these configurations.
1. When NSSF is installed on CNE version 23.2.0 or above
  Note:
  - In CNE version 23.2.0 or above, the default CNE 23.2.0 Kyverno policy, disallow-capabilities, do not allow NET_ADMIN and NET_RAW capabilities that are required for debug tool.
  - To run Debug tool on CNE 23.2.0 and above, the user must modify the existing Kyverno policy, disallow-capabilities, as below.
  Adding a Namespace to an Empty Resource
  1. Run the following command to verify if the current disallow capabilities cluster policy has namespace in it.
    Example:
```
$ kubectl get clusterpolicies disallow-capabilities -oyaml
```
    Sample output:
```
apiVersion: kyverno.io/v1
kind: ClusterPolicy
...
...
spec:
  rules:
  -exclude:
      any:
      -resources:{}
```
  2. If there are no namespaces, then patch the policy using the following command to add <namespace> under resources:
```
$ kubectl patch clusterpolicy disallow-capabilities --type=json \
  -p='[{"op": "add", "path": "/spec/rules/0/exclude/any/0/resources", "value": {"namespaces":["<namespace>"]} }]'
```
    Example:
```
$ kubectl patch clusterpolicy disallow-capabilities --type=json \
  -p='[{"op": "add", "path": "/spec/rules/0/exclude/any/0/resources", "value": {"namespaces":["ocnssf"]} }]'
```
    Sample output:
```
apiVersion: kyverno.io/v1
kind: ClusterPolicy
...
...
spec:
  rules:
  -exclude:
      resources:
        namespaces:
        -ocnssf
```
  3. If in case it is needed to remove the namespace added in the above step, use the following command:
```
$ kubectl patch clusterpolicy disallow-capabilities --type=json \
  -p='[{"op": "replace", "path": "/spec/rules/0/exclude/any/0/resources", "value": {} }]'
```
    Sample output:
```
apiVersion: kyverno.io/v1
kind: ClusterPolicy
...
...
spec:
  rules:
  -exclude:
      any:
      -resources:{}
```
  Adding a Namespace to an Existing Namespace List
  1. Run the following command to verify if the current disallow-capabilities cluster policy has namespaces in it.
    Example:
```
$ kubectl get clusterpolicies disallow-capabilities -oyaml
```
    Sample output:
```
apiVersion: kyverno.io/v1
kind: ClusterPolicy
...
...
spec:
  rules:
  -exclude:
      any:
      -resources:
          namespaces:
          -namespace1
          -namespace2
          -namespace3
```
  2. If there are namespaces already added, then patch the policy using the following command to add <namespace> to the existing list:
```
$ kubectl patch clusterpolicy disallow-capabilities --type=json \
  -p='[{"op": "add", "path": "/spec/rules/0/exclude/any/0/resources/namespaces/-", "value": "<namespace>" }]'
```
    Example:
```
$ kubectl patch clusterpolicy disallow-capabilities --type=json \
  -p='[{"op": "add", "path": "/spec/rules/0/exclude/any/0/resources/namespaces/-", "value": "ocnssf" }]'
```
    Sample output:
```
apiVersion: kyverno.io/v1
kind: ClusterPolicy
...
...
spec:
  rules:
  -exclude:
      resources:
        namespaces:
        -namespace1
        -namespace2
        -namespace3
        -ocnssf
```
  3. If in case it is needed to remove the namespace added in the above step, use the following command:
```
$ kubectl patch clusterpolicy disallow-capabilities --type=json \
  -p='[{"op": "remove", "path": "/spec/rules/0/exclude/any/0/resources/namespaces/<index>"}]'
```
    Example:
```
$ kubectl patch clusterpolicy disallow-capabilities --type=json \
  -p='[{"op": "remove", "path": "/spec/rules/0/exclude/any/0/resources/namespaces/3"}]'
```
    Sample output:
```
apiVersion: kyverno.io/v1
kind: ClusterPolicy
...
...
spec:
  rules:
  -exclude:
      resources:
        namespaces:
        -namespace1
        -namespace2
        -namespace3
```
    Note:
    While removing the namespace, provide the index value for namespace within the array. The index starts from '0'.
2. When NSSF is installed on CNE version prior to 23.2.0
  PodSecurityPolicy (PSP) Creation
  1. Log in to the Bastion Host.
  2. Run the following command from the Bastion Host to create a new PSP. The parameters readOnlyRootFileSystem, allowPrivilegeEscalation, allowedCapabilities are required by debug container.
    
    Note:
    Other parameters are mandatory for PSP creation and can be customized as per the CNE environment. Default values are recommended.
```
$ kubectl apply -f - <<EOF

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: debug-tool-psp
spec:
  readOnlyRootFilesystem: false
  allowPrivilegeEscalation: true
  allowedCapabilities:
  - NET_ADMIN
  - NET_RAW
  fsGroup:
    ranges:
    - max: 65535
      min: 1
    rule: MustRunAs
  runAsUser:
    rule: MustRunAsNonRoot
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  volumes:
  - configMap
  - downwardAPI
  - emptyDir
  - persistentVolumeClaim
  - projected
  - secret
EOF
```
  Role Creation
  Run the following command to create a role for the PSP:
```
kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: debug-tool-role
  namespace: cncc
rules:
- apiGroups:
  - policy
  resources:
  - podsecuritypolicies
  verbs:
  - use
  resourceNames:
  - debug-tool-psp
EOF
```
  RoleBinding Creation
  Run the following command to attach the service account for your NF namespace with the role created for the tool PSP:
```
$ kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: debug-tool-rolebinding
  namespace: ocnssf
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: debug-tool-role
subjects:
- kind: Group
  apiGroup: rbac.authorization.k8s.io
  name: system:serviceaccounts
EOF
```
Configuration in NF Specific Helm
Perform the following updates in ocnssf_custom_values_23.4.0.yaml file.
1. Log in to the NF server.
2. Open the ocnssf_custom_values_23.4.0.yaml file:
```
$ vim <custom_values file>
```
  Example:
```
$vim ocnssf_custom_values_23.4.0.yaml
```
3. Under global configuration, add the following:
```
# Allowed Values: DISABLED, ENABLED
  podSecurityPolicy: "DISABLED"
  extraContainers: "DISABLED"
  debugToolContainerMemoryLimit: 4Gi
  extraContainersImageDetails:
    image: ocdebugtool/ocdebug-tools
    tag: debug_container_tag
    imagePullPolicy: Always
  extraContainersVolumesTpl: |
    - name: debug-tools-dir
      emptyDir:
        medium: Memory
        sizeLimit: {{ .Values.global.debugToolContainerMemoryLimit | quote }}
  extraContainersTpl: |-
    - command:
        - /bin/sleep
        - infinity
      name: tools
      resources:
        requests:
          ephemeral-storage: "512Mi"
          cpu: "0.5"
          memory: {{ .Values.global.debugToolContainerMemoryLimit | quote }}
        limits:
          ephemeral-storage: "512Mi"
          cpu: "1"
          memory: {{ .Values.global.debugToolContainerMemoryLimit | quote }}
      securityContext:
        allowPrivilegeEscalation: true
        capabilities:
          drop:
          - ALL
          add:
          - NET_RAW
          - NET_ADMIN
        runAsUser: 1012
      volumeMounts:
      - mountPath: /tmp/tools
        name: debug-tools-dir
```
  Note:
  - Debug Tool Container comes up with the default user ID - 7000. To override the default value, use "runAsUser" field, otherwise the field can be skipped.
    
    Default value: uid=7000 (debugtool) gid=7000 (debugtool) groups=7000 (debugtool)
  - In case you want to customize the container name, replace the `name` field in the ocnssf_custom_values_23.4.0.yaml file with the following:
```
name: {{ printf "%s-tools-%s" (include "getprefix" .) (include "getsuffix" .) | trunc 63 | trimPrefix "-" | trimSuffix "-"  }}
```
    This ensures that the container name is prefixed and suffixed with the necessary values.
4. Under service specific configurations for which debugging is required, add the following:
```
# Allowed Values: DISABLED, ENABLED, USE_GLOBAL_VALUE
extraContainers: USE_GLOBAL_VALUE
```
  Note:
  - At the global level, use extraContainers flag to enable or disable injection of extra containers globally. This ensures that all the services that use this global value have extra containers enabled or disabled using a single flag.
  - At the service level, use extraContainers flag to determine whether to use the extra container configuration from the global level or enable or disable injecting extra containers for the specific service.

Run the Debug Tool

To run Debug Tool, perform the following steps:

Run the following command to retrieve the POD details:

$ kubectl get pods -n <k8s namespace>

Example:

$ kubectl get pods -n ocnssf

Sample Output:


NAME                                                 READY  STATUS  RESTARTS  AGE
ocnssf-appinfo-659745778c-58jqc                       1/1    Running    6      2d
ocnssf-egress-gateway-5d5cd8bb9-fln4w                 1/1    Running    7      2d
ocnssf-ingress-gateway-98b65b4d9-cs8jb                1/1    Running    7      2d
ocnssf-nsavailability-8699784c8d-8wwd5                1/1    Running    9      2d
ocnssf-nsconfig-5dd646cf76-66k9g                      1/1    Running    9      2d
ocnssf-nsselection-67b7bd9bcf-2nv56                   1/1    Running    7      2d
ocnssf-nssubscription-5c96d8b9cc-fzwnt                1/1    Running    11     2d
ocnssf-ocnssf-nrf-client-nfdiscovery-7646cfc547-5hpsd 1/1    Running    5      2d
ocnssf-ocnssf-nrf-client-nfdiscovery-7646cfc547-lc96j 1/1    Running    5      2d
ocnssf-ocnssf-nrf-client-nfmanagement-8f7bfb98-s4b4b  1/1    Running    5      2d
ocnssf-ocpm-config-5c475d4646-lxvzq                   1/1    Running    5      2d

Run the following command to enter Debug Tool Container:

$ kubectl exec -it <pod name> -c <debug_container name> -n <namespace> bash

Example:

$ kubectl exec -it ocnssf-nsselection-67b7bd9bcf-2nv56 -c tools -n ocnssf bash

Run the debug tools:
```
bash -4.2$ <debug_tools>
```
Example:
```
bash -4.2$ tcpdump
```

Copy the output files from container to host:

$ kubectl cp -c <debug_container name> <pod name>:<file location in container> -n <namespace> <destination location>

Example:

$ kubectl cp -c tools ocnssf-nsselection-67b7bd9bcf-2nv56:/tmp/capture.pcap -n ocnssf /tmp/

Tools Tested in Debug Container

Following is the list of debug tools that are tested:

tcpdump

The details of tcpdump debug tool are as follows:

Table 3-1 tcpdump

Options Tested	Description	Details
`-D`	Print the list of the network interfaces available on the system and on which tcpdump can capture packets.	Output: `tcpdump -D` eth02. nflog (Linux netfilter log (NFLOG) interface) nfqueue (Linux netfilter queue (NFQUEUE) interface) any (Pseudo-device that captures on all interfaces) lo [Loopback] Capabilities: NET_ADMIN, NET_RAW
`-i`	Listen on interface.	Output: `tcpdump -i eth0` tcpdump: verbose output suppressed, use -v or -vv for full protocol decodelistening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes12:10:37.381199 IP cncc-core-ingress-gateway-7ffc49bb7f-2kkhc.46519 > kubernetes.default.svc.cluster.local.https: Flags [P.], seq 1986927241:1986927276, ack 1334332290, win 626, options [nop,nop,TS val 849591834 ecr 849561833], length 3512:10:37.381952 IP cncc-core-ingress-gateway-7ffc49bb7f-2kkhc.45868 > kube-dns.kube-system.svc.cluster.local.domain: 62870+ PTR? 1.0.96.10.in-addr.arpa. (40) Capabilities: NET_ADMIN, NET_RAW
`-w`	Write the raw packets to file rather than parsing and printing them out.	Output: `tcpdump -w capture.pcap -i eth0` Capabilities: NET_ADMIN, NET_RAW
`-r`	Read packets from file (which was created with the -w option).	Output: `tcpdump -r capture.pcap` reading from file /tmp/capture.pcap, link-type EN10MB (Ethernet)12:13:07.381019 IP cncc-core-ingress-gateway-7ffc49bb7f-2kkhc.46519 > kubernetes.default.svc.cluster.local.https: Flags [P.], seq 1986927416:1986927451, ack 1334332445, win 626, options [nop,nop,TS val 849741834 ecr 849711834], length 3512:13:07.381194 IP kubernetes.default.svc.cluster.local.https > cncc-core-ingress-gateway-7ffc49bb7f-2kkhc.46519: Flags [P.], seq 1:32, ack 35, win 247, options [nop,nop,TS val 849741834 ecr 849741834], length 3112:13:07.381207 IP cncc-core-ingress-gateway-7ffc49bb7f-2kkhc.46519 > kubernetes.default.svc.cluster.local.https: Flags [.], ack 32, win 626, options [nop,nop,TS val 849741834 ecr 849741834], length 0 Capabilities: NET_ADMIN, NET_RAW

The details of ip debug tool are as follows:

Table 3-2 ip

Options Tested Description Details

Options Tested	Description	Details
`addr show`	Look at protocol addresses.	Output: `ip addr show` lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group defaultlink/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host lovalid_lft forever preferred_lft forever tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group defaultlink/ipip 0.0.0.0 brd 0.0.0.0 eth0@if190: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group defaultlink/ether aa:5a:27:8d:74:6f brd ff:ff:ff:ff:ff:ff link-netnsid 0inet 192.168.219.112/32 scope global eth0valid_lft forever preferred_lft forever
`route show`	List routes.	Output: `ip route show` default via 169.254.1.1 dev eth0 169.254.1.1 dev eth0 scope link
`addrlabel list`	List address labels	Output: `ip addrlabel list` prefix ::1/128 label 0 prefix ::/96 label 3 prefix ::ffff:0.0.0.0/96 label 4 prefix 2001::/32 label 6 prefix 2001:10::/28 label 7 prefix 3ffe::/16 label 12 prefix 2002::/16 label 2 prefix fec0::/10 label 11 prefix fc00::/7 label 5 prefix ::/0 label 1

addr show

Look at protocol addresses.

Output:

ip addr show

lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group defaultlink/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host lovalid_lft forever preferred_lft forever
tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group defaultlink/ipip 0.0.0.0 brd 0.0.0.0
eth0@if190: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group defaultlink/ether aa:5a:27:8d:74:6f brd ff:ff:ff:ff:ff:ff link-netnsid 0inet 192.168.219.112/32 scope global eth0valid_lft forever preferred_lft forever

route show

List routes.

Output:

ip route show

default via 169.254.1.1 dev eth0

169.254.1.1 dev eth0 scope link

addrlabel list

List address labels

Output:

ip addrlabel list

prefix ::1/128 label 0

prefix ::/96 label 3

prefix ::ffff:0.0.0.0/96 label 4

prefix 2001::/32 label 6

prefix 2001:10::/28 label 7

prefix 3ffe::/16 label 12

prefix 2002::/16 label 2

prefix fec0::/10 label 11

prefix fc00::/7 label 5

prefix ::/0 label 1

netstat

The details of netstat debug tool are as follows:

Table 3-3 netstat

Options Tested	Description	Details
`-a`	Show both listening and non-listening sockets. For TCP, this means established connections.	Output: `netstat -a` Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 0.0.0.0:tproxy 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:websm 0.0.0.0:* LISTEN tcp 0 0 ocnssf-ingress:websm 10-178-254-194.ku:47292 TIME_WAIT tcp 0 0 ocnssf-ingress:46519 kubernetes.defaul:https ESTABLISHED tcp 0 0 ocnssf-ingress:websm 10-178-254-194.ku:47240 TIME_WAIT tcp 0 0 ocnssf-ingress:websm 10-178-254-194.ku:47347 TIME_WAIT udp 0 0 localhost:59351 localhost:ambit-lm ESTABLISHED Active UNIX domain sockets (servers and established) Proto RefCnt Flags Type State I-Node Path unix 2 [ ] STREAM CONNECTED 576064861
`-l`	Show only listening sockets.	Output: `netstat -l` Active Internet connections (only servers)Proto Recv-Q Send-Q Local Address Foreign Address Statetcp 0 0 0.0.0.0:tproxy 0.0.0.0:* LISTENtcp 0 0 0.0.0.0:websm 0.0.0.0:* LISTENActive UNIX domain sockets (only servers)Proto RefCnt Flags Type State I-Node Path
`-s`	Display summary statistics for each protocol.	Output: `netstat -s` `Ip: 4070 total packets received 0 forwarded 0 incoming packets discarded 4070 incoming packets delivered 4315 requests sent out Icmp: 0 ICMP messages received 0 input ICMP message failed. ICMP input histogram: 2 ICMP messages sent 0 ICMP messages failed ICMP output histogram: destination unreachable: 2`
`-i`	Display a table of all network interfaces.	Output: `netstat -i` Kernel Interface tableIface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flgeth0 1440 4131 0 0 0 4355 0 0 0 BMRUlo 65536 0 0 0 0 0 0 0 0 LRU

curl

The details of curl debug tool are as follows:

Table 3-4 curl

Options Tested Description Details

Options Tested	Description	Details
`-o`	Write output to <file> instead of stdout.	Output: `curl -o file.txt http://abc.com/file.txt`
`-x`	Use the specified HTTP proxy.	Output: `curl -x proxy.com:8080 -o http://abc.com/file.txt`
--http2	HTTP/2	--

-o

Write output to <file> instead of stdout.

Output:

curl -o file.txt http://abc.com/file.txt

-x

Use the specified HTTP proxy.

Output:

curl -x proxy.com:8080 -o http://abc.com/file.txt

--http2 HTTP/2 --

ping

The details of ping debug tool are as follows:

Table 3-5 ping

Options Tested Description Details

<ip>

Run a ping test to see whether the target host is reachable or not.

Options Tested	Description	Details
<ip>	Run a ping test to see whether the target host is reachable or not.	Output: `ping 10.178.254.194` `PING 10.178.254.194 (10.178.254.194) 56(84) bytes of data. 64 bytes from 10.178.254.194: icmp_seq=1 ttl=64 time=0.044 ms 64 bytes from 10.178.254.194: icmp_seq=2 ttl=64 time=0.048 ms 64 bytes from 10.178.254.194: icmp_seq=3 ttl=64 time=0.047 ms 64 bytes from 10.178.254.194: icmp_seq=4 ttl=64 time=0.057 ms 64 bytes from 10.178.254.194: icmp_seq=5 ttl=64 time=0.046 ms` Capabilities: NET_ADMIN, NET_RAW
`-c`	Stop after sending 'c' number of ECHO_REQUEST packets.	Output: `ping -c 5 10.178.254.194` PING 10.178.254.194 (10.178.254.194) 56(84) bytes of data. 64 bytes from 10.178.254.194: icmp_seq=1 ttl=64 time=0.051 ms 64 bytes from 10.178.254.194: icmp_seq=2 ttl=64 time=0.036 ms 64 bytes from 10.178.254.194: icmp_seq=3 ttl=64 time=0.037 ms 64 bytes from 10.178.254.194: icmp_seq=4 ttl=64 time=0.037 ms 64 bytes from 10.178.254.194: icmp_seq=5 ttl=64 time=0.049 ms --- 10.178.254.194 ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 3999ms rtt min/avg/max/mdev = 0.036/0.042/0.051/0.006 ms Capabilities: NET_ADMIN, NET_RAW
`-f` (with non zero interval)	Flood ping. For every ECHO_REQUEST sent a period ''.'' is printed, while for every ECHO_REPLY received a backspace is printed.	Output: `ping -f -i 2 10.178.254.194` `PING 10.178.254.194 (10.178.254.194) 56(84) bytes of data. --- 10.178.254.194 ping statistics --- 251 packets transmitted, 251 received, 0% packet loss, time 75ms rtt min/avg/max/mdev = 0.029/0.048/0.210/0.019 ms, ipg/ewma 300.001/0.044 ms` Capabilities: NET_ADMIN, NET_RAW

Output:

ping 10.178.254.194

PING 10.178.254.194 (10.178.254.194) 56(84) bytes of data.
64 bytes from 10.178.254.194: icmp_seq=1 ttl=64 time=0.044 ms
64 bytes from 10.178.254.194: icmp_seq=2 ttl=64 time=0.048 ms
64 bytes from 10.178.254.194: icmp_seq=3 ttl=64 time=0.047 ms
64 bytes from 10.178.254.194: icmp_seq=4 ttl=64 time=0.057 ms
64 bytes from 10.178.254.194: icmp_seq=5 ttl=64 time=0.046 ms

Capabilities:

NET_ADMIN, NET_RAW

-c

Stop after sending 'c' number of ECHO_REQUEST packets.

Output:

ping -c 5 10.178.254.194

PING 10.178.254.194 (10.178.254.194) 56(84) bytes of data.
64 bytes from 10.178.254.194: icmp_seq=1 ttl=64 time=0.051 ms
64 bytes from 10.178.254.194: icmp_seq=2 ttl=64 time=0.036 ms
64 bytes from 10.178.254.194: icmp_seq=3 ttl=64 time=0.037 ms
64 bytes from 10.178.254.194: icmp_seq=4 ttl=64 time=0.037 ms
64 bytes from 10.178.254.194: icmp_seq=5 ttl=64 time=0.049 ms

--- 10.178.254.194 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 3999ms
rtt min/avg/max/mdev = 0.036/0.042/0.051/0.006 ms

Capabilities:

NET_ADMIN, NET_RAW

-f (with non zero interval)

Flood ping. For every ECHO_REQUEST sent a period ''.'' is printed, while for every ECHO_REPLY received a backspace is printed.

Output:

ping -f -i 2 10.178.254.194

PING 10.178.254.194 (10.178.254.194) 56(84) bytes of data.
--- 10.178.254.194 ping statistics ---
251 packets transmitted, 251 received, 0% packet loss, time 75ms
rtt min/avg/max/mdev = 0.029/0.048/0.210/0.019 ms, ipg/ewma 300.001/0.044 ms

Capabilities:

NET_ADMIN, NET_RAW

nmap

The details of nmap debug tool are as follows:

Table 3-6 nmap

Options Tested Description Details

<ip>

Scan for Live hosts, Operating systems, packet filters, and open ports running on remote hosts.

Options Tested	Description	Details
<ip>	Scan for Live hosts, Operating systems, packet filters, and open ports running on remote hosts.	Output: `nmap 10.178.254.194` `Starting Nmap 6.40 ( http://nmap.org ) at 2020-09-29 05:54 UTC Nmap scan report for 10-178-254-194.kubernetes.default.svc.cluster.local (10.178.254.194) Host is up (0.00046s latency). Not shown: 995 closed ports PORT STATE SERVICE 22/tcp open ssh 179/tcp open bgp 6666/tcp open irc 6667/tcp open irc 30000/tcp open unknown Nmap done: 1 IP address (1 host up) scanned in 0.04 seconds`
`-v`	Increase verbosity level	Output: `nmap -v 10.178.254.194` Starting Nmap 6.40 ( http://nmap.org ) at 2020-09-29 05:55 UTC Initiating Ping Scan at 05:55 Scanning 10.178.254.194 [2 ports] Completed Ping Scan at 05:55, 0.00s elapsed (1 total hosts) Initiating Parallel DNS resolution of 1 host. at 05:55 Completed Parallel DNS resolution of 1 host. at 05:55, 0.00s elapsed Initiating Connect Scan at 05:55 Scanning 10-178-254-194.kubernetes.default.svc.cluster.local (10.178.254.194) [1000 ports] Discovered open port 22/tcp on 10.178.254.194 Discovered open port 30000/tcp on 10.178.254.194 Discovered open port 6667/tcp on 10.178.254.194 Discovered open port 6666/tcp on 10.178.254.194 Discovered open port 179/tcp on 10.178.254.194 Completed Connect Scan at 05:55, 0.02s elapsed (1000 total ports) Nmap scan report for 10-178-254-194.kubernetes.default.svc.cluster.local (10.178.254.194) Host is up (0.00039s latency). Not shown: 995 closed ports PORT STATE SERVICE 22/tcp open ssh 179/tcp open bgp 6666/tcp open irc 6667/tcp open irc 30000/tcp open unknown Read data files from: /usr/bin/../share/nmap Nmap done: 1 IP address (1 host up) scanned in 0.04 seconds
`-iL`	Scan all the listed IP addresses in a file. Sample file `localhost 10.178.254.194`	Output: `nmap -iL sample.txt` Starting Nmap 6.40 ( http://nmap.org ) at 2020-09-29 05:57 UTC Nmap scan report for localhost (127.0.0.1) Host is up (0.00036s latency). Other addresses for localhost (not scanned): 127.0.0.1 Not shown: 998 closed ports PORT STATE SERVICE 8081/tcp open blackice-icecap 9090/tcp open zeus-admin Nmap scan report for 10-178-254-194.kubernetes.default.svc.cluster.local (10.178.254.194) Host is up (0.00040s latency). Not shown: 995 closed ports PORT STATE SERVICE 22/tcp open ssh 179/tcp open bgp 6666/tcp open irc 6667/tcp open irc 30000/tcp open unknown Nmap done: 2 IP addresses (2 hosts up) scanned in 0.06 seconds

Output:

nmap 10.178.254.194

Starting Nmap 6.40 ( http://nmap.org ) at 2020-09-29 05:54 UTC
Nmap scan report for 10-178-254-194.kubernetes.default.svc.cluster.local (10.178.254.194)
Host is up (0.00046s latency).
Not shown: 995 closed ports
PORT STATE SERVICE
22/tcp open ssh
179/tcp open bgp
6666/tcp open irc
6667/tcp open irc
30000/tcp open unknown

Nmap done: 1 IP address (1 host up) scanned in 0.04 seconds

-v

Increase verbosity level

Output:

nmap -v 10.178.254.194

Starting Nmap 6.40 ( http://nmap.org ) at 2020-09-29 05:55 UTC
Initiating Ping Scan at 05:55
Scanning 10.178.254.194 [2 ports]
Completed Ping Scan at 05:55, 0.00s elapsed (1 total hosts)
Initiating Parallel DNS resolution of 1 host. at 05:55
Completed Parallel DNS resolution of 1 host. at 05:55, 0.00s elapsed
Initiating Connect Scan at 05:55
Scanning 10-178-254-194.kubernetes.default.svc.cluster.local (10.178.254.194) [1000 ports]
Discovered open port 22/tcp on 10.178.254.194
Discovered open port 30000/tcp on 10.178.254.194
Discovered open port 6667/tcp on 10.178.254.194
Discovered open port 6666/tcp on 10.178.254.194
Discovered open port 179/tcp on 10.178.254.194
Completed Connect Scan at 05:55, 0.02s elapsed (1000 total ports)
Nmap scan report for 10-178-254-194.kubernetes.default.svc.cluster.local (10.178.254.194)
Host is up (0.00039s latency).
Not shown: 995 closed ports
PORT STATE SERVICE
22/tcp open ssh
179/tcp open bgp
6666/tcp open irc
6667/tcp open irc
30000/tcp open unknown

Read data files from: /usr/bin/../share/nmap
Nmap done: 1 IP address (1 host up) scanned in 0.04 seconds

-iL

Scan all the listed IP addresses in a file. Sample file


localhost
10.178.254.194

Output:

nmap -iL sample.txt

Starting Nmap 6.40 ( http://nmap.org ) at 2020-09-29 05:57 UTC
Nmap scan report for localhost (127.0.0.1)
Host is up (0.00036s latency).
Other addresses for localhost (not scanned): 127.0.0.1
Not shown: 998 closed ports
PORT STATE SERVICE
8081/tcp open blackice-icecap
9090/tcp open zeus-admin

Nmap scan report for 10-178-254-194.kubernetes.default.svc.cluster.local (10.178.254.194)
Host is up (0.00040s latency).
Not shown: 995 closed ports
PORT STATE SERVICE
22/tcp open ssh
179/tcp open bgp
6666/tcp open irc
6667/tcp open irc
30000/tcp open unknown

Nmap done: 2 IP addresses (2 hosts up) scanned in 0.06 seconds

dig

The details of dig debug tool are as follows:

Table 3-7 dig

Options Tested Description Output

<ip>

It performs DNS lookups and displays the answers that are returned from the name server(s) that were queried.

Options Tested	Description	Output
<ip>	It performs DNS lookups and displays the answers that are returned from the name server(s) that were queried.	Output: `dig 10.178.254.194` Note: The IP should be reachable from inside the container.
`-x`	Query DNS Reverse Look-up.	Output: `dig -x 10.178.254.194` ; <<>> DiG 9.9.4-RedHat-9.9.4-61.el7 <<>> -x 10.178.254.194 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 59845 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;194.254.178.10.in-addr.arpa. IN PTR ;; ANSWER SECTION: 194.254.178.10.in-addr.arpa. 0 IN PTR master.openstack.internal. ;; Query time: 1 msec ;; SERVER: 10.178.254.131#53(10.178.254.131) ;; WHEN: Tue Sep 29 08:08:29 GMT 2020 ;; MSG SIZE rcvd: 95

Output:

dig 10.178.254.194

Note: The IP should be reachable from inside the container.

-x

Query DNS Reverse Look-up.

Output:

dig -x 10.178.254.194


; <<>> DiG 9.9.4-RedHat-9.9.4-61.el7 <<>> -x 10.178.254.194
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 59845
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;194.254.178.10.in-addr.arpa. IN PTR

;; ANSWER SECTION:
194.254.178.10.in-addr.arpa. 0 IN PTR master.openstack.internal.

;; Query time: 1 msec
;; SERVER: 10.178.254.131#53(10.178.254.131)
;; WHEN: Tue Sep 29 08:08:29 GMT 2020
;; MSG SIZE rcvd: 95

3.1 Debug Tool Configuration Parameters

Following are the parameters used to configure NSSF debug tool.

CNE Parameters

Table 3-8 CNE Parameters

Parameter	Description
`apiVersion`	`apiVersion` defines the version schema of this representation of an object.
`kind`	Kind is a string value representing the REST resource this object represents.
`metadata`	Standard object's metadata.
`metadata.name`	Name must be unique within a namespace.
`spec`	`spec` defines the policy enforced.
`spec.readOnlyRootFilesystem`	Controls whether the containers run with a read-only root filesystem (i.e. no writable layer).
`spec.allowPrivilegeEscalation`	Gates whether or not a user is allowed to set the security context of a container to `allowPrivilegeEscalation=true`.
`spec.allowedCapabilities`	Provides a list of capabilities that are allowed to be added to a container.
`spec.fsGroup`	Controls the supplemental group applied to some volumes. `RunAsAny` allows any `fsGroup` ID to be specified.
`spec.runAsUser`	Controls which user ID the containers are run with. `RunAsAny` allows any `runAsUser` to be specified.
`spec.seLinux`	`RunAsAny` allows any `seLinuxOptions` to be specified.
`spec.supplementalGroups`	Controls which group IDs containers add. `RunAsAny` allows any `supplementalGroups` to be specified.
`spec.volumes`	Provides a list of allowed volume types. The allowed values correspond to the volume sources that are defined when creating a volume.

Role Creation Parameters

Table 3-9 Role Creation

Parameter	Description
`apiVersion`	`apiVersion` defines the versioned schema of this representation of an object.
`kind`	Kind is a string value representing the REST resource this object represents.
`metadata`	Standard object's metadata.
`metadata.name`	Name must be unique within a namespace.
`metadata.namespace`	Namespace defines the space within which each name must be unique.
`rules`	Rules holds all the Policy Rules for this Role
`apiGroups`	`apiGroups` is the name of the API Group that contains the resources.
`rules.resources`	Resources is a list of resources this rule applies to.
`rules.verbs`	Verbs is a list of verbs that apply to ALL the `ResourceKinds` and `AttributeRestrictions` contained in this rule.
`rules.resourceNames`	`ResourceNames` is an optional white list of names that the rule applies to.

Table 3-10 Role Binding Creation

Parameter	Description
`apiVersion`	`apiVersion` defines the versioned schema of this representation of an object.
`kind`	`kind` is a string value representing the REST resource this object represents.
`metadata`	Standard object's metadata.
`metadata.name`	Name must be unique within a namespace.
`metadata.namespace`	Namespace defines the space within which each name must be unique.
`roleRef`	`RoleRef` can reference a Role in the current namespace or a `ClusterRole` in the global namespace.
`roleRef.apiGroup`	`apiGroup` is the group for the resource being referenced
`roleRef.kind`	Kind is the type of resource being referenced
`roleRef.name`	Name is the name of the resource being referenced
`subjects`	Subjects hold references to the objects the role applies to.
`subjects.kind`	Kind of object being referenced. Values defined by this API group are "User", "Group", and "ServiceAccount".
`subjects.apiGroup`	APIGroup holds the `apiGroup` of the referenced subject.
`subjects.name`	Name of the object being referenced.

Debug Tool Configuration Parameters

Table 3-11 Debug Tool Configuration Parameters

Parameter	Description
`extraContainers`	Container for debug
`debugToolContainerMemoryLimit`	Indicates the memory assigned for the debug tool container.
`extraContainersVolumesTpl`	Specifies the extra container template for the debug tool volume.
`extraContainersVolumesTpl.name`	Indicates the name of the volume for debug tool logs storage.
`extraContainersVolumesTpl.emptyDir.medium`	Indicates the location where emptyDir volume is stored.
`extraContainersVolumesTpl.emptyDir.sizeLimit`	Indicates the emptyDir volume size.
`command`	String array used for container command.
`image`	Docker image name
`imagePullPolicy`	Image Pull Policy
`name`	Name of the container
`resources`	Compute Resources required by this container
`resources.limits`	Limits describe the maximum amount of compute resources allowed
`resources.requests`	Requests describe the minimum amount of compute resources required
`resources.limits.cpu`	CPU limits
`resources.limits.memory`	Memory limits
`resources.limits.ephemeral-storage`	Ephemeral Storage limits
`resources.requests.cpu`	CPU requests
`resources.requests.memory`	Memory requests
`resources.requests.ephemeral-storage`	Ephemeral Storage requests
`securityContext`	Security options the container should run with.
`securityContext.allowPrivilegeEscalation`	`AllowPrivilegeEscalation` controls whether a process can gain more privileges than its parent process. This directly controls if the `no_new_privs` flag will be set on the container process
`secuirtyContext.readOnlyRootFilesystem`	Whether this container has a read-only root `filesystem`. Default is `false`.
`securityContext.capabilities`	The capabilities to add or drop when running containers. Defaults to the default set of capabilities granted by the container runtime.
`securityContext.capabilities.drop`	Removed capabilities
`secuirtyContext.capabilities.add`	Added capabilities
`securityContext.runAsUser`	The UID to run the entry point of the container process. Debug Tool Container comes up with the default `<user-id>` as 7000. If the operator wants to override this default value, it can be done using the ``runAsUser`` field. Otherwise, the field can be skipped. Default value: uid=7000(debugtool) gid=7000(debugtool) groups=7000(debugtool)
`volumeMounts.mountPath`	Indicates the path for volume mount.
`volumeMounts.name`	Indicates the name of the directory for debug tool logs storage.