Kafka Cluster Management Procedures

6.1 Creating Topics for OCNADD

Create topics (MAIN, SCP, SEPP, PCF, BSF, and NRF) using the configuration service before starting data ingestion.
For more details on topics and partitions, see the Oracle Communications Network Analytics Data Director Benchmarking Guide Section "OCNADD resource requirement". The guidelines on the replicationFactor and retentionMs are also available in the planning guide.
The segmentMs property determines the maximum time a Kafka log segment can remain active before being rolled over, regardless of its size. This ensures that old data is periodically cleaned up or compacted by the retention mechanism. For low-traffic topics, a smaller segmentMs value is recommended to facilitate timely data retention. In contrast, for high-traffic topics, the retentionMs property takes precedence, making segmentMs less impactful.

To create a topic, invoke the API endpoint described below:

For Relay Agent Kafka Cluster: <MgmtGWIP:MgmtGW Port>.<mgmt-ns>/ocnadd-configuration/v2/topic?relayAgentGroup=<relayAgentGroupName>
Where:

where <relayAgentGroupName> = <siteName>:<workerGroupName>:<relayAgentNamespace>:<relayAgentClusterName>
For Mediation Kafka Cluster: <MgmtGWIP:MgmtGW Port>.<mgmt-ns>/ocnadd-configuration/v2/topic?mediationGroup=<mediationGroup>
Where:

where <mediationGroup> = <siteName>:<workerGroupName>:<mediationNamespace>:<mediationClusterName>

Example Usage:

If the Management Gateway has a load balancer IP, use the Load Balancer IP in the commands; otherwise, use the Management Gateway service name. The examples below use the Management Gateway service name.

Exec into management gateway pod and execute the below command:

1) To create SCP topic in Relay Agent Kafka cluster:

curl -k --location "http://ocnaddmanagementgateway.ddmgmt:12889/ocnadd-configuration/v2/topic?relayAgentGroup=BLR:ddworker1:dd-relay-ns:dd-relay-cluster" \
-H "Content-Type: application/json" \
-d '{"topicName":"<topicname>","partitions":3,"replicationFactor":2,"retentionMs":300000,"segmentMs":300000}'

If secure communication for OCNADD is enabled (mTLS: true), use:

curl -k --location --cert-type P12 --cert /var/securityfiles/keystore/serverKeyStore.p12:$OCNADD_SERVER_KS_PASSWORD \
"https://ocnaddmanagementgateway.ddmgmt:12889/ocnadd-configuration/v2/topic?relayAgentGroup=BLR:ddworker1:dd-relay-ns:dd-relay-cluster" \
-H "Content-Type: application/json" \
-d '{"topicName":"<topicname>","partitions":3,"replicationFactor":2,"retentionMs":300000,"segmentMs":300000}'

2) To create MAIN topic in Mediation Kafka cluster:

curl -k --location "http://ocnaddmanagementgateway.ddmgmt:12889/ocnadd-configuration/v2/topic?mediationGroup=BLR:ddworker1:dd-mediation-ns:dd-mediation-cluster" \
-H "Content-Type: application/json" \
-d '{"topicName":"<topicname>","partitions":3,"replicationFactor":2,"retentionMs":300000,"segmentMs":300000}'

If secure communication for OCNADD is enabled (mTLS: true), use:

curl -k --location --cert-type P12 --cert /var/securityfiles/keystore/serverKeyStore.p12:$OCNADD_SERVER_KS_PASSWORD \
"https://ocnaddmanagementgateway.ddmgmt:12889/ocnadd-configuration/v2/topic?mediationGroup=BLR:ddworker1:dd-mediation-ns:dd-mediation-cluster" \
-H "Content-Type: application/json" \
-d '{"topicName":"<topicname>","partitions":3,"replicationFactor":2,"retentionMs":300000,"segmentMs":300000}'

JSON for creating the topic:

{
    "topicName": "<topicname>",
    "partitions": 3,
    "replicationFactor": 2,
    "retentionMs": 300000,
    "segmentMs": 300000
}

6.2 Kafka Cluster Capacity Expansion

The Kafka cluster capacity expansion may be needed in case the traffic throughput requirement for the customer deployment has changed significantly. The standard approach is to first scale the brokers, then increase the topic partitions, and finally execute a single, comprehensive partition reassignment to distribute all partitions (existing and new) evenly across the larger cluster.

The capacity expansion can be carried out in the following three-step process:

Adding more brokers to the existing Kafka Cluster
Increasing the partitions in the existing topics, for example, SCP, SEPP, MAIN
Partition reassignment for the topics

The procedures should be performed in the described sequence as needed in the Relay Agent or Mediation group's Kafka cluster.

Wait for each action to finish successfully before proceeding.
Monitor logs for errors, especially during reassignment and partition increase steps.
Adjust hostnames, broker lists, and file paths as needed for your environment.

6.3 Adding Kafka Brokers to Existing Kafka Cluster

The Kafka brokers in the cluster should be increased based on the benchmarking guide recommendations for the desired throughput.

Refer to the Oracle Communications Network Analytics Benchmarking Guide for the Kafka broker resource profile, number of brokers, and number of partitions for the Source and MAIN topic information.

Perform the following steps to increase the brokers in the existing Kafka Cluster in Relay Agent or Mediation group:

Update the custom values for the Relay Agent or Mediation group to increase the Kafka broker replicas:

# Update the helm chart to increase the number of Kafka brokers say 11

For Relay Agent: Update the ocnadd-relayagent-custom-values.yaml for the current release as indicated below
global.ocnaddrelayagent.kafka.kafkareplicas: 4 ======================> 11 increase this value to the desired value

For Mediation group: Update the ocnadd-mediation-custom-values.yaml for the current release as indicated below
global.ocnaddmediation.kafka.kafkareplicas: 4 ======================> 11 increase this value to the desired value

# Save the custom values in the corresponding helm charts

Scale the Kafka brokers: Scale the Kafka StatefulSet to the required replica count (for example, 11).

Scale the Kafka StatefulSet to the required replica count say 11

For Relay Agent:
# kubectl scale sts -n dd-relay kafka-broker --replicas 11

For Mediation group:
# kubectl scale sts -n dd-med kafka-broker --replicas 11

Verify that new Kafka Brokers are spawned:

Verify the Kafka broker pods in the corresponding group.

For Relay Agent: Check the Kafka brokers in the below command, match the count of Kafka brokers pods with the replicas configured. The brokers should be in the running state
# kubectl get po -n dd-relay 

For Mediation group: Check the Kafka brokers in the below command, match the count of Kafka brokers pods with the replicas configured. The brokers should be in the running state
# kubectl get po -n dd-med

Continue to the next procedure to increase the partitions for the topics.

6.4 Adding Partitions to an Existing Topic

The topics in OCNADD should already be created with the necessary partitions for the supported MPS during deployment. In case of a traffic load increase beyond the current supported MPS, it may be necessary to increase the number of partitions in existing topics. The procedure below can be executed to increase the number of partitions in the corresponding topic.

Caution:

The number of partitions cannot be decreased using this procedure. If partition reduction is required, the entire topic must be recreated with the desired number of partitions, which will result in complete data loss of the concerned topic.

Steps to add partitions to an existing topic:

Login to the bastion host and enter the OCNADD Management Gateway service pod in the deployed OCNADD management group namespace:
```
kubectl exec -it <mgmtgw-pod-name> -n <namespace> -- bash
```
Example:
```
kubectl exec -ti -n dd-mgmt ocnaddmanagementgateway-xxxxxx -- bash
```
Determine the OCNADD component (Relay Agent or Mediation) to which the topic belongs and where partitions need to be increased.

Describe the corresponding topic using the below command. Also provide the Relay Agent or Mediation group name in the command below:

For Relay Agent:

curl -k --location "http://ocnaddmanagementgateway.dd-mgmt:12889/ocnadd-configuration/v2/topic/SCP?relayAgentGroup=BLR:ddworker1:dd-relay-ns:dd-relay-cluster"

If secure communication (intraTlsEnabled: true and mTLS: true) is enabled:

curl -k --location --cert-type P12 --cert /var/securityfiles/keystore/serverKeyStore.p12:$OCNADD_SERVER_KS_PASSWORD \
"https://ocnaddmanagementgateway.dd-mgmt:12889/ocnadd-configuration/v2/topic/SCP?relayAgentGroup=BLR:ddworker1:dd-relay-ns:dd-relay-cluster"

For Mediation:

curl -k --location "http://ocnaddmanagementgateway.dd-mgmt:12889/ocnadd-configuration/v2/topic/MAIN?mediationGroup=BLR:ddworker1:dd-mediation-ns:dd-mediation-cluster"