Bulk Import Provisioning

Understanding Bulk Import Tool

With the help of Bulk Import tool, you can provision 5G UDR subscribers data in bulk. This tool:

Reads the subscriber data from a text file in the CSV format.
Must be run manually as it reads the CSV file (pre-determined format) from a specified directory path present on the pod.
Sends requests to the UDR for provisioning the subscribers.
Supports PCF subscribers data and NRF(SLF) subscribers data.
Supports different type of operations: CREATE, UPDATE, DELETE.

Note:

The Bulk Import Tool uses PVC (Persistent Volume Claim) and PV (Persistent Volume) that you must create before deploying the tool. For more details on creating PVC and PV, refer to UDR installation document.
The user/operator needs to manually copy all the CSV files to the PV mountpath inside the container, which is /home/udruser/import/.

Viewing CSV File Format

The CSV file format is as follows:

Key Type	Key value	OpType	Type	Subs_data(JSON)
MSISDN	1234	CREATE	PCF	{ .. }
IMSI	6786677	MODIFY	PCF	{ .. }
NAI	abcd@oracle.com	DELETE	PCF	{ .. }

Note:

The subs_data column must have JSON body enclosed in double quotes(").

To import CSV, you can use the following coding:

keyType,keyValue,operationType,nfType,jsonPayload
msisdn,1111111114,CREATE,PCF,
"{
  "profile-data": {
    "MSISDN": [
      "9111111112",
      "9211111112",
      "1111111114"
    ]
  },
  "policy-data": {
    "am-data": {
      "praInfos": {
        "p1": {
          "praId": "p1",
          "trackingAreaList": [
            {
              "plmnId": {
                "mcc": "976",
                "mnc": "32"
              },
              "tac": "5CB6"
            },
            {
              "plmnId": {
                "mcc": "977",
                "mnc": "33"
              },
              "tac": "5CB7"
            }
          ],
          "ecgiList": [
            {
              "plmnId": {
                "mcc": "976",
                "mnc": "32"
              },
              "eutraCellId": "92FFdBE"
            },
            {
              "plmnId": {
                "mcc": "977",
                "mnc": "33"
              },
              "eutraCellId": "8F868C4"
            }
          ],
          "ncgiList": [
            {
              "plmnId": {
                "mcc": "976",
                "mnc": "32"
              },
              "nrCellId": "b2fB6fE9D"
            },
            {
              "plmnId": {
                "mcc": "977",
                "mnc": "33"
              },
              "nrCellId": "5d1B4127b"
            }
          ],
          "globalRanNodeIdList": [
            {
              "plmnId": {
                "mcc": "965",
                "mnc": "235"
              },
              "n3IwfId": "fFf0f2AFbFa16CEfE7"
            },
            {
              "plmnId": {
                "mcc": "967",
                "mnc": "238"
              },
              "gNbId": {
                "bitLength": 25,
                "gNBValue": "1A8F1D"
              }
            }
          ]
        }
      },
      "subscCats": [
        "cat1",
        "cat2"
      ]
    }
  }
}"
 
accountID,12345678912345678912345678,MODIFY,SLF,"{
    "profile-data": {
        "imsi": [
            "2222222221",
            "2222222222"
        ],
        "nai": [
            "test@vzw.com"
        ],
        "accountID": [
            "12345678912345678912345678"
        ],
        "msisdn": [
            "19195225555",
            "19195225556"
        ],
        "extid": [
            "user@vzw.com"
        ]
    },
    "slfGroupName": "IMSGrp1"
}"

Features Supported

The Bulk Import tool supports following operation types:

CREATE: Allows to create subscribers with all keys mentioned in the payload. The PCF create and SLF create request payloads are available in the REST Guide.
UPDATE: Allows to update the existing subscribers with new data mentioned in the subs_data column.

Note:
You should provide complete subscriber information while using this operation as it replaces the old information with new data.
DELETE: Allows to delete complete subscriber information with all keys attached to the subscriber.

The Bulk Import tool supports following NF Types:

PCF: Using this tool, you can provision all 5G PCF data in bulk for all resource types: AM data, SM Data and UEPolicySet. You can refer to: Operations supported for PCF data and Management URI#Provisioningsystemasconsumer
SLF: You can create SLF Subscribers with all keys and SLFGroupName in the payload. For more details, you can refer to: SLF lookup support on UDR#SubscriberProvisioningAPIsonUDR(SLF)

Note:

The Bulk Import tool does not support SLFGroupName provisioning. You can provision this using REST APIs or CNC-Console GUI.

Installing Bulk Import Tool

You can implement the Bulk Import tool as a kubernetes job that comes up as a pod when you deploy it using a template yaml file. Before installing the Bulk Import tool, you must create a PVC.

Creating Persistent Volume Claim (PVC)

To create a PVC:

Create a namespace and then create a PVC using below yaml file:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: bulkimportpersistentclaim
spec:
  storageClassName: #<Please Provide your StorageClass  Name>
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

Note:

You must name the PVC as 'bulkimportpersistentclaim'.

After creating a yaml file, execute the following command to create a PVC under the namespace:
kubectl create -f <yaml file for PVC formed using the above code block> -n <namespace>

Example: kubectl create -f bulk_pvc.yaml -n bulktest
Execute the following command to verify whether PVC is created or not:
kubectl get pvc -n <namespace>

Figure 3-1 Verifying PVC Creation
Execute the following command to verify whether PV is created or not:
kubectl get pv

Figure 3-2 Verifying PV Creation

Installing Bulk Import Tool as a Job

To install Bulk Import tool as a job:

Use the below template yaml file to install Bulk Import tool as a job:

apiVersion: batch/v1
kind: Job
metadata:
  name: ocudr-nudr-bulk-import     # <Please use releaseName-nudr-bulk-import>
  namespace: ocudr                 # <Use the namespace created>
spec:
  backoffLimit: 0
  template:
    metadata:
      name: ocudr-nudr-bulk-import  # <Please use releaseName-nudr-bulk-import>
      annotations:
        "prometheus.io/port": "9000"
        "prometheus.io/path": "/actuator/prometheus"
        "prometheus.io/scrape": "true"
    spec:
      restartPolicy: Never
      securityContext:
        fsGroup: 1002
      volumes:
        - name: bulkimportpersistentvolume
          persistentVolumeClaim:
            claimName: bulkimportpersistentclaim
      containers:
      - env:
        - name: MYSQL_DATABASE
          valueFrom:
            secretKeyRef:
              name: ocudr-secrets
              key: dbname
        - name: DATASOURCE_USERNAME                           
          valueFrom:
            secretKeyRef:
              name: ocudr-secrets                            
# <Please use secrets created under UDR deployment >
              key: dsusername                               
 # < key for username mentioned in secrets yaml file under UDR deployment >
        - name: DATASOURCE_PASSWORD                        
          valueFrom:
            secretKeyRef:
              name: ocudr-secrets
              key: dspassword                               
 # < key for password mentioned in secrets yaml file under UDR deployment >
        - name: DB_SERVICE_NAME
          value: "mysql-connectivity-service.occne-infra"   
 # <Default mysql-connectivity service used under occne-infra namespace >
        - name: DB_SERVICE_PORT
          value: "3306"                                     
 # <Default port for mysql-connectivity service >
        - name: HIKARI_POOL_SIZE
          value: "10"                  
        - name: LOGGING_LEVEL_ROOT
          value: "INFO"
        - name: MANAGEMENT_SERVER_PORT
          value: "9000"                                       
        - name: DATA_SERVICE_BASEURL
          value: "http://ocudr-ingressgateway.ocudr:80"       
 # <Please use servicename.namespace:portNumber of ingressgateway>                       
        - name: POLLING_TIME_INTERVAL                          
# <Time interval in milliseconds after which the csv file is polled >
          value: "5000"
        - name: HTTP_RETRY_CONFIGURE                           
# <Number of retries made to dr-service for failed request>
          value: "2"     
        - name: APPLICATION_NAME
          value: "ocudr-nudr-bulk-import" 
        - name: ENGINEERING_VERSION
          value: "1.8.0" 
        - name: MARKETING_VERSION
          value: "1.8.0.0.0"
        - name: MICROSERVICE_NAME
          value: "ocudr-nudr-bulk-import"
        - name: K8S_CLUSTER_NAME
          value: "ocudr"
        - name: K8S_NAMESPACE
          value: "ocudr" 
        - name: K8S_NODE
          value: "5g-udr-dev-1-k8s-node-2"                                
        name: ocudr-nudr-bulk-import
        image: "ocudr-registry.us.oracle.com:5000/ocudr/nudr_bulk_import:1.8.0"  
# <Use the dockerregistry path for image:image tag>
        imagePullPolicy: Always
        volumeMounts:
          - name: bulkimportpersistentvolume
            mountPath: /home/udruser/import                   
# <Path where csv files needs to be placed for polling>
        ports:
        - containerPort: 9000
          name: management
        resources:
          requests:
            cpu: "4"
            memory: "6Gi"
          limits:
            cpu: "4"
            memory: "6Gi"

Execute the following command to install Bulk Import tool as a job:
kubectl create -f <template yaml >-n <namespace>

Example: kubectl create -f bulk_import_tool.yaml -n bulktest

Output: Container comes up as shown below:

Figure 3-3 Bulk Import Install Output
Execute the following command to check the pod details:
kubectl describe pods <nudr-bulk-import pod name> -n <namespace>

Example: kubectl describe pods bulktest-nudr-bulk-import-pgh2p -n bulktest
After the pod is up and running, copy the csv file. The Bulk Import tool starts processing the records.

Using Bulk Import Tool

By default, the Bulk Import tool polls the /home/udruser/import path every 5000 ms. You can use the Bulk Import tool when:

The nudr-bulk-import pod is up and running.
Execute the following command to place the file inside the container:
kubectl cp <csv file> <namespace>/<nudr-bulk-import pod name>:/home/udruser/import

kubectl cp create_pcf.csv bulktest/bulktest-nudr-bulk-import-pgh2p:/home/udruser/import
Login to the bash of container and execute the following command to check whether file is present inside the container or not:
kubectl exec -it <nudrbulk-import pod name>-n <namespace> -- bash

Example: kubectl exec -it bulktest-nudr-bulk-import-pgh2p -n bulktest -- bash

Note:
After login to bash, you can find the file under /home/udruser/import location.
```
bash-4.2$ pwd
  /home/udruser/import
bash-4.2$ ls -l
   -rw-r--r--. 1 root udruser 28471 Aug 19 09:33 create_pcf.csv
```
The Bulk Import tool processes each CSV file at a time and provisions the subscribers on 5G UDR. It's summary report is available in the /home/udruser/import path with filename (<CSV_filename>.log). You can monitor its metrics on Prometheus/Grafana.
Sample: create_pcf.log
```
bash-4.2$ pwd
  /home/udruser/import
bash-4.2$ ls -l
   -rw-r--r--. 1 root udruser 28471 Aug 19 09:33 create_pcf.csv  create_pcf.log
```
The Bulk Import container logs contains the summary report, when the INFO logging level is enabled.
Execute the following command to copy the report from pod container:
kubectl cp <pod name>:home/udruser/import/modify_slf.log <file name> -n <namespace>

Example: kubectl cp ocudr-nudr-bulk-import-8g8c8:home/udruser/import/modify_slf.log modify_slf.log -n myudr