4 Benchmark Testing

This section describes the environment used for benchmarking Oracle Communications Cloud Native Core, Network Repository Function (NRF). Benchmarking is performed with the settings described in this section.

The default values or recommendations for any required software or resource are available from third-party vendors. The operators may choose different values.

4.1 Overview

To qualify the test run, the following elements are considered:

  • Pod restart
  • CPU and Memory utilization
  • Error rate
  • Ingress and egress traffic rate
  • Success rate
  • Message request and response processing time
  • Infrastructure resource requirements and utilization

4.2 NRF Features

The following table lists the features that are enabled during benchmarking.

Table 4-1 NRF Features

NRF Features Feature Status Notes
Subscriber Location Function ENABLED SLF is enabled for 3 NfTypes (AUSF, UDR, and UDM) in slfLookupConfig.
SLF Selection from Registered NF Profiles ENABLED Dynamic SLF feature is enabled.
Access Token Caching during SLF Query ENABLED Access Token Caching during SLF Query is enabled with 24 hour validity period based caching. For more information about the accessTokenCacheEnabled under slfOptions, see Oracle Communications Cloud Native Core, Network Repository Function REST Specification Guide.
Overload Control ENABLED For more information about the configuration, see Oracle Communications Cloud Native Core, Network Repository Function REST Specification Guide.
Preferred Locality Feature Set ENABLED Extended Preferred locality is configured with 2 locationTypes, 8 locationMappings, 56 preferredLocationDetails.
NRF Forwarding per NF Type/Service Type ENABLED for UDM Forwarding is enabled for discovery services. 1 host is present in nrfHostConfig and 1 rule is present in forwardingRulesConfig.
EmptyList in Discovery Response ENABLED EmptyList feature is enabled and discoveryValidityPeriodCfg under nfDiscoveryOptions is configured with 5 rules.
Access Token Request Authorization ENABLED AuthFeature is enabled with 8 rules in authRulesConfig under nfAccessTokenOptions.
Pod Protection Support for NRF Subscription Microservice ENABLED For more information about the configuration, see Oracle Communications Cloud Native Core, Network Repository Function REST Specification Guide.
Subscription Limit ENABLED Set the subscription limit to 1000. For more information about the configuration, see Oracle Communications Cloud Native Core, Network Repository Function REST Specification Guide.
User-Agent Header for Outgoing Requests ENABLED For more information about the configuration, see Oracle Communications Cloud Native Core, Network Repository Function REST Specification Guide.
NRF to Pick Port from ipEndpoints and FQDN from the NF Service or NF level when Selecting SLF ENABLED For more information about the configuration, seeOracle Communications Cloud Native Core, Network Repository Function REST Specification Guide.
ASM Sidecar ENABLED For more information about the configuration, see Oracle Communications Cloud Native Core, Network Repository Function REST Specification Guide.
Maximum Number of Profiles in Discovery Response ENABLED Set the value as 12 for the profilesCountInDiscoveryResponse parameter. For more information about the configuration, see Oracle Communications Cloud Native Core, Network Repository Function REST Specification Guide.

Note: If the value is greater than 12, there will be a performance impact.

NRF Georedundancy ENABLED For more information about the configuration, see Oracle Communications Cloud Native Core, Network Repository Function REST Specification Guide.
NRF Growth ENABLED For more information about the configuration, see Oracle Communications Cloud Native Core, Network Repository Function REST Specification Guide.
SLF Routing using SCP ENABLED For more information about the configuration, seeOracle Communications Cloud Native Core, Network Repository Function Rest Specification Guide.
Discovery Validity ENABLED Discovery validity feature is enabled with a value as 1 minute (for all NF Types).

For more information about the configuration, see Oracle Communications Cloud Native Core, Network Repository Function REST Specification Guide.

Access Token Expiry Time NA The expiry duration for a NRF signed certificate. The limit is 24 hours.

For more information about NRF Access Token Service Usage Details, see Oracle Communications Cloud Native Core, Network Repository Function User Guide.

Load Change Threshold for Notifications from NRF NA This feature is configured using nfNotifyLoadThreshold parameter. Set the value to 10%.

For more information about the configuration, see Oracle Communications Cloud Native Core, Network Repository Function REST Specification Guide.

Monitoring the Availability of SCP Using SCP Health APIs ENABLED For more information about the configuration, see Oracle Communications Cloud Native Core, Network Repository Function REST Specification Guide.
NRF to Retry Alternate Producer to Support Maximum Diversity NA

The value of maxSlfAttempts parameter is set to 1 and the value of useAlternateScp parameter is set to true.

For more information about the configuration, see Oracle Communications Cloud Native Core, Network Repository Function REST Specification Guide.

Error Response Enhancement NA For more information about the configuration, see Oracle Communications Cloud Native Core, Network Repository Function REST Specification Guide.
Error Log Enhancements ENABLED For more information about the configuration, see Oracle Communications Cloud Native Core, Network Repository Function REST Specification Guide.
NRF support for A_PFD DataSetId in registration and discovery (DataSetId Enhancements) ENABLED For more information about the configuration, see Oracle Communications Cloud Native Core, Network Repository Function REST Specification Guide.
NFService Priority Update ENABLED For more information about the configuration, see Oracle Communications Cloud Native Core, Network Repository Function REST Specification Guide.
Subscriber Identity in messages (3GPP-SBI-Correlation-Info) ENABLED For more information about the configuration, see Oracle Communications Cloud Native Core, Network Repository Function REST Specification Guide.
Ingress Gateway Pod Protection DISABLED For more information about the configuration, see Oracle Communications Cloud Native Core, Network Repository Function REST Specification Guide.

Note: The Ingress Gateway Pod Protection using Rate Limiting feature is recommended over the Ingress Gateway Pod Protection feature.

Egress Gateway Pod Throttling DISABLED For more information about the configuration, see Oracle Communications Cloud Native Core, Network Repository Function Installation, Upgrade and Fault Recovery Guide.

Note: The Egress Gateway Pod Protection using Rate Limiting feature is recommended over the Egress Gateway Pod Throttling feature.

NF Profile Size Limit ENABLED For more information about the configuration, see Oracle Communications Cloud Native Core, Network Repository Function REST Specification Guide.
Ingress Gateway Pod Protection Using Rate Limiting ENABLED For more information about the configuration, see Oracle Communications Cloud Native Core, Network Repository Function REST Specification Guide.
Egress Gateway Pod Protection Using Rate Limiting ENABLED For more information about the configuration, see Oracle Communications Cloud Native Core, Network Repository Function REST Specification Guide.

Note:

Apart from these features being enabled, rest of the feature configurations have been set to their default values. For more information about the feature configurations, see Oracle Communications Cloud Native Core, Network Repository Function REST Specification Guide. For more information about the features, see Oracle Communications Cloud Native Core, Network Repository Function User Guide.

4.3 Software Test Constraints

Table 4-2 Software Test Constraints

Test Constraint Details
NRF Version 25.1.202
Average HTTP Request Packet Size ~630 to 15000 bytes (Based on NfRegister Request)
Average HTTP Response Packet Size ~5500-131000 bytes (Discovery Response with nfProfiles)
Average NF Heartbeats Timer per NF 30s
Average size of NF Profiles ~630 to 15000 bytes

4.4 Service Operations

This section list the various service operations with the required query parameters.

NfDiscover Service Operation

The NfDiscover service operations are sent with the following query parameters:

  • Target-nf-type, guami, and requester-nssais
  • Target-nf-type, requester-nf-type, routing-indicator, and group-id-list
  • Target-nf-type, requester-nf-type, supi, and service-names
  • Target-nf-type, requester-nf-type, and service-names
  • Target-nf-type, requester-nf-type, and group-id-list
  • Target-nf-type and pgw
  • Target-nf-type, requester-nf-type, supi, and service-names
  • Target-nf-type, requester-nf-type, and service-names
  • Target-nf-type, dnn, snssai, TAI, and pgw-ind
  • Target-nf-type, SUPI, group-id-list, and service-names
  • Target-nf-type, dnn, snssai, preferred-locality, and service-names
  • Target-nf-type and guami
  • Target-nf-type, requester-nf-type, and preferred-locality
  • Target-nf-type, requester-nf-type, preferred-locality, and SUPI
  • Target-nf-type, requester-nf-type, SUPI, group-id-list, requester-nf-type, and data-set
  • Target-nf-type, requester-nf-type, routing-indicator, group-id-list, and service-names

4.5 CNE Common Services Resource Utilization

The following table describes CNE microservices and their utilization.

Table 4-3 CNE Common Services Resource Utilization

CNE Observability Services Pod Count
occne-prometheus-server-1 2
occne-opensearch-cluster 10
occne-prometheus-node-exporter 16

4.6 53K TPS per NRF Set/106K TPS for Two NRF Sets

The per set capacity of NRF is 53K TPS. Higher TPS rate can be achieved by deploying additional NRF sets. This section provides the details for 106K TPS.

4.6.1 NRF Services Resource Requirements

This section provides the NRF services resource requirement per NRF deployment.

Note:

Changes in the resource requirements are highlighted in bold.

Table 4-4 NRF Services Resource Requirements

Service Name Min Pod Replica Max Pod Replica Min CPU/Pod Max CPU/Pod Min Memory/Pod (in Gi) Max Memory/Pod (in Gi) Min (Mi) Ephemeral Storage Max (Gi) Ephemeral Storage
Helm test 1 1 1 2 1 2 78.1 1
<helm-release-name>-nfregistration 2 2 4 4 3 3 78.1 1
<helm-release-name>-nfdiscovery 2 90 8 8 5 5 78.1 2
<helm-release-name>-nfsubscription 2 2 2 2 3 3 78.1 1
<helm-release-name>-nrfauditor 2 2 2 2 3 3 78.1 1
<helm-release-name>-nrfconfiguration 1 1 2 2 2 2 78.1 1
<helm-release-name>-nfaccesstoken 2 2 2 2 2 2 78.1 1
<helm-release-name>-nrfartisan 1 1 2 2 2 2 78.1 1
<helm-release-name>-nrfcachedata 2 2 4 4 4 4 78.1 1
<helm-release-name>-ingressgateway 2 34 4 4 5 5 78.1 1
<helm-release-name>-egressgateway 2 30 4 4 4 4 78.1 1
<helm-release-name>-alternate-route 2 2 2 2 4 4 78.1 1
<helm-release-name>-appinfo 2 2 1 1 1 1 78.1 1
<helm-release-name>-perfinfo 2 2 1 1 1 1 78.1 1

Table 4-5 Service Mesh Sidecar Resource Requirements

Service Name CPU/Pod Memory/Pod (in G) Concurrency
Min Max Min Max
Helm test 0 0 0 0 NA
<helm-release-name>-nfregistration 2 2 3 3 2
<helm-release-name>-nfdiscovery 2 2 3 3 4
<helm-release-name>-nfsubscription 2 2 3 3 2
<helm-release-name>-nrfauditor 2 2 3 3 2
<helm-release-name>-nrfconfiguration 2 2 3 3 2
<helm-release-name>-nfaccesstoken 2 2 3 3 2
<helm-release-name>-nrfartisan 2 2 3 3 2
<helm-release-name>-nrfcachedata 2 2 3 3 2
<helm-release-name>-ingressgateway 4 4 3 3 8
<helm-release-name>-egressgateway 4 4 3 3 8
<helm-release-name>-alternate-route 2 2 3 3 2
<helm-release-name>-appinfo 2 2 3 3 2
<helm-release-name>-perfinfo 2 2 3 3 2

4.6.2 cnDBTier Resource Requirements per NRF Deployment

This section provides the details of cnDBTier services resource requirements.

Note:

Changes in the resource requirements are highlighted in bold.
The following table describes the resources required by cnDBTier pods to perform NRF benchmark tests.

Table 4-6 cnDBTier Services Resource Requirements

Service Name Pod Replica # CPU/Pod Memory/Pod (in Gi) PVC Size (in Gi) Ephemeral Storage
Min Min Max Min Max PVC1 PVC2 Min (Mi) Max (Gi)
MGMT (ndbmgmd) 2 4 4 8 10 15 NA 90 1
DB (ndbmtd) 4 4 4 5 5 4 5 90 1
SQL (ndbmysqld) 2 4 4 20 20 13 NA 90 1
SQL (ndbappmysqld) 2 8 8 3 3 1 NA 90 1
Monitor Service (db-monitor-svc) 1 4 4 4 4 NA NA 90 1
Backup Manager Service (db-backup-manager-svc) 1 1 1 1 1 NA NA 90 1
Replication Service - Leader 1 2 2 12 12 4 NA 90 1
Replication Service - Other 0 0.6 1 1 2 NA NA 90 1

The following table describes the resources required by cnDBTier service mesh sidecar to perform NRF benchmark tests.

Table 4-7 Service Mesh Sidecar

Service Name CPU Memory (in Gi) Concurrency
Min Max Min Max
MGMT (ndbmgmd) 2 2 1 1 8
DB (ndbmtd) 2 2 1 1 8
SQL (ndbmysqld) 2 2 1 1 8
SQL (ndbappmysqld) 4 4 1 1 8
Monitor Service (db-monitor-svc) 2 2 1 1 2
Backup Manager Service (db-backup-manager-svc) 2 2 1 1 2
Replication Service-Leader 2 2 1 1 2
Replication Service-Other 2 2 1 1 2

The following default values are added for the service mesh specific annotations in the ocnrf_dbtier_CNDBTIER_VERSION_custom_values_NRF_VERSION.yaml file.

The following table describes the default values for resources required by cnDBTier service mesh sidecar to perform NRF benchmark tests.

Table 4-8 Default Values for Service Mesh Specific Annotations

Parameter Name Annotations
mgm.annotations 

sidecar.istio.io/inject: "true"
sidecar.istio.io/proxyCPU: "2"
sidecar.istio.io/proxyCPULimit: "2"
sidecar.istio.io/proxyMemory: "1Gi"
sidecar.istio.io/proxyMemoryLimit: "1Gi"
proxy.istio.io/config: '{concurrency: 8}'
traffic.sidecar.istio.io/excludeInboundPorts: "8081"
ndb.annotations 

sidecar.istio.io/inject: "true"
sidecar.istio.io/proxyCPU: "2"
sidecar.istio.io/proxyCPULimit: "2"
sidecar.istio.io/proxyMemory: "1Gi"
sidecar.istio.io/proxyMemoryLimit: "1Gi"
proxy.istio.io/config: '{concurrency: 8}'
traffic.sidecar.istio.io/excludeInboundPorts: "8081"
api.annotations 

sidecar.istio.io/inject: "true"
sidecar.istio.io/proxyCPU: "2"
sidecar.istio.io/proxyCPULimit: "2"
sidecar.istio.io/proxyMemory: "1Gi"
sidecar.istio.io/proxyMemoryLimit: "1Gi"
proxy.istio.io/config: '{concurrency: 8}'
traffic.sidecar.istio.io/excludeInboundPorts: "8081"
ndbapp.annotations 

sidecar.istio.io/inject: "true"
sidecar.istio.io/proxyCPU: "4"
sidecar.istio.io/proxyCPULimit: "4"
sidecar.istio.io/proxyMemory: "1Gi"
sidecar.istio.io/proxyMemoryLimit: "1Gi"
proxy.istio.io/config: '{concurrency: 8}'
traffic.sidecar.istio.io/excludeInboundPorts: "8081"
db-replication-svc.podAnnotations 

sidecar.istio.io/inject: "true"
sidecar.istio.io/proxyCPU: "2"
sidecar.istio.io/proxyCPULimit: "2"
sidecar.istio.io/proxyMemory: "1Gi"
sidecar.istio.io/proxyMemoryLimit: "1Gi"
proxy.istio.io/config: '{concurrency: 2}'

Note: The annotations for db-replication-svc.podAnnotations varies depending on the replication cardinality. podAnnotations should be added for each such replication deployment section in the ocnrf_dbtier_CNDBTIER_VERSION_custom_values_NRF_VERSION.yaml file.

db-replication-svc.podAnnotations traffic.sidecar.istio.io/excludeInboundPorts: "8081"

The inbound ports are added only to the db-replication-svc (leader pod).
db-monitor-svc.podAnnotations 

sidecar.istio.io/inject: "true"
sidecar.istio.io/proxyCPU: "2"
sidecar.istio.io/proxyCPULimit: "2"
sidecar.istio.io/proxyMemory: "1Gi"
sidecar.istio.io/proxyMemoryLimit: "1Gi"
proxy.istio.io/config: '{concurrency: 2}'
traffic.sidecar.istio.io/excludeInboundPorts: "8081"
db-backup-manager-svc.pod.annotations 

sidecar.istio.io/inject: "true"
sidecar.istio.io/proxyCPU: "2"
sidecar.istio.io/proxyCPULimit: "2"
sidecar.istio.io/proxyMemory: "1Gi"
sidecar.istio.io/proxyMemoryLimit: "1Gi"
proxy.istio.io/config: '{concurrency: 2}'

4.6.3 State Data Quantum at NRF

Table 4-9 STATE Data Quantum at NRF

Parameters Values Details
Number of Subscriptions 1000 (500 for NRF set 1 + 500 for NRF set 2)

Subscriptions are created with the below conditions:

  • Subscription to a set of NF Instances identified by their NF Type - 253
  • Subscription to a set of NF Instances identified by their NF Instance Id - 130
  • Subscription to a set of NF Instances identified by their NF Set Id - 78
  • Miscellaneous subscriptions - 39
  • NotificationCondition with monitored attributes set to “/priority” or “/load”.
  • reqNotifEvents set to NF_REGISTERED,NF_DEREGISTERED, NF_PROFILE_CHANGE
4.6.3.1 NF Count Per NF Types

The total number of NFs supported by per NRF set is 300. For two sets a total of 600 NFs are supported.

The following table lists the NF count per NF type along with the NF profile size.

Table 4-10 NF Count Per NF Types

Target-NF Types NF Count Profile Size (in bytes)
AMF 46 15000
AUSF 48 4640
BSF 3 1700
CBCF 3 11300
CHF 3 1800
GMLC 3 0630
LMF 3 2400
NEF 3 11300
PCF 79 1500-2500
SCP 27 1200
SLF 3 2100
SMF 148 15000
SMSF 3 11300
UDM 48 3200
UDR 180 1700-2000

4.6.4 Call-Mix

This section describes the various type of service requests with the required traffic distribution and targeted TPS per NRF set.

Table 4-11 Call-Mix

S.No Type of Request Traffic Distribution (in %) Targeted TPS
A NF Register 0.0094 5
B NF Update 0.0810 43
C NF Heartbeat Not considered 155
D NF Subscribe + Update Subscription 0.0377 20
E NF Discover 97.24 51600
F NRF Forwarded Discovery (Incoming) 10% of Incoming discovery traffic 0
G % of traffic forwarded to remote PLMN (case Visited NRF) 0 Not included in this test.
H % of traffic received from remote PLMN (case Home NRF) 0 Not included in this test.
I SLF Query 88.4% of Discovery 45600
J NF Access Token for SLF Extra 1
K NF Profile Retrieval 2.280 1210
L NF List Retrieval 0.0358 19
M NF Status Notify 4.0612 2155
N NF Status UnSubscribe 0.0056 3
O NF DeRegister 0.0056 3
P NF Access Token for Consumers 0.0188 10
Q NRF Forwarding (Outgoing) 10% of discovery traffic 5160

Table 4-12 Total Traffic

Traffic Type Message Considered Targeted TPS
Total Incoming Traffic A + B + C + D + E + F + G + H + J + K + L + N + O + P 53069
Total Outgoing Traffic I +J+ M + Q 52916
4.6.4.1 Discovery TPS per NfType and Number of Profiles in Response

The following table indicates the discovery TPS per NfType per NRF set and the number of profiles sent in the response.

Table 4-13 Discovery TPS per NfType and Number of Profiles in Response

Target-Nf Types TPS (in %) Absolute Discovery TPS SLF Queries Posted Number of Profiles in Response
AMF 4.19 2162.4 0 4
AUSF 0.40 204 0 6
BSF 0.00075 0.4 0 3
CBCF 0.0000 0 0 3
CHF 0.02 9.6 0 3
GMLC 0.0000 0 0 4
LMF 0.00299 1.6 0 4
NEF 0.0001 0.0516 0 3
PCF 0.37 191.4 0 10
SLF 0.00075 0.4 0 3
SMF 8.03 4143.4 0 4
SMSF 0.0001 0.0516 0 3
UDM 85.85 44297 44297 6
UDR 1.14 590.4 590.4 12

4.6.5 Test Observations

This section following table provides the details of test duration used for the benchmarking test.

Table 4-14 Test Observations

Parameter Values
Test Duration 72 hours
Ingress incoming connections 337
Ingress outgoing connections 2.719K
Egress incoming connections 240
Egress outgoing connections 1.200K

4.6.6 NRF Resource Utilization

The following table describes NRF microservices and their utilization for 53K TPS per NRF set with 600 NFs in two sets.

Table 4-15 NRF Resource Utilization

NRF Microservices Number of Pods Average CPU Average Memory
nfregistration 2 41% 43.9%
nfdiscovery 90 59% 47.5%
nfsubscription 2 41.8% 32%
nrfauditor 2 1.34% 23.2%
nrfconfiguration 1 2.09% 33.9%
nfaccesstoken 2 5.76% 39.4%
nrfartisan 1 0.10% 24.6%
nrfcachedata 2 48.0% 39.7%
alternate-route 2 0.10% 17%
perf-info 2 8.65% 13.5%
appinfo 2 5.41% 26.3%
ingressgateway 34 63.0% 56.5%
egressgateway 30 62.0% 40.4%

Note:

If you enable Message Feed feature at Ingress Gateway and Egress Gateway, approximately 33% pod capacity is impacted. The features enabled for this benchmark testing does not utilize alternate-route service.

4.6.7 cnDBTier Services Resource Utilization

The following table provides observed values of cnDBTier services.

Table 4-16 cnDBTier Services Resource Utilization

Service Name Pod Count Average CPU Memory
SQL (ndbappmysqld) 2 17.8% 36%
DB (ndbmtd) 4 14.1% 87%
SQL (ndbmysqld) 2 1.7% 5%
MGMT (ndbmgmd) 2 0.3% 0.38%
Monitor Service (db-monitor-svc) 1 0.5% 11.3 %
Backup Manager Service (db-backup-manager-svc) 1 0.6% 9.5%

Table 4-17 cnDBTier Statistics

DBTier Statistics Value
Disk Write operations per second 2.3K
Disk Read operations per second 10.2K
Transaction rates on data nodes 3.7K

4.6.8 Latency Observations

This section provides information about the latency observations.

Table 4-18 Latency Observations

Latency Parameters 99th Percentile (ms) 95th Percentile (ms) 90th Percentile (ms) 50th Percentile (ms)
Average Turnaround time at Ingress Gateway, Discovery Processing, and SLF Egress Transaction 200 196 191 151
Average Turnaround time for Discovery Processing and SLF Egress Transaction 200 195 190 150
Average Turnaround time for SLF Egress Transaction (see Note) 200 195 190 150

Note:

Following simulators are used for performance measurement and their induced latency:
  • NF simulator to NRF: 50ms
  • NRF to forwarding simulator and forwarding simulator to NRF: 150ms
  • NRF to another remote set NRF: 150 ms
  • NRF to SLF: 150 ms

4.6.9 NRF Configuration

The configurations for the following features are specific for 53K TPS per NRF set. These features must be configured with the recommended values as mentioned in the Oracle Communications Cloud Native Core, Network Repository Function REST Specification Guide.

  • Egress Gateway Pod Protection Using Rate Limiting- For more information about the configurations, see the following sections in Oracle Communications Cloud Native Core, Network Repository Function REST Specification Guide.
    • "Pod Protection By Rate Limiting" section in Egress Gateway Configuration
    • "Congestion Level Configuration" section in Ingress Gateway Configuration
  • Ingress Gateway Pod Protection Using Rate Limiting - For more information about the configurations, see the following sections in Oracle Communications Cloud Native Core, Network Repository Function REST Specification Guide.
    • Pod Protection By Rate Limiting section in Ingress Gateway Configuration
    • Congestion Level Configuration section in Ingress Gateway Configuration
  • Subscription Pod Protection- For more information about the configuration, see the "Pod Protection Options" section in Oracle Communications Cloud Native Core, Network Repository Function REST Specification Guide.
  • Overload Control- For more information about the configurations, see the following sections in Oracle Communications Cloud Native Core, Network Repository Function REST Specification Guide.
    • Overload Level Threshold Configuration in Perf-Info
    • Discard Policy Configuration in Ingress Gateway Configuration
    • Policy Mapping Configuration in Ingress Gateway Configuration