1. Feature Guide
  2. DSR Features and Functions
  3. Security Assertion Markup Language (SAML)
  4. Measurement Data Streaming (MDS)

2.18.9 Measurement Data Streaming (MDS)

Measurements Data streaming solution provides DSR's statistics to remote server communicating through Apache Kafka severs in the network. Using Kafka producer, MDS feature sends selected DSR measurement statistics to Kafka server. Measurements collected for all the DSR components (A, B, C levels) at NOAM and SOAM are sent to remote Kafka server in JSON format. Hence, avoiding users to enter directly into the DSR system and interpret measurements. Statistics, which are pushed over to Kafka server can be used for DSR system analysis.

Figure 2-72 DSR Mediation with MDS


DSR Mediation with MDS

The MDS collects user-enabled measurement statistics data and sends the collected data to a user-configured Kafka cluster at user configured intervals. Additionally, MDS stores any unsent data on disk in the event of a connection loss with the Kafka cluster. It attempts to resend this unsent data once the connection to the Kafka cluster is reestablished, provided it occurs before the expiration of the user-configured retention period. Kafka connection properties and DSR properties can be configured at NOAM, but measurements can be enabled at both NOAM and SOAM.

Selection of measurements

There are many measurements for DSR nodes, together, which produce a large quantity of data. Since it takes more time to generate this data and high CPU power to process it, MDS feature provides an option to stream only the required measurements by enabling them. Using NOAM GUI, the user can enable measurements whose scope is A(A level server). Using SOAM GUI, the user can enable measurements whose scope is DSR component B(B level server).

Interval

This property defines the regular interval at which MDS collects, parse, and sends the statistics data of measurements to the remote Kafka server. The user can configure this property from NOAM GUI. If the transfer of measurements data to the Kafka cluster is not completed within the user configured interval, an event will be triggered.

The different Interval Options are:

Table 2-13 Interval

Interval Default
5 minutes Yes
15 minutes -
30 minutes -

Retention

MDS feature ensures that there is no data loss when there is a disconnection or unable to establish a connection to both primary and backup Kafka clusters. It stores the unsent data into the MdsPendingRecordsA or MdsPendingRecordsB tables.

MDS sends this unsent data to Kafka server if connection to Kafka reestablishes before the expiry of retention period. MDS deletes the old unsent data from MdsPendingRecordsA or MdsPendingRecordsB after the expiry of configured retention period. MDS provides the following options for maximum retention time:

Table 2-14 Retention

Data Table Name
Unsent data of A-level metrics MdsPendingRecordsA
Unsent data of B, C –level metrics MdsPendingRecordsB

Table 2-15 Retention time

Retention Time Default
No Retention -
5 minutes -
15 minutes -
30 minutes Yes
1 hour -

Note:

“No Retention” option disables data retention in case of send failure.

Security

MDS feature supports the users to send statistics data to remote Kafka server securely. It allows the users to configure any one of the following protocols:
  • Additional configurable properties required by the security protocols are listed in the MDS Configuration.
  • Kafka properties, refer to Diameter Signaling Router Operations, Administration and Maintenance Guide.

Data Size and Data Format

The dimension of measurements can be single or arrayed. Since the arrayed measurements can contain the data in large quantity, MDS provides "Max Records Size" option to configure the maximum size of the message. MDS divides the large data into segments based on the configured “Max Record Size” value and adds segment information to each segment.

Table 2-16 Segment Field

Info Description
“segmentInfo":''F0" Indicates the first segment.
“segmentInfo":"In" 'I' indicates the intermediate segment and 'n' is the segment number.
“segmentInfo":"Ln" 'L' indicates the last segment and 'n' is the segment number.

Note:

If message is not segmented it contains only one segment, then segmentInfo header will contain only L0 segmentInfo: "L0" indicates that the message contains only one segment.

The user can configure the “Batch Size” Kafka property, based on which the Kafka producer collects multiple messages together until the batch size is reached and sends it through the network as one batch.

Every full and split message will contain a header with the following characteristics:

Table 2-17 Header characteristics

Name Description
hostname Server name defined by the server.
node Contains a detailed server type, example:
  • MP_DA-MP
  • SYSTEM_OAM_SOAM
  • MP_IPFE
date Indicates when the measurement was taken.
interval Indicates the interval in which the measurements are collected.
metric Name of the measurement.
group Name of the group, which the measurement belongs.
type Indicates the measurement type, example:
  • Single
  • Average
  • Max
  • Min
dim Indicates the measurements dimension, example:
  • Single
  • Arrayed
segmentInfo Indicates the segment number.
Total It contains actual statistics data of measurement. This is only used when dim is single.
Sets It contains actual statistics data of measurement. This is only used when dim is arrayed and contains all elements and their values.

Example of Single dimensioned measurement record:

[{
"hostname":"MDS-NOAM",
"node":"NETWORK_OAMP",
"date":"04/03/2023 13:30 ",
"interval":"13:30:00-13:34:59",
"dim":"Single",
"metric":"System.Disk_UtilPct_Peak",
"group":"OAM.SYSTEM",
"type":"max",
"segmentInfo":"L0",
"TOTAL": “0”,
}]

Example of Arrayed measurement record without applying “Max Record Size” limit: 

 [{
   "hostname":"DSRTESTSO-DSO00",
   "node":"SYSTEM_OAM",
   "date":"04/03/2023 13:30",
   "interval":"13:30:00-13:34:59",
   "metric":"TxRequestSuccessAllConn",
   "group":"Diameter Egress Transaction",
   "type":"simple",
   "dim":“Arrayed",
   “segmentInfo":"L0",

"sets":
[
       {“id":"Connection_1", "TOTAL":"3”},
        {“id":"Connection_2", "TOTAL":"5”}
        ----------
        {“id":"Connection_20", "TOTAL":"6”},
        {“id":"Connection_21", "TOTAL":"2”}
        ----------
        {“id":"Connection_30", "TOTAL":"1”},
        {“id":"Connection_31", "TOTAL":"51”}
         ----------
          -----------
        {“id":"Connection_n", "TOTAL":"10”}
 
   ]
}]

Example of Arrayed measurement record after applying 2KB record size limit

The default value for the Max Record Size is 2KB for the parameter Record 1.

Record 1:

[{
   "hostname":"DSRTESTSO-DSO00",
   "node":"SYSTEM_OAM",
   "date":" 04/03/2023 13:30",
   "interval":"13:30:00-13:34:59",
   "metric":"TxRequestSuccessAllConn",
   "group":"Diameter Egress Transaction",
   "type":"simple",
   "dim":“Arrayed",
   “segmentInfo":"F0",
   "sets”: [
        {“id":"Connection_1", "TOTAL":"3”},
         {“id":"Connection_2", "TOTAL":"5”}
        ----------
        {“id":"Connection_20", "TOTAL":"6”},
   ]
      }]

Record 2:

[{
   "hostname":"DSRTESTSO-DSO00",
   "node":"SYSTEM_OAM",
    "date":"04/03/2023 13:30",
   "interval":"13:30:00-13:34:59",
   "metric":"TxRequestSuccessAllConn",
   "group":"Diameter Egress Transaction",
   "type":"simple",
   "dim":“Arrayed",
   “segmentInfo":"I1"
   "sets”: [
        {“id":"Connection_21", "TOTAL":"2”}
        ----------
        {“id":"Connection_30", "TOTAL":"1" },
   ]
 
}]

Record n:

[{
   "hostname":"DSRTESTSO-DSO00",
   "node":"SYSTEM_OAM",
   "date":"04/03/2023 13:30",
   "interval":"13:30:00-13:34:59",
   "metric":"TxRequestSuccessAllConn",
   "group":"Diameter Egress Transaction",
   "type":"simple",
   "dim":“Arrayed",
   “segmentInfo":"Ln",
    "sets”: [
         ----------
           ----------
        {“id":"Connection_n", "TOTAL":"10" }
   ]
 
}]