4.8 Data Size and Data Format

Data of each measurement represents only one Kafka message, but some measurements that contains arrayed data might be larger in size. MDS provides an option, Max Record Size, in the GUI to configure the maximum size of message size Max Record Size. MDS segments the larger data into smaller parts based on the user configured Max Record Size value and it adds segment information to the splitted parts.

Based on the value of user configured Batch Size kafka property, Kafka producer joins multiple messages until that Batch size is reached and sends it through the network as one.

Each one of the full and sliced messages contains a header with the following characteristics:

Table 4-4 Format of Kafka Message

Name	Description
node	Contains a detailed server type, example: NETWORK_OAMP SYSTEM_OAM MP
hostname	It is the server name defined by client.
date	Indicates when was the measurement taken
interval	Indicates the interval from where the measurements are collected
metric	Name of the Measurement
group	Name of the group which the measurement belongs
type	Indicates the measurement’s type, such as: Single Average Max Min
dim	Indicates the measurement's dimension, such as: Single Arrayed
SegmentInfo	Indicates the segment number.
TOTAL	It contains actual statistics data of measurement.
Sets	It contains actual statistics data of measurement. This is only used when the dim is arrayed and contains all elements and their values.

Measurement Statistics data after converting into JSON format

Example of Single dimensioned measurement record

[{
   "hostname":"DSRTESTSO-DSO00",    
   "node":"SYSTEM_OAM",
   "date":"04/01/2022 20:29",
   "interval":"21:25:00-21:29:59",
   "metric":"System.RAM_UtilPct_Peak",
   "group":"OAM.SYSTEM",
   "type":"max",
   "dim":"single",
   "SegmentInfo":"L0",
   "TOTAL":"37"
      }]

Example of Arrayed measurement record without applying 2KB record size limit:

[{
   "hostname":"DSRTESTSO-DMP00",
   "node":"MP",
   "date":"04/01/2022 20:29",
   "interval":"21:25:00-21:29:59",
   "metric":"TxRequestSuccessAllConn",
   "group":"Diameter Egress Transaction",
   "type":"simple",
   "dim":"arrayed",
   "SegmentInfo":"L0",
   "sets":[
       { "id":"Connection_1", "TOTAL":"3" },
        { "id":"Connection_2", "TOTAL":"5" }
        ----------
        { "id":"Connection_20", "TOTAL":"6" },
        { "id":"Connection_21", "TOTAL":"2" }
        ----------
        { "id":"Connection_30", "TOTAL":"1" },
        { "id":"Connection_31", "TOTAL":"51" }
         ----------
          -----------
        { "id":"Connection_n", "TOTAL":"10" }
 
   ]
}]

Example of Arrayed measurement record after applying 2KB record size limit

Record 1

[{
   "hostname":"DSRTESTSO-DMP00",
   "node":"MP",
   "date":"04/01/2022 20:29",
   "interval":"21:25:00-21:29:59",
   "metric":"TxRequestSuccessAllConn",
   "group":"Diameter Egress Transaction",
   "type":"simple",
   "dim":"arrayed",
   "SegmentInfo":"F0",
   "sets":[
        { "id":"Connection_1", "TOTAL":"3" },
        { "id":"Connection_2", "TOTAL":"5" }
        ----------
        { "id":"Connection_20", "TOTAL":"6" },
   ]
   
      }]

Record 2

[{
   "hostname":"DSRTESTSO-DMP00",
   "node":"MP",
   "date":"04/01/2022 20:29",
   "interval":"21:25:00-21:29:59",
   "metric":"TxRequestSuccessAllConn",
   "group":"Diameter Egress Transaction",
   "type":"simple",
   "dim":"arrayed",
   "SegmentInfo":"I1"
   "sets":[
        { "id":"Connection_21", "TOTAL":"2" }
        ----------
        { "id":"Connection_30", "TOTAL":"1" },
   ]

}]

Record n

[{
   "hostname":"DSRTESTSO-DMP00",
   "node":"MP",
   "date":"04/01/2022 20:29",
   "interval":"21:25:00-21:29:59",
   "metric":"TxRequestSuccessAllConn",
   "group":"Diameter Egress Transaction",
   "type":"simple",
   "dim":"arrayed",
   "SegmentInfo":"Ln",
   "sets":[
         ----------
           ----------
        { "id":"Connection_n", "TOTAL":"10" }
   ]

}]