5.2 Stream Options
The following table describes the Stream options.
Table 5-3 Stream Options
| Attribute | Description |
|---|---|
| Primary IP and port |
Bootstrap Server IP and Port of Primary Kafka Cluster A list of host and port pairs to use for establishing the initial connection to the Kafka cluster. Single value of hostIp: port shall be configured. Since these servers are just used for the initial connection to discover the full cluster membership (which may change dynamically), this list need not contain the full set of servers (you may want more than one, though, in case a server is down). |
| Backup IP and port |
Bootstrap Server IP and Port of Backup Kafka Cluster. MDS will create Kafka connections using Backup IP and Port, if Primary IP and port is not reachable.
|
| Topic name | Topic is a fundamental unit for Kafka event or Kafka message. Kafka broker categorizes the messages which are received from DSR based on this Topic parameter only. |
| Client ID | An ID string to pass to the external Kafka Server when making requests. The purpose of this is to track the source of requests beyond just IP/port by allowing a logical application name to be included in server-side request logging. |
| Compression codec |
Compression codec to use for compressing message sets. |
|
Compression level |
Compression level parameter for algorithm selected by configuration property Compression codec. Higher values will result in better compression at the cost of more CPU usage. |
| Batch size | MDS measures batch size in total bytes instead of the number of messages. It controls how many bytes of data to collect before sending messages to the external Kafka broker. |
| Linger |
Delay in milliseconds to wait for messages in the producer queue to accumulate before constructing message batches (MessageSets) to transmit to Kafka brokers. Default value: 10 |
| Number of acknowledgments |
The number of acknowledgments the producer requires the leader to have received before considering a request complete. This controls the durability of records that are sent. Default value: 12 |
| Security protocol |
Protocol used to communicate with external Kafka brokers. Default value: SSL |
| Connection timeout |
Indicates how much time (in milliseconds) should wait after a connection to a Kafka cluster has been requested. If the timeout is over, then the connection will be damned unsuccessful. Default value: 1000
|
| SSL Protocol |
The SSL protocol used to generate the SSLContext. Default value: "TLSv1.3" |
| Record Size |
This parameter represents the maximum size of single record that can be sent over Kafka Broker. Default value: 2KB |
| Delivery Timeout |
An upper bound on the time to report success or failure after a call to send() returns. This limits the total time that a record will be delayed prior to sending, the time to await acknowledgement from the broker, and the time allowed for retriable send failures. Default value: 120000 Lapse of time: 30 seconds |
| Request Timeout |
This configuration controls the maximum amount of time the MDS will wait for the response of a Kafka request. If the response is not received before the timeout elapses the MDS will resend the request if necessary or fail the request if retries are exhausted. Default value: 30000 Lapse of time: 30 seconds |
| Retries |
The number of retries if MDS (Producer) does not get an acknowledgement from Kafka Broker. |
|
User should configure the below mentioned Additional Parameters if Security Protocol is configured as "SSL": |
|
| SSL CA location |
Filename (along with path) of ca file to use in certificate verification |
| SSL certificate location |
Filename (along with path) of file in pem format containing the client certificate as well as any ca certificates needed to establish the certificate's authenticity. |
| SSL key location |
Filename (along with path ) containing the client private key. |
| SSL key password | Password to be used when loading the certificate chain |
|
User should configure the below mentioned Additional Parameters if Security Protocol is configured as "SASL_Plauntext": |
|
| SASL Mechanisms |
SASL mechanism to use for authentication. Types of SASL Mechanisms
|
| SASL Kerberos Service Name | Kerberos principal name that Kafka runs as, not including /hostname@REALM |
| SASL Kerberos Principal | This client's Kerberos principal name. (Not supported on Windows, will use the logon user's principal). |
| SASL Kerberos Kinit Cmd | Shell command to refresh or acquire the client's Kerberos ticket. This command is executed on client creation and every sasl.kerberos.min.time.before.relogin (0=disable). |
| SASL Kerberos Keytab | Path to Kerberos keytab file. |
| SASL Kerberos Min Time Before Relogin | Minimum time in milliseconds between key refresh attempts. Disable automatic key refresh by setting this property to 0. |
| SASL Username | SASL username for use with the PLAIN and SASL-SCRAM-.. mechanisms |
| SASL Password | SASL password for use with the PLAIN and SASL-SCRAM-.. mechanism |
Table 5-4 DSR Properties
| Fields | Value | Description |
|---|---|---|
| Interval size | Default: 5 minutes | Indicates the time it should wait to read and send the metrics.
Interval Size should be lesser than the retention time.
Range: 5 minutes, 15 minutes, 30 minutes. |
| Maximum retention time | Default: 30 minutes | It is the maximum amount of time to store metrics that could not
be sent to the Kafka cluster.
Range: No Retention, 15 minutes, 30 minutes, 1 hour. |
| Output format | Default: JSON | Output format of Kafka's messages.
Note: This is a read-only attribute. |
Note:
For more information about Stream Options, see Operations, Administration, and Maintainance (OAM) Guide.