Using Partitions in Kafka with Siebel CRM Event Publication and Subscription

You can configure Siebel CRM Event Publication and Subscription to use Kafka partitions to improve scalability and enhance the speed of data exchange between Siebel CRM and other integrated applications through parallelism. When you use Kafka partitions (each partition is a sequential, ordered record of messages), the topic data is written to all the partitions specified for that topic. The messages within a topic are available in a number of partitions across multiple Kafka servers allowing:

• Multiple producer threads to write topic data to multiple Kafka partitions.
• Multiple consumer threads to read topic data from multiple partitions at the same time.

The AI Sidecars run the producer and consumer threads/applications inside them to integrate with Kafka. The number and specifications of producer and consumer threads running in the AI sidecars determine the latency and throughput of Siebel-Kafka message exchanges; hence their numbers should be planned carefully. For more information, refer the table Planning Number of Consumers and Partitions.

Using partitions with Siebel CRM Event Publication and Subscription is optional. But if you chose to use partitions, you must configure appropriate parameters in the configuration files (aieventconfig.txt and eventpayloadconfig.txt)and also plan Kafka configuration accordingly. For more information, see Configuration Files for Payload Generation and Configuration Files for OM-AI Integration.