Using Partitions in Kafka with Siebel CRM Event Publication and Subscription

You can configure Siebel CRM Event Publication and Subscription to use Kafka partitions to improve scalability and enhance the speed of data exchange between Siebel CRM and other integrated applications through parallelism. When you use Kafka partitions (each partition is a sequential, ordered record of messages), the topic data is written to all the partitions specified for that topic. The messages within a topic are available in a number of partitions across multiple Kafka servers allowing:

• Multiple producer threads to write topic data to multiple Kafka partitions.
• Multiple consumer threads to read topic data from multiple partitions at the same time.

The AI Sidecars run the producer and consumer threads/applications inside them to integrate with Kafka. The number and specifications of producer and consumer threads running in the AI sidecars determine the latency and throughput of Siebel-Kafka message exchanges; hence their numbers should be planned carefully. For more information, refer the table Planning Number of Consumers and Partitions.

Consumer Group Configuration

When you configure consumers for a Kafka topic in Event Pub/Sub, you can either:

• Allow Kafka to manage partition assignment automatically: If you do not specify any partitions for a consumer, Kafka uses the topic's configured partitions and manages partition assignment automatically.
  In this configuration:

  • Consumers participate in a consumer group.
  • Kafka automatically assigns partitions to consumers in the group.
  • Each message is processed by only one consumer within the consumer group.
• Assign specific partitions to consumers. If you specify one or more partition numbers for a consumer, Kafka disables automatic partition assignment and consumer group coordination for that consumer.
  In this configuration:

  • The consumer reads directly from the specified partitions.
  • Kafka does not perform dynamic partition assignment.
  • Multiple consumers can read the same partition and receive the same messages.

Examples:

Assume a Kafka topic is configured with a single partition (0) and two Sidecar AI instances are configured with consumers that use the same consumer group ID:

• Scenario 1: The consumer configuration specifies - Partitions: "0". In this case:
  • Both consumers read directly from partition 0.
  • If a message is published to the topic, both consumers receive the message.
• Scenario 2: The consumer configuration does not specify any partitions. In this case:
  • Both consumers participate in the same consumer group.
  • Kafka assigns the partition to only one consumer.

  • If a message is published to the topic, only one consumer receives and processes the message.

Using partitions with Siebel CRM Event Publication and Subscription is optional. But if you chose to use partitions, you must configure appropriate parameters in the configuration files (aieventconfig.txt and eventpayloadconfig.txt) and also plan Kafka configuration accordingly. For more information, see Configuration Files for Payload Generation and Configuration Files for OM-AI Integration.