Kafka Consumer Application Scalability Guidelines

These are a few guidelines for Kafka consumer application scalability:

• Producer and consumer applications to subscribe to topics in Kafka from Siebel are implemented in AI sidecar. For higher throughput in event message processing and exchange, it is suggested that component cloning and AI sidecar replication are done. These in turn require careful considerations for scaling the number of consumers running inside AI sidecars.

• The total number of consumers for a given topic is equal to the total number of consumers running for the given topic in each AI side car and should be equal to the number of partitions of the given topic created inside Kafka. If the number of partitions is less than the total number of consumers for a given topic, some of the consumers will be unutilized.

• In a favorable situation, the total number of consumers for a given topic is equal to total of number of sessions created on components (for example, cloned telco OMs) which is going to process the consumed record. So, the number of component instances that should be running to handle the total number of consumers for a given topic should be decided depending on the number of sessions an instance of component can handle.