Appendix B: Set Up Event Framework Metadata

To configure additional details you must set up the Event Framework Metadata. The sections in this topic provide detailed information on hot to set it up.

Topics:

·        Prerequisites

·        Set Up Topics

Prerequisites

The Kafka Server must be installed and running.

Set Up Topics

To store and publish data from a Producer writing to a Kafka Topic for Consumers to read the data in the Topic in the OFSAA System certain configurations are required for Topic Metadata details and creation of the Topic in the Kafka Server, which is available in the following sections.

Create a Topic in the Kafka Server

The instruction in this section is an example for reference on how to create Topics in the Kafka Server.

To create a Topic in the Kafka Server, follow these steps:

a.       Open the kafka-topics.sh file in the $FIC_HOME/ficdb/bin directory.

b.      Execute the kafka-topics.sh file in the format shown in the following:

./kafka-topics.sh --create --zookeeper <host of zookeeper:<port in zookeeper.properties> --replication-factor 1 --partitions <more than given while creating topic in PMF> --topic <topic_name>

For example,

./kafka-topics.sh --create --zookeeper 192.0.2.1:7780 --replication-factor 1 --partitions 4 --topic ECM_TF_SWIFT

Add Topic Metadata Details

To add the details for Topic Metadata, configure the arguments and Run the script in the following format which adds data to the relevant table in the AAI Config Schema:

./saveTopic.sh <TOPIC_NAME> <BOOTSTRAP_SERVERS> <TOTAL_PARTITIONS>

Table:  Argument Description Table for Topic Metadata

For example:

./saveTopic.sh ECM_TF_SWIFT 192.0.2.1:7780 4

Activate Consumer Group

The script to activate a Consumer Group is used to activate deactivated Consumer Groups. The Consumers are initialized and instantiated at Server startup. Only Consumer Groups that are active are instantiated at Server startup. The activation sets the Active Flag to ‘Y’ for a given Consumer.

The Consumer ID and Topic ID have to be passed to the Shell Script which requires activation. Configure the arguments and run the following script in the Event tables in the AAI Config Schema:

./activateconsumergroup.sh <N_CONSUMER_ID> <N_TOPIC_ID>

Table: Argument Description Table for Activate Consumer Group

For example:

./activateconsumergroup.sh 1612 SWIFT_TF_01

Deactivate Consumer Group

The script to deactivate a Consumer Group is used to deactivate a Consumer Group that is active. The Consumers are initialized and instantiated at Server startup. Only Consumer Groups that are active are instantiated at Server startup. The activation sets the Active Flag to ‘N’ for a given Consumer. The Consumer Groups that are deactivated are not be instantiated.

The Consumer ID and Topic ID have to be passed to the Shell Script which requires deactivation. 

Configure the arguments and Run the following script in the Event tables in the AAI Config Schema:

./deactivateconsumergroup.sh <N_CONSUMER_ID> <N_TOPIC_ID>

Table: Argument Description Table for Deactivate Consumer Group

For example:

./deactivateconsumergroup.sh 1612 SWIFT_TF_01

Decrease Consumers

This script is used to decrease the number of running Consumers for a given Consumer Group Name, Topic, and Application Node. It applies only till the current application instance is running.

Configure the arguments and run the following script in the Event tables in the AAI Config Schema:

./consumerdecrease.sh <TOPIC_NAME> <CONSUMERGRPNAME> <DECREASEBY> <HOSTNAME:PORT>

Table: Argument Description Table for Decrease Consumers

For example:

./consumerdecrease.sh ECM_TF_SWIFT ECM Users 15 192.0.2.1:7780

Increase Consumers

The script is used to increase the number of running Consumers for a given Consumer Group Name, Topic, and Application Node. It applies only till the current application instance is running.

Configure the arguments and run the following script in the Event tables in the AAI Config Schema:

./consumerincrease.sh <TOPIC_NAME> <CONSUMERGRPNAME> <INCREASEBY> <HOSTNAME:PORT>

Table: Argument Description Table for Increase Consumers

For example:

./consumerincrease.sh ECM_TF_SWIFT ECM Users 15 192.0.2.1:7780

Replay Messages

This script replays messages persisted to the Message Audit Tables to reprocess messages for failure cases or other reasons. The message is picked from the audit tables and processed instead of the Kafka Topic. The Event ID, which is available as a key in the Message Audit Tables, is used to replay the message. Any message that is read from the Kafka Topic is persisted to Message Audit Table with a Unique Event ID and processed later.

Configure the arguments and run the following script in the Event tables in the AAI Config Schema:

./replaymessage.sh <EVENT_ID>

Table: Argument Description Table for Replay Messages

For example:

./replaymessage.sh JOB1234

Save Consumer Properties

This script adds or updates Consumer Properties for a given Consumer ID (Consumer Group). The Properties are used to fine-tune the functioning of the Kafka Consumers.

The Shell Script accepts Consumer ID, Kafka Consumer Property Name, Value, and Datatype.

The properties have to be as given by the Kafka Consumer API. See the following link for a list of properties:

https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html   

Configure the arguments and run the following script in the Event tables in the AAI Config Schema:

./saveConsumerProperties.sh <CONSUMER_ID> <CONSUMER_PROP_NAME> <CONSUMER_PROP_VALUE> <CONSUMER_PROP_DATATYPE>

Table: Argument Description Table for Save Consumer Properties

For example:

./saveConsumerProperties.sh 1612 ECM_WR WRITE CHAR

Save Producer Properties

This script adds or updates Consumer Properties for a given Producer ID.  The properties are used to fine-tune the functioning of the Kafka Producer.

The Shell Script accepts Producer_ID, Kafka Producer Property Name, Value, and Datatype.

The properties have to as given by the Kafka Producer API. See the following link for a list of properties:

https://docs.confluent.io/platform/current/installation/configuration/producer-configs.html  

Configure the arguments and Run the following script in the Event tables in the AAI Config Schema:

./saveProducerProperties.sh <PRODUCER_ID> <PRODUCER_PROP_NAME> <PRODUCER_PROP_VALUE> <PRODUCER_PROP_DATATYPE>

Table: Argument Description Table for Save Producer Properties

For example:

./saveProducerProperties.sh PROD1234 ECM_SWIFT_WR  WRITE CHAR

Start Consumers

This Shell Script is used to start Consumers that are stopped for a given Application Node, Consumer Group Name, and Topic.

If you pass ALL in place of the consumergroupname argument, it results in the start of all Consumers in the given Application Node and Topic.

If the number of running Consumers is equal to the configured count in the Consumer Config Tables, no action is applied.

Configure the arguments and Run the following script in the Event tables in the AAI Config Schema:

./startconsumers.sh <TOPIC_NAME> <CONSUMERGRPNAME>/<ALL> <HOSTNAME:PORT>

Table: Argument Description Table for Start Consumers

For example:

./startconsumers.sh ECM_TF_SWIFT ECM Users/ALL 192.0.2.1:7780

Stop Consumers

This Shell Script is used to stop consumers for a given Application Node, Consumer ID (Consumer Group Name), and Topic. After the servers are restarted, the consumers are initialized and instantiated.

If you pass ALL in place of the consumergroupname argument, it results in the stop of all Consumers in the given Application Node and Topic.

Configure the arguments and Run the following script in the Event tables in the AAI Config Schema:

./stopconsumers.sh <TOPIC_NAME> <CONSUMERGRPNAME>/<ALL> <HOSTNAME:PORT>

Table: Argument Description Table for Stop Consumers

For example:

./stopconsumers.sh ECM_TF_SWIFT ECM Users 15 192.0.2.1:7780