3.1.2 Utilizing the Local Kafka Cluster

You can use the local Kafka cluster as a source or target in GGSA pipelines. The instance provides a range of Kafka utilities to read and write data from local Kafka. These utilities are in the README.txt file in the/u01/app/osa/utilities/kafka-utils folder. Below is the description of the scripts for quick reference:

• For Kafka-related help, type ./kafka.sh help.
• To list Kafka topics, run ./kafka-topics.sh.
• To listen to a Kafka topic, type ./kafka-listen.sh {topic-name}.

• To feed a csv file into Kafka, run

  cat {your-csv-file} | ./csv2json.sh | ./sampler.sh 1 1 | ./kafka.sh feed {your-topic-name}

• To feed a json file, run cat {your-json-file} | ./sampler.sh 1 1 | ./kafka.sh feed {your-topic-name}

• To loop through your csv file, run ./loop-csv.sh {your-csv-file} | ./csv2json.sh | ./sampler.sh 1 1 | ./kafka.sh feed {your-topic-name}

• The command sampler.sh {n} {t} reads {n} lines in each {t} seconds from STDIN and writes to STDOUT. Default values: n=1 t=1

• The command csv2json.sh reads the CSV from the STDIN and writes JSON to STDOUT. First line must be the CSV header.

• There are data files to simulate GoldenGate capture:

  cat ./gg_orders.json | ./sampler.sh | ./kafka.sh feed goldengate

• There are data files to simulate nested json structures:

  ./loop-file.sh ./complex.json | ./sampler.sh 1 1 | ./kafka.sh feed complex

• There are data files to simulate movement of buses:

  cat ./30-min-at-50-rps.json | ./sampler.sh 20 1 | kafka.sh feed buses

• In general to feed test data to your running kafka installation, run {your-shell-script} | kafka.sh feed {your-topic-name}

  Note:

  your-shell-script generates data in JSON format.

  Sample output:

  {"price":284.03,"symbol":"USDHUF=x","name":"USD/HUF"}

  {"price":316.51,"symbol":"EURHUF=x","name":"EUR/HUF"}

  {"price":0.8971,"symbol":"USDEUR=x","name":"USD/EUR"}