Setting up Mirror Maker

If you have two Oracle Event Hub Cloud Service — Dedicated clusters, you can setup mirror for the topic that is present in one cluster to another topic that is present in a different cluster.

Perform the following steps to create a mirror.

  1. Create a Oracle Event Hub Cloud Service topic in the cluster that you want to mirror. This is considered as the source.

  2. Create another Oracle Event Hub Cloud Service topic in another cluster. This is the destination in which the source topic will be mirrored.

  3. The destination topic must have the same name as the source topic and must be appended with some suffix.

    The general format for the destination topic name should be {sourcetopicname}{anysuffix}. For Example, If the source topic name is ‘topic1’ , then the destination topic name should be ‘topic1replica’.

    Note:

    We cannot have two topics with the same name within a PSM tenancy. So we are including a suffix to the destination topic name.

    Note:

    You may want to create the destination topic with similar configuration to the source topic. But this is not mandatory.
  4. Create an access rule that allows specific/public traffic to connect to the destination cluster. See Oracle Event Hub Cloud Service — Dedicated: Access Rules Page. Without this access rule, the destination cluster cannot connect to the source cluster.

    Provide the below details for creating the access rule:
    • Rule Name: mirrormakerpublicaccess

    • Source: PUBLIC-INTERNET

    • Destination: kafka_KAFKA_ZK_SERVER

    • Destination Ports: 6667

    • Protocol: TCP

  5. SSH into the destination cluster. See Connecting to a Cluster Node Through Secure Shell (SSH). Create a folder to store the consumer and producer config files. Example: /u01/oehpcs/confluent/etc/mirror-maker

  6. Create the consumer configuration file and name it as sourceClusterConsumer.config. You can provide any name you require.

  7. Provide the following code in the sourceClusterConsumer.config file.

    bootstrap.servers=<public ip of the source cluster>:6667
    group.id=<consumer_group_id_for_this_mirrormaker .e.g. topic1-group>
    exclude.internal.topics=true
    auto.offset.reset=earliest
    partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor
  8. Create the producer configuration file and name it as targetClusterProducer.config. You can provide any name you require.

  9. Provide the following code in the targetClusterProducer.config file.

    bootstrap.servers=<public ip of the target cluster>:6667
    acks=-1
    max.in.flight.requests.per.connection=1
    compression.type=none
  10. Execute the following command in the destination cluster.

    nohup \
    /u01/oehpcs/confluent/bin/kafka-mirror-maker.sh \
    --consumer.config /u01/oehpcs/confluent/etc/mirror-maker/sourceClusterConsumer.config \
    --producer.config /u01/oehpcs/confluent/etc/mirror-maker/targetClusterProducer.config \
    --num.streams 2 \
    --whitelist ".*" \
    --message.handler kafka.tools.OehcsTopicSuffixMirrorMakerHandler
    --message.handler.args <destination topic suffix .e.g. replica>
    > log-file-name.log 2&1 &
  11. If you produce the message in the source topic on the source cluster, the message will be mirrored in the destination topic in the destination cluster.