Export Messages from Apache Kafka to Amazon S3
Use this recipe to export Apache Kafka messages to an Amazon Simple Storage Service (Amazon S3) bucket.
Note:
This recipe is available as Apache Kafka — Amazon S3 | Export Messages in the Integration Store. Oracle provides this recipe as a sample only. The recipe is meant only for guidance, and is not warranted to be error-free. No support is provided for this recipe.
Overview
This recipe exports messages from an Apache Kafka topic to an Amazon S3 bucket as JSON files. Each Apache Kafka message is exported as a JSON file that contains the contents of the message. The recipe uses the standard Apache Kafka Adapter and the standard REST Adapter.
To use the recipe, you must install the recipe package and configure the connections and other resources within the recipe package. Subsequently, you can activate and run the integration flow manually or specify an execution schedule for it. When triggered, the integration flow queries for messages in the specified Apache Kafka topic and loads predefined number of messages. Then the integration iterates over each message and exports them as JSON files into the specified Amazon S3 bucket.
System and Access Requirements
- Oracle Integration, Version 21.2.1 or higher
- Amazon Web Services (AWS) with Amazon S3
- Apache Kafka
- Confluent
- An account in AWS with administrator role
- An account in Confluent with administrator role
Recipe Schema
This section provides an architectural overview of the recipe.
Description of the illustration apachekafka-amazons3.png
When the integration flow of the recipe is triggered by an execution schedule or manual submission, it queries the Apache Kafka instance's topic for messages. If messages are present in the specified topic, the integration fetches a predefined number of messages. It iterates over the messages and exports them into Amazon S3. For each exported message a corresponding JSON file that contains the contents of the message is created in the specified Amazon S3 bucket.