Learn About Streaming Kafka topics to Oracle Autonomous Database using Oracle Integration 3
Kafka topics have various use cases and the data has to be constantly received and processed. There are several ways you can consume a streaming Kafka in Oracle Cloud Infrastructure (OCI).
In this solution playbook, we use Oracle Integration 3 to capture Kafka streaming messages from Oracle GoldenGate Stream Analytics and store the data in Oracle Autonomous Database. Oracle GoldenGate Stream Analytics allows users to build scalable data transformation and analytics pipelines for agile real-time business insights. You can also consume a Kafka stream from an on-premises instance.
Before You Begin
You must download and install Kafka in your environment and the following Kafka topics, producers should be up and running to produce messages.
Architecture
This architecture shows on-premises Kafka Streams, Oracle Integration 3, Oracle GoldenGate Stream Analytics and Oracle Autonomous Database in an OCI region.
Use this architecture to capture Kafka streaming messages from an on-premises Kafka stream or Oracle GoldenGate Stream Analytics into Oracle Autonomous Database using Oracle Integration 3.
The OCI region containing Oracle Integration 3 ingests data from the on-premises Kafka streams or Oracle GoldenGate Stream Analytics.
Oracle Integration 3 stores the data in a relational table in Oracle Autonomous Database.
This architecture supports the following components:
- Kafka Streams
Kafka Streams is a client library for building applications and microservices, where the input and output data are stored in Kafka clusters. It combines the simplicity of writing and deploying standard Java and Scala applications on the client side with the benefits of Kafka's server-side cluster technology.
- Autonomous Database
Oracle Autonomous Database is a fully-managed, preconfigured database environment that you can use for transaction processing and data warehousing workloads. You do not need to configure or manage any hardware, or install any software. Oracle Cloud Infrastructure handles creating, backing up, patching, upgrading, and tuning the database.
- Oracle Integration 3
Oracle Integration 3 is a fully managed, preconfigured environment that gives you the power to integrate your cloud and on-premises applications, automate business processes, develop visual applications, use an SFTP-compliant file server to store and retrieve files, and exchange business documents with a B2B trading partner.
- Oracle GoldenGate Stream Analytics
GoldenGate Stream Analytics allows for the creation of custom operational dashboards that provide real-time monitoring and analyses of event streams in an Apache Spark-based system. It enables customers to identify events of interest in their Apache Spark-based system, execute queries against those event streams in real time and drive operational dashboards or raise alerts based on that analysis.
- Oracle SQL Developer
Oracle SQL Developer is a free, integrated development environment that simplifies the development and management of Oracle Database in both traditional and Cloud deployments. SQL Developer offers complete end-to-end development of your PL/SQL applications, a worksheet for running queries and scripts, a DBA console for managing the database, a reports interface, a complete data modeling solution, and a migration platform for moving your 3rd party databases to Oracle.
About Required Products and Roles
This solution requires the following products and roles:
- Oracle Autonomous Database
- Oracle Integration 3
- Oracle Cloud Infrastructure
- Oracle GoldenGate Stream Analytics
These are the roles needed for each service.
| Product Name: Role | Required to... |
|---|---|
| Oracle Autonomous Database: admin | Create the credentials. |
| Oracle Integration 3: admin | Create the credentials. |
| Oracle Cloud Infrastructure: admin | Create and manage OCI resources. |
| Oracle GoldenGate Stream Analytics: admin | Configure Kafka producer to ingest data. |
See Oracle Products, Solutions, and Services to get what you need.