Oracle® Cloud

Migrating Oracle Event Hub Cloud Service to Oracle Cloud Infrastructure

F17006-02

March 2019

Migrating Oracle Event Hub Cloud Service to Oracle Cloud Infrastructure

Learn about the benefits of migrating your existing Oracle Event Hub Cloud Service to Oracle Cloud Infrastructure and get an overview of the migration process.

Why Migrate to Oracle Cloud Infrastructure

Oracle encourages you to migrate your existing cloud resources to Oracle Cloud Infrastructure regions. You can gain several advantages by doing so.

In Oracle Cloud, you provision resources in specific regions, which are localized to geographic locations. A region supports either the Oracle Cloud Infrastructure Classic or Oracle Cloud Infrastructure platform.

Oracle Cloud Infrastructure is Oracle's more modern infrastructure platform that's based on the latest cloud technologies and standards. It typically provides better performance than Oracle Cloud Infrastructure Classic. Oracle Cloud Infrastructure also has more predictable pricing and lower costs in terms of Oracle Compute Units (OCPUs) per hour. Most importantly, Oracle continues to invest in Oracle Cloud Infrastructure, including the addition of new regions, services, and features. See Data Regions for Platform and Infrastructure Services.

You can benefit from these additional administrative features in Oracle Cloud Infrastructure when you migrate your cloud resources from Oracle Cloud Infrastructure Classic:

  • Organize cloud resources into a hierarchy of logical compartments.
  • Create fine-grained access policies for each compartment.

About Oracle Cloud Infrastructure

Get familiar with basic Oracle Cloud Infrastructure security, network, and storage concepts.

Cloud resources in Oracle Cloud Infrastructure are created in logical compartments. You also create fine-grained policies to control access to the resources within a compartment.

You create instances within an Oracle Cloud Infrastructure region. You also specify an availability domain (AD), if supported in the selected region.

A virtual cloud network (VCN) is comprised of one or more subnets, and an instance is assigned to a specific subnet. Oracle Cloud Infrastructure does not allow you to reserve specific IP addresses for platform services.

A subnet's security lists permit and block traffic to and from specific IP addresses and ports.

Instances can communicate with resources outside of Oracle Cloud by using Oracle Cloud Infrastructure FastConnect, which provides a fast, dedicated connection to your on-premises network. Alternatively, use an IPSec VPN.

A bucket in Oracle Cloud Infrastructure Object Storage can be used to store files and share them with multiple instances. A user's generated authentication token (auth token) is required to access the bucket.

To learn more, see Key Concepts and Terminology in the Oracle Cloud Infrastructure documentation.

About the Migration Scope

The following alternatives are available to migrate from Oracle Event Hub Cloud Service to Oracle Cloud Infrastructure.
  1. Migrate to Oracle Cloud Infrastructure Streaming service

    Oracle Cloud Infrastructure Streaming service provides a fully managed, scalable, persistent storage option for continuous, high-volume streams of data that you can consume and process in real-time. See Oracle Cloud Infrastructure Streaming.

  2. Migrate to Confluent Kafka running on Oracle Cloud Infrastructure

    Run user managed Confluent Community Edition 5.0 on Oracle Cloud Infrastructure Compute without losing much of the functionality offered by Oracle Event Hub Cloud Service. Oracle Event Hub Cloud Service is built on Confluent Community Edition 5.0 with additional features of its own.

Factors to Consider while Selecting Migration Alternatives

Consider the following factors while selecting the migration alternatives.
  1. Current usage of Oracle Event Hub Cloud Service
    • Do you use REST API/Oracle PaaS Service Manager CLI to manage Topics, Produce/Consume messages?
    • Do you also use REST API/ Oracle PaaS Service Manager CLI for other functionality like Kafka Connect, Schema Registry, to administer Oracle Event Hub Cloud Service Kafka cluster?
  2. Throughput requirements
    • Understand your application's Kafka throughput requirements and limits on Oracle Cloud Infrastructure Streaming service. See Limits on Streaming Resources.
  3. Access to Oracle Event Hub Cloud Service cluster
    • Do you need SSH access to the Oracle Event Hub Cloud Service cluster?
    • Do you need SSH access to the cluster on an ongoing basis to use Oracle Event Hub Cloud Service?

Guidance for Selecting Migration Alternatives

Select Migrating to Oracle Cloud Infrastructure Streaming Service, if the following conditions are true:
  • You only use Oracle Event Hub Cloud Service REST API to Produce/Consume messages and/or create Topics AND
  • Your required throughput requirements are within the limits of Oracle Cloud Infrastructure Streaming Service AND
  • SSH or root access to Oracle Event Hub Cloud Service cluster is not required.

If any of these conditions is not true, then select Migrating to Confluent Kafka running on Oracle Cloud Infrastructure.

Migrating to Oracle Cloud Infrastructure Streaming Service

The following section lists the steps to migrate to Oracle Cloud Infrastructure Streaming Service with an example. Let’s assume you have an application named APP_PRODUCER which produces messages for Oracle Event Hub Cloud Service using REST API and an application named APP_CONSUMER which consumes messages from Oracle Event Hub Cloud Service using REST API.
  1. For each topic used by your application on Oracle Event Hub Cloud Service, create a stream on Oracle Cloud Infrastructure Streaming Service. See Managing Streams.
  2. Update APP_PRODUCER to send any new messages to Oracle Cloud Infrastructure Streaming service. See Publishing Messages.
  3. Update APP_CONSUMER to consume messages from both Oracle Event Hub Cloud Service topics as well as Oracle Cloud Infrastructure Streaming Service streams until all messages from Oracle Event Hub Cloud Service are consumed or expired. See Consuming Messages.
  4. Update APP_CONSUMER to now consume messages only from Oracle Cloud Infrastructure Streaming Service streams.

Manage Topics

Console: If you used Oracle Event Hub Cloud Service console to manage topics, you can use Oracle Cloud Infrastructure Streaming Service console to manage streams.

REST API/Oracle PaaS Service Manager CLI: Let’s assume, you have an application named EVENTHUB_MGR which was used to administer topics on Oracle Event Hub Cloud Service using REST API or Oracle PaaS Service Manager CLI. Update EVENTHUB_MGR to replace Oracle Event Hub Cloud Service admin/topics REST API/CLI calls with Oracle Cloud Infrastructure Streaming Service REST API/CLI calls to manage streams. See Using the API.

Following is a sample REST API call to create streams in Oracle Cloud Infrastructure Streaming Service.
POST /20180418/streams
Host: streaming.us-phoenix-1.oraclecloud.com
<authorization and other headers>
{
  "compartmentId": "ocid1.tenancy.oc1..exampleasgadvszj5gcdefghijk",
  "name": "mynewstream",
  "partitions": "4"
}

Produce Messages

Use Oracle Cloud Infrastructure Streaming Service’s REST API/CLI PutMessages to publish messages. See PutMessages.

Following is a sample REST API call to put message in Oracle Cloud Infrastructure Streaming Service.
POST /20180418/streams/<streamId>/messages
Host: streaming-api.us-phoenix-1.oraclecloud.com
<authorization and other headers>
{
  "messages":
  {
    {
      "key": null,
      "value": "VGhlIHF1aWNrIGJyabcdefghikjlbGF6eSBkb2cu"
    },
    {
      "key": null,
      "value": "UGFjayBteSaebecdaedfgdhijnklBqdWsdfsdg=="
    }
  }
}

Consume Messages

Use Oracle Cloud Infrastructure Streaming Service’s REST API/CLI GetMessages to consume messages. See GetMessages.

Following is a sample REST API call to get message in Oracle Cloud Infrastructure Streaming Service.
GET /20180418/streams/<streamId>/messages?cursor=<cursor>&limit=100
Host: streaming.us-phoenix-1.oraclecloud.com
<authorization and other headers>

Note:

Instead of using REST API, you can also use CLI or SDK to access Oracle Cloud Infrastructure resources. For a list of available SDKs, see Software Development Kits and Command Line Interface.

Migrating to Confluent Kafka running on Oracle Cloud Infrastructure

You can migrate your existing Oracle Event Hub Cloud Service cluster to Confluent Kafka on Oracle Cloud Infrastructure IaaS using MirrorMaker, which comes in-built with Apache Kafka. You can use MirrorMaker to mirror or replicate a cluster to another cluster running on Oracle Cloud Infrastructure. For example, you can use it to migrate your Apache Kafka cluster to Confluent Kafka on Oracle Cloud Infrastructure. See Mirroring data between clusters.

If you have an Oracle Event Hub Cloud Service cluster named CLUSTER_EVENTHUB which is populated with topics and data and if you want to migrate that cluster to a newly created Oracle Cloud Infrastructure Confluent Kafka cluster named CLUSTER_OCI_CONFLUENT, then the following section provides a high-level view of the steps that you need to follow.

Steps to migrate your existing Oracle Event Hub Cloud Service cluster to Confluent Kafka on Oracle Cloud Infrastructure

  1. Deploy Confluent Community Edition 5.0 on Oracle Cloud Infrastructure in your tenancy. Oracle Cloud Infrastructure team has built Terraform deployment template to automate deployment of Confluent Community Edition 5.0 on Oracle Cloud Infrastructure. Reach out to your Oracle Sales team to gain access to the Terraform template and its documentation.
  2. In CLUSTER_OCI_CONFLUENT, create all the topics that you want to migrate.

    You can't use MirrorMaker for this step because it doesn't automatically re-create the topics that you want to migrate with the right replication level. You need to create the topics in CLUSTER_OCI_CONFLUENT with the same replication factors and numbers of partitions that they had in CLUSTER_EVENTHUB. Optionally, you can also create the topics with different replication factors and numbers of partitions.

  3. Start MirrorMaker from an instance that has read access to CLUSTER_EVENTHUB and write access to CLUSTER_OCI_CONFLUENT.
  4. Run the following command to mirror all topics:
    /usr/bin/kafka-mirror-maker --consumer.config consumer.properties --producer.config producer.properties --whitelist '.*'

    In this command, consumer.properties points to a bootstrap broker in CLUSTER_EVENTHUB; for example, bootstrap.servers=localhost:9092. And producer.properties points to a bootstrap broker in CLUSTER_OCI_CONFLUENT; for example, bootstrap.servers=10.0.0.237:9092,10.0.2.196:9092,10.0.1.233:9092

  5. Keep running MirrorMaker in the background and continue to use CLUSTER_EVENTHUB. MirrorMaker mirrors all new data.
  6. Check the progress of mirroring by inspecting the lag between the last offset for each topic and the current offset from which MirrorMaker is consuming.

    Remember that MirrorMaker is simply using a consumer and a producer. So, you can check the lag using the /usr/bin/kafka-consumer-groups tool. To find the consumer group name, look inside the consumer.properties file for the group.id, and use its value. If there is no such key in the file, you can create it. For example, set group.id=mirrormaker-consumer-group.

  7. After MirrorMaker finishes mirroring all topics, stop all producers and consumers, and then stop MirrorMaker. Then redirect the producers and consumers to the CLUSTER_OCI_CONFLUENT cluster by changing their producer and consumer bootstrap brokers values. Restart all producers and consumers on CLUSTER_OCI_CONFLUENT.

Confluent Connectors

Confluent Community Edition 5.0 comes with a lot of connectors. See Confluent Hub.
  • JDBC connector can be used to move data from/to Kafka into Oracle Data Stores (like Oracle databases, Autonomous Data Warehouse, Autonomous Transaction Processing). See Kafka Connect JDBC.
  • S3 connector can be used to move data from Kafka into Oracle Cloud Infrastructure Object Storage. Oracle Cloud Infrastructure Object Storage service includes Amazon S3 Compatibility API, so customers can continue to use existing S3 Connector to integrate with Oracle Cloud Infrastructure Object Storage service. See Kafka Connect S3, Overview of Object Storage, Amazon S3 Compatibility API.
By directly running Confluent Community Edition 5.0, customers can get most of the features except the following:
  • User Interface console (UI) for monitoring and managing the Kafka cluster. The Confluent Community Edition 5.0 does not come with a UI for managing and monitoring clusters. Customers can purchase Confluent Enterprise Edition license to get UI support.
  • Asynchronous messaging and notifications. Consumers essentially pull data from Kafka, but Oracle Event Hub Cloud Service also allows customers to add web-hooks and avoid polling. This feature is not available from Confluent or in their Enterprise Edition of Kafka.
  • Topic-level security and integration with Oracle Identity Cloud Service security realms. This is Oracle Event Hub Cloud Service specific and will not be available from third-party Kafka distributors.

Options to deploy Confluent Kafka on Oracle Cloud Infrastructure

  • Oracle Cloud Infrastructure team has built Terraform deployment template to automate deployment of Confluent Community Edition 5.0 on Oracle Cloud Infrastructure. The template deploys all required infrastructure resources (virtual cloud network, subnet, security list, compute instances, storage) as well as Confluent Kafka. Reach out to your Oracle Sales team to gain access to the Terraform template and its documentation.
  • Both Confluent Community and Enterprise Editions are available for download from Confluent’s website. The customer is required to obtain a license from Confluent when installing the Enterprise Edition. Follow the Confluent’s documentation for installing and running Kafka.

Oracle Cloud Migrating Oracle Event Hub Cloud Service to Oracle Cloud Infrastructure

F17006-02

Copyright © 2019, Oracle and/or its affiliates. All rights reserved.

This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited.

The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us in writing.

If this is software or related documentation that is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, then the following notice is applicable:

U.S. GOVERNMENT END USERS: Oracle programs, including any operating system, integrated software, any programs installed on the hardware, and/or documentation, delivered to U.S. Government end users are "commercial computer software" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such, use, duplication, disclosure, modification, and adaptation of the programs, including any operating system, integrated software, any programs installed on the hardware, and/or documentation, shall be subject to license terms and license restrictions applicable to the programs. No other rights are granted to the U.S. Government.

This software or hardware is developed for general use in a variety of information management applications. It is not developed or intended for use in any inherently dangerous applications, including applications that may create a risk of personal injury. If you use this software or hardware in dangerous applications, then you shall be responsible to take all appropriate fail-safe, backup, redundancy, and other measures to ensure its safe use. Oracle Corporation and its affiliates disclaim any liability for any damages caused by use of this software or hardware in dangerous applications.

Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.

Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices. UNIX is a registered trademark of The Open Group.

This software or hardware and documentation may provide access to or information about content, products, and services from third parties. Oracle Corporation and its affiliates are not responsible for and expressly disclaim all warranties of any kind with respect to third-party content, products, and services unless otherwise set forth in an applicable agreement between you and Oracle. Oracle Corporation and its affiliates will not be responsible for any loss, costs, or damages incurred due to your access to or use of third-party content, products, or services, except as set forth in an applicable agreement between you and Oracle.