Connect to OCI Streaming

Learn to create a connection to OCI Streaming to use as a source or target with OCI GoldenGate data replication and Stream Analytics.

Before you begin

Ensure that you:

• Review how OCI GoldenGate connects to your source and targets.

• Configure the required policies to enable secure Vault and Secrets access, such as use secrets, use vaults, and read secret-bundles. For more information, see Minimum recommended policies.

• If using resource principals to authenticate access to OCI Streaming, you must:

  1. Create a dynamic group for your deployment. You can choose to authorize a single deployment or all deployment in a compartment or tenancy. See Writing matching rules to define dynamic groups.

  2. Add the required policies for the dynamic group

• Obtain the Stream Pool username:

  1. From the Oracle Cloud console navigation menu, select Streaming, and then Stream Pools.

  2. On the Stream Pools page, select your pool to view its details.

  3. On the Stream Pool details page, under Resources, select Kafka Connection Settings.

  4. Copy the username for SASL Connection Strings.

     Note: If you’re using Oracle Identity Cloud service, you must add OracleIdentityCloudService to your user name. For example, <tenancy-name>/OracleIdentityCloudService/<username>/<streampool-ocid>

  5. Create an Auth token:

     1. In the Oracle Cloud console global header, select Profile, and then select User settings.

     2. On the User Details page, under Resources, select Auth Tokens, and then select Generate Token.

     3. In the Generate Token dialog, enter a description, and then select Generate Token.

     4. Copy the auth token from the dialog to a secure location from where you can retrieve it later, and then select Close.

  The Stream Pool username and Auth token will be entered for the Stream connection’s username and password.

Create the source connection

To create a source OCI Streaming connection:

1. From the OCI GoldenGate Overview page, select Connections.

   You can also select Create Connection under the Get started section and skip to step 3.

2. On the Connections page, select Create Connection.

3. On the Create Connection page, complete the fields as follows:

   1. For Name, enter a name for the connection.

   2. (Optional) For Description, enter a description that helps you distinguish this connection from others.

   3. (For GoldenGate on Multicloud only) Select your Subscription, and then complete the following fields.

      1. From the Compartment dropdown, select the compartment in which the Resource Anchor resides.

      2. Select the Multicloud partner region.

      3. Select your Partner availability zone. The available options populate based on the selected Multicloud partner region.

   4. For Compartment, select the compartment in which to create the connection.

   5. From the Type dropdown, select OCI Streaming.

   6. For Stream pool details, you can choose:

      • Select a stream pool from the dropdown, or select Change compartment to select one in a different compartment.

      • Enter stream pool information to manually enter the Bootstrap server's Host and Port.

        Note: If you enter a private IP, then OCI GoldenGate rewrites the private IP in the format, ip-10-0-0-0.ociggsvc.oracle.vcn.com.

   7. For User, select how to authenticate and access the stream pool:

      • Username and password: enter the Stream Pool username copied from the SASL Connection Settings in the prerequisite steps above.

      • Use resource principal: authenticates using resource principal to access other OCI resources.

   8. Select the Database user password secret. If located in a different compartment, use the dropdown to change compartments.

      Note:

      • Secrets are credentials such as passwords, certificates, SSH keys, or authentication tokens that you use with OCI services. To create a secret, see Creating a secret. Ensure that you:

        • Select Manual secret generation.

        • Paste the credentials into Secret contents.

      • If you prefer not to use password secrets, ensure that you deselect Use secrets in vault in the Security section under Advanced Options, located at the bottom of this form.

      • When you need to update the Secret content, ensure that you:

        • Create a new Secret version using the Plain-Text template and provide the updated content. For more information, see Updating a Secret’s Content.

        • Refresh the connection to clear cached Secret content.

   9. Expand Show advanced options. You can configure the following options:

      • Security

        • Deselect Use vault secrets you prefer not to use password secrets for this connection. If not selected:

          • Select Use Oracle-managed encryption key to leave all encryption key management to Oracle.

          • Select Use customer-managed encryption key to select a specific encryption key stored in your OCI Vault to encrypt your connection credentials.

      • Network connectivity

        • Shared endpoint, to share an endpoint with the assigned deployment. You must allow connectivity from the deployment's ingress IP.

        • Dedicated endpoint, for network traffic through a dedicated endpoint in the assigned subnet in your VCN. You must allow connectivity from this connection's ingress IPs.

          Note:

          • If a dedicated connection remains unassigned for seven days, then the service converts it to a shared connection.
          • Learn more about Oracle GoldenGate connectivity.

      • Settings

        To capture from OCI Streaming, create a consumer.properties file with one of the following deserializers or converters, and then drag and drop the file to Consumer properties:

        • Kafka Consumer properties for JSON deserializer:

          key.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer
          value.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer

        • Kafka Consumer properties for JSON converter:

          key.converter=org.apache.kafka.connect.json.JsonConverter
          value.converter=org.apache.kafka.connect.json.JsonConverter

        • Kafka Consumer properties for Avro converter:

          key.converter=io.confluent.connect.avro.AvroConverter
          value.converter=io.confluent.connect.avro.AvroConverter

      • Security attributes: Add security attributes to control access to this connection using Zero Trust Packet Routing (ZPR).

      • Tags: Add tags to organize your resources.

4. Select Create.

After the connection is created, it appears in the Connections list. Ensure that you assign the connection to a deployment to use it as a source or target in a replication.

Create the target connection

To create a target OCI Streaming connection:

1. From the OCI GoldenGate Overview page, select Connections.

   You can also select Create Connection under the Get started section and skip to step 3.

2. On the Connections page, select Create Connection.

3. On the Create Connection page, complete the fields as follows:

   1. For Name, enter a name for the connection.

   2. (Optional) For Description, enter a description that helps you distinguish this connection from others.

   3. (For GoldenGate on Multicloud only) Select your Subscription, and then complete the following fields.

      1. From the Compartment dropdown, select the compartment in which the Resource Anchor resides.

      2. Select the Multicloud partner region.

      3. Select your Partner availability zone. The available options populate based on the selected Multicloud partner region.

   4. For Compartment, select the compartment in which to create the connection.

   5. From the Type dropdown, select OCI Streaming.

   6. For Stream pool details, you can choose:

      • Select a stream pool
        1. Select a stream pool from the dropdown, or select Change compartment to select one in a different compartment.
      • Enter stream pool information to manually enter the stream pool information.
        1. Enter the Bootstrap server's Host and Port.

        Note: If you enter a private IP, then OCI GoldenGate rewrites the private IP in the format, ip-10-0-0-0.ociggsvc.oracle.vcn.com.

   7. For User, select how to authenticate and access the stream pool:

      • Username and password: enter the Stream Pool username copied from the SASL Connection Settings in the prerequisite steps above.

      • Use resource principal: authenticates using resource principal to access other OCI resources.

   8. Select the Database user password secret. If located in a different compartment, use the dropdown to change compartments.

      Note:

      • Secrets are credentials such as passwords, certificates, SSH keys, or authentication tokens that you use with OCI services. To create a secret, see Creating a secret. Ensure that you:

        • Select Manual secret generation.

        • Paste the credentials into Secret contents.

      • If you prefer not to use password secrets, ensure that you deselect Use secrets in vault in the Security section under Advanced Options, located at the bottom of this form.

      • When you need to update the Secret content, ensure that you:

        • Create a new Secret version using the Plain-Text template and provide the updated content. For more information, see Updating a Secret’s Content.

        • Refresh the connection to clear cached Secret content.

   9. Expand Show advanced options. You can configure the following options:

      • Security

        • Deselect Use vault secrets you prefer not to use password secrets for this connection. If not selected:

          • Select Use Oracle-managed encryption key to leave all encryption key management to Oracle.

          • Select Use customer-managed encryption key to select a specific encryption key stored in your OCI Vault to encrypt your connection credentials.

      • Network connectivity

        • Shared endpoint, to share an endpoint with the assigned deployment. You must allow connectivity from the deployment's ingress IP.

        • Dedicated endpoint, for network traffic through a dedicated endpoint in the assigned subnet in your VCN. You must allow connectivity from this connection's ingress IPs.

          Note:

          • If a dedicated connection remains unassigned for seven days, then the service converts it to a shared connection.
          • Learn more about Oracle GoldenGate connectivity.

      • Settings: To use Snappy compression in Kafka replication, drag and drop or select Producer properties, and change replication settings as discussed in Using Compression OCI GoldenGate (Confluent) Kafka Replication.

      • Security attributes: Add security attributes to control access to this connection using Zero Trust Packet Routing (ZPR).

      • Tags: Add tags to organize your resources.

4. Select Create.

After the connection is created, it appears in the Connections list. Ensure that you assign the connection to a deployment to use it as a source or target in a replication.

Next steps

• Assign a connection to a deployment
• Manage connections