1. Oracle GoldenGate for Distributed Applications and Analytics
  2. Replicate Data
  3. Target
  4. Oracle Cloud Infrastructure Object Storage

9.2.30 Oracle Cloud Infrastructure Object Storage

The Oracle Cloud Infrastructure Event Handler is used to load files generated by the File Writer Handler into an Oracle Cloud Infrastructure Object Store.

The Oracle Cloud Infrastructure Event Handler is used to load files generated by the Flat Files into an Oracle Cloud Infrastructure Object Store. This topic describes how to use the OCI Event Handler.

Parent topic: Target

9.2.30.1 Overview

The Oracle Cloud Infrastructure Object Storage service is an internet-scale, high-performance storage platform that offers reliable and cost-efficient data durability. The Object Storage service can store an unlimited amount of unstructured data of any content type, including analytic data and rich content, like images and videos, see https://cloud.oracle.com/en_US/cloud-infrastructure.

You can use any format handler that the File Writer Handler supports.

Parent topic: Oracle Cloud Infrastructure Object Storage

9.2.30.2 Detailing the Functionality

The Oracle Cloud Infrastructure Event Handler requires the Oracle Cloud Infrastructure Java software development kit (SDK) to transfer files to Oracle Cloud Infrastructure Object Storage. Oracle GoldenGate for Distributed Applications and Analytics (GG for DAA) does not include the Oracle Cloud Infrastructure Java SDK, see https://docs.cloud.oracle.com/iaas/Content/API/Concepts/sdkconfig.htm.

You must download the Oracle Cloud Infrastructure Java SDK at:

https://docs.us-phoenix-1.oraclecloud.com/Content/API/SDKDocs/javasdk.htm

Extract the JAR files to a permanent directory. There are two directories required by the handler, the JAR library directory that has Oracle Cloud Infrastructure SDK JAR and a third-party JAR library. Both directories must be in the gg.classpath.

Specify the gg.classpath environment variable to include the JAR files of the Oracle Cloud Infrastructure Java SDK.

Example

gg.classpath=/usr/var/oci/lib/*:/usr/var/oci/third-party/lib/*

Setting of the proxy server settings requires additional dependency libraries identified by the following Maven coordinates:

Group ID: com.oracle.oci.sdk

Artifact ID: oci-java-sdk-addons-apache

The best way to get all of the dependencies is to use the Dependency Downloading utility scripts. The OCI script downloads both the OCI Java SDK and the Apache Addons libraries.

For more information on this dependency, see OCI Documentation - README.

Parent topic: Oracle Cloud Infrastructure Object Storage

9.2.30.3 Configuration

You configure the Oracle Cloud Infrastructure Event Handler operation using the properties file. These properties are located in the Java Adapter properties file (and not in the Replicat properties file).

The Oracle Cloud Infrastructure Event Handler works only in conjunction with the File Writer Handler.

To enable the selection of the Oracle Cloud Infrastructure Event Handler, you must first configure the handler type by specifying gg.eventhandler.name.type=oci and the other Oracle Cloud Infrastructure properties as follows:

Table 9-37 Oracle Cloud Infrastructure Event Handler Configuration Properties

Properties Required/ Optional Legal Values Default Explanation

gg.eventhandler.name.type

Required

oci

None

Selects the Oracle Cloud Infrastructure Event Handler.
gg.eventhandler.name.contentType Optional Valid content type value which is used to indicate the media type of the resource. application/octet-stream The content type of the object.
gg.eventhandler.name.contentEncoding Optional Valid values indicate which encoding to be applied. utf-8 The content encoding of the object.
gg.eventhandler.name.contentLanguage Optional Valid language intended for the audience. en The content language of the object.

gg.eventhandler.name.configFilePath

Optional

Path to the event handler config file.

None

The configuration file name and location.

If gg.eventhandler.name.configFilePath is not set, then the following authentication parameters are required:
  • gg.eventhandler.name.userId
  • gg.eventhandler.name.tenancyID
  • gg.eventhandler.name.region
  • gg.eventhandler.name.privateKeyFile
  • gg.eventhandler.name.publicKeyFingerprint
These parameters take precedence over gg.eventhandler.name.configFilePath.
gg.eventhandler.name.userId Optional Valid user ID None OCID of the user calling the API. To get the value, see (Required Keys and OCIDs)https://docs.oracle.com/en-us/iaas/Content/API/Concepts/apisigningkey.htm#Required_Keys_and_OCIDs. Example: ocid1.user.oc1..<unique_ID> (shortened for brevity)
gg.eventhandler.name.tenancyId Optional Valid tenancy ID None OCID of your tenancy. To get the value, see (Required Keys and OCIDs)https://docs.oracle.com/en-us/iaas/Content/API/Concepts/apisigningkey.htm#Required_Keys_and_OCIDs. in Oracle Cloud Infrastructure documentation. Example: ocid1.tenancy.oc1..<unique_ID>
gg.eventhandler.name.privateKeyFile Optional A valid path to the file None Full path and filename of the private key.

Note:

The key pair must be in PEM format. For more information about generating a key pair in PEM format, see (Required Keys and OCIDs)https://docs.oracle.com/en-us/iaas/Content/API/Concepts/apisigningkey.htm#Required_Keys_and_OCIDs in Oracle Cloud Infrastructure documentation. Example: /home/opc/.oci/oci_api_key.pem
gg.eventhandler.name.publicKeyFingerprint Optional String None Fingerprint for the public key that was added to this user. To get the value, see (Required Keys and OCIDs) https://docs.oracle.com/en-us/iaas/Content/API/Concepts/apisigningkey.htm#Required_Keys_and_OCIDs in Oracle Cloud Infrastructure documentation.

gg.eventhandler.name.profile

Required

Valid string representing the profile name.

DEFAULT

In the Oracle Cloud Infrastructure config file, the entries are identified by the profile name. The default profile is DEFAULT. You can have an additional profile like ADMIN_USER. Any value that isn't explicitly defined for the ADMIN_USER profile (or any other profiles that you add to the config file) is inherited from the DEFAULT profile.

gg.eventhandler.name.region

Required

Oracle Cloud Infrastructure region

None

Oracle Cloud Infrastructure Servers and Data is hosted in a region and is a localized geographic area.

The valid Region Identifiers are listed at Oracle Cloud Infrastructure Documentation - Regions and Availability Domains.

gg.eventhandler.name.compartmentID

Required

Valid compartment id.

None

A compartment is a logical container to organize Oracle Cloud Infrastructure resources. The compartmentID is listed in Bucket Details while using the Oracle Cloud Infrastructure Console.

gg.eventhandler.name.pathMappingTemplate

Required

A string with resolvable keywords and constants used to dynamically generate the path in the Oracle Cloud Infrastructure bucket to write the file.

None

Use keywords interlaced with constants to dynamically generate unique Oracle Cloud Infrastructure path names at runtime. See Template Keywords.

gg.eventhandler.name.fileNameMappingTemplate

Optional

A string with resolvable keywords and constants used to dynamically generate the Oracle Cloud Infrastructure file name at runtime.

None

Use resolvable keywords and constants to dynamically generate the Oracle Cloud Infrastructure data file name at runtime. If not set, the upstream file name is used. See Template Keywords.

gg.eventhandler.name.bucketMappingTemplate

Required

A string with resolvable keywords and constants used to dynamically generate the path in the Oracle Cloud Infrastructure bucket to write the file.

None

Use resolvable keywords and constants used to dynamically generate the Oracle Cloud Infrastructure bucket name at runtime. The event handler attempts to create the Oracle Cloud Infrastructure bucket if it does not exist. See Template Keywords.

gg.eventhandler.name.finalizeAction

Optional

none | delete

None

Set to none to leave the Oracle Cloud Infrastructure data file in place on the finalize action. Set to delete if you want to delete the Oracle Cloud Infrastructure data file with the finalize action.

gg.eventhandler.name.eventHandler

Optional

A unique string identifier cross referencing a child event handler.

No event handler is configured.

Sets the event handler that is invoked on the file roll event. Event handlers can do file roll event actions like loading files to S3, converting to Parquet or ORC format, loading files to HDFS, loading files to Oracle Cloud Infrastructure Storage Classic, or loading file to Oracle Cloud Infrastructure.
gg.eventhandler.name.proxyServer Optional The host name of your proxy server. None Set to the host name of the proxy server if OCI connectivity requires routing through a proxy server.
gg.eventhandler.name.proxyPort Optional The port number of the proxy server. None Set to the port number of the proxy server if OCI connectivity requires routing through a proxy server.
gg.eventhandler.name.proxyProtocol Optional HTTP | HTTPS HTTP Sets the proxy protocol connection to the proxy server for additional level of security. The majority of proxy servers support HTTP. Only set this if the proxy server supports HTTPS and HTTPS is required.
gg.eventhandler.name.proxyUsername Optional The username for the proxy server. None Sets the username for connectivity to the proxy server if credentials are required. Most proxy servers do not require credentials.
gg.eventhandler.name.proxyPassword Optional The password for the proxy server. None Sets the password for connectivity to the proxy server if credentials are required. Most proxy servers do not require credentials.
gg.handler.name.SSEKey Optional A legal Base64 encoded OCI server side encryption key. None Allows you to control the encryption of data files loaded to OCI. OCI encrypts by default. This property allows an additional level of control by supporting encryption with a specific key. That key must also be used to decrypt data files.

Sample Configuration

gg.eventhandler.oci.type=oci
gg.eventhandler.oci.configFilePath=~/.oci/config
gg.eventhandler.oci.profile=DEFAULT
gg.eventhandler.oci.namespace=dwcsdemo
gg.eventhandler.oci.region=us-ashburn-1
gg.eventhandler.oci.compartmentID=ocid1.compartment.oc1..aaaaaaaajdg6iblwgqlyqpegf6kwdais2gyx3guspboa7fsi72tfihz2wrba
gg.eventhandler.oci.pathMappingTemplate=${schemaName}
gg.eventhandler.oci.bucketMappingTemplate=${schemaName}
gg.eventhandler.oci.fileNameMappingTemplate=${tableName}_${currentTimestamp}.txt
gg.eventhandler.oci.finalizeAction=NONE
goldengate.userexit.writers=javawriter

Parent topic: Oracle Cloud Infrastructure Object Storage

9.2.30.3.1 Automatic Configuration

OCI Object storage replication involves configuring multiple components, such as the File Writer Handler, formatter, and the target OCI Object Storage Event Handler.

The Automatic Configuration functionality helps you to auto configure these components so that the manual configuration is minimal.

The properties modified by auto-configuration is also logged in the handler log file.

To enable auto configuration to replicate to the OCI Object Storage target, set the parameter gg.target=oci.

Parent topic: Configuration

9.2.30.3.1.1 File Writer Handler Configuration

The File Writer Handler name is pre set to the value oci.

You can add or edit a property of the File Writer Handler. For example: gg.handler.oci.pathMappingTemplate=./dirout

Parent topic: Automatic Configuration

9.2.30.3.1.2 Formatter Configuration

The json row formatter is set by default.

You can add or edit a property of the formatter. For example: gg.handler.oci.format=json_row

Parent topic: Automatic Configuration

9.2.30.4 Configuring Credentials for Oracle Cloud Infrastructure

Basic configuration information like user credentials and tenancy Oracle Cloud IDs (OCIDs) of Oracle Cloud Infrastructure is required for the Java SDKs to work, see https://docs.cloud.oracle.com/iaas/Content/General/Concepts/identifiers.htm.

The ideal configuration file include keys user, fingerprint, key_file, tenancy, and region with their respective values. The default configuration file name and location is ~/.oci/config.

Create the config file as follows:

  1. Create a directory called .oci in the Oracle GoldenGate for Distributed Applications and Analytics (GG for DAA) home directory
  2. Create a text file and name it config.
  3. Obtain the values for these properties:
    user
    1. Login to the Oracle Cloud Infrastructure Console https://console.us-ashburn-1.oraclecloud.com.
    2. Click Username.
    3. Click User Settings.

      The User's OCID is displayed and is the value for the key user.

    tenancy

    The Tenancy ID is displayed at the bottom of the Console page.

    region

    The region is displayed with the header session drop-down menu in the Console.

    fingerprint

    To generate the fingerprint, use the How to Get the Key's Fingerprint instructions at:

    https://docs.cloud.oracle.com/iaas/Content/API/Concepts/apisigningkey.htm

    key_file

    You need to share the public and private key to establish a connection with Oracle Cloud Infrastructure. To generate the keys, use the How to Generate an API Signing Keyat:

    https://docs.cloud.oracle.com/iaas/Content/API/Concepts/apisigningkey.htm

    pass_phrase
    This is an optional property. It is used to configure the passphrase if the private key in the pem file is protected with a passphase. The following openssl command can be used to take an unprotected private key pem file and add a passphrase.
    The following command prompts the user for the passphrase:
    openssl rsa -aes256 -in in.pem -out out.pem

Sample Configuration File

user=ocid1.user.oc1..aaaaaaaat5nvwcna5j6aqzqedqw3rynjq
fingerprint=20:3b:97:13::4e:c5:3a:34
key_file=~/.oci/oci_api_key.pem
tenancy=ocid1.tenancy.oc1..aaaaaaaaba3pv6wkcr44h25vqstifs

Parent topic: Oracle Cloud Infrastructure Object Storage

9.2.30.5 Troubleshooting

Connectivity Issues

If the OCI Event Handler is unable to connect to the OCI object storage when running on premise, it’s likely your connectivity to the public internet is protected by a proxy server. Proxy servers act a gateway between the private network of a company and the public internet. Contact your network administrator to get the URL of your proxy server.

Oracle GoldenGate for Distributed Applications and Analytics (GG for DAA) connectivity to OCI can be routed through a proxy server by setting the following configuration properties:

gg.eventhandler.name.proxyServer={insert your proxy server name}
gg.eventhandler.name.proxyPort={insert your proxy server port number}

ClassNotFoundException Error

The most common initial error is an incorrect classpath that does not include all the required client libraries so results in a ClassNotFoundException error. Specify the gg.classpath variable to include all of the required JAR files for the Oracle Cloud Infrastructure Java SDK, see Detailing the Functionality.

Parent topic: Oracle Cloud Infrastructure Object Storage

9.2.30.6 OCI Dependencies

The maven coordinates for OCI are as follows:

Maven groupId: com.oracle.oci.sdk

Maven artifactId: oci-java-sdk-full

Version: 1.34.0

The following are the Apache add-ons to which, support routing through a proxy server:

Maven groupId: com.oracle.oci.sdk

Maven artifactId: oci-java-sdk-addons-apache

Version: 1.34.0

Parent topic: Oracle Cloud Infrastructure Object Storage

9.2.30.6.1 OCI 1.34.0 

accessors-smart-1.2.jar
aopalliance-repackaged-2.6.1.jar
asm-5.0.4.jar
bcpkix-jdk15on-1.68.jar
bcprov-jdk15on-1.68.jar
checker-qual-3.5.0.jar
commons-codec-1.15.jar
commons-io-2.8.0.jar
commons-lang3-3.8.1.jar
commons-logging-1.2.jar
error_prone_annotations-2.3.4.jar
failureaccess-1.0.1.jar
guava-30.1-jre.jar
hk2-api-2.6.1.jar
hk2-locator-2.6.1.jar
hk2-utils-2.6.1.jar
httpclient-4.5.13.jar
httpcore-4.4.13.jar
j2objc-annotations-1.3.jar
jackson-annotations-2.12.0.jar
jackson-core-2.12.0.jar
jackson-databind-2.12.0.jar
jackson-datatype-jdk8-2.12.0.jar
jackson-datatype-jsr310-2.12.0.jar
jackson-module-jaxb-annotations-2.10.1.jar
jakarta.activation-api-1.2.1.jar
jakarta.annotation-api-1.3.5.jar
jakarta.inject-2.6.1.jar
jakarta.ws.rs-api-2.1.6.jar
jakarta.xml.bind-api-2.3.2.jar
javassist-3.25.0-GA.jar
jcip-annotations-1.0-1.jar
jersey-apache-connector-2.32.jar
jersey-client-2.32.jar
jersey-common-2.32.jar
jersey-entity-filtering-2.32.jar
jersey-hk2-2.32.jar
jersey-media-json-jackson-2.32.jar
json-smart-2.3.jar
jsr305-3.0.2.jar
listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar
nimbus-jose-jwt-8.5.jar
oci-java-sdk-addons-apache-1.34.0.jar
oci-java-sdk-analytics-1.34.0.jar
oci-java-sdk-announcementsservice-1.34.0.jar
oci-java-sdk-apigateway-1.34.0.jar
oci-java-sdk-apmcontrolplane-1.34.0.jar
oci-java-sdk-apmsynthetics-1.34.0.jar
oci-java-sdk-apmtraces-1.34.0.jar
oci-java-sdk-applicationmigration-1.34.0.jar
oci-java-sdk-artifacts-1.34.0.jar
oci-java-sdk-audit-1.34.0.jar
oci-java-sdk-autoscaling-1.34.0.jar
oci-java-sdk-bds-1.34.0.jar
oci-java-sdk-blockchain-1.34.0.jar
oci-java-sdk-budget-1.34.0.jar
oci-java-sdk-cims-1.34.0.jar
oci-java-sdk-circuitbreaker-1.34.0.jar
oci-java-sdk-cloudguard-1.34.0.jar
oci-java-sdk-common-1.34.0.jar
oci-java-sdk-computeinstanceagent-1.34.0.jar
oci-java-sdk-containerengine-1.34.0.jar
oci-java-sdk-core-1.34.0.jar
oci-java-sdk-database-1.34.0.jar
oci-java-sdk-databasemanagement-1.34.0.jar
oci-java-sdk-datacatalog-1.34.0.jar
oci-java-sdk-dataflow-1.34.0.jar
oci-java-sdk-dataintegration-1.34.0.jar
oci-java-sdk-datasafe-1.34.0.jar
oci-java-sdk-datascience-1.34.0.jar
oci-java-sdk-dns-1.34.0.jar
oci-java-sdk-dts-1.34.0.jar
oci-java-sdk-email-1.34.0.jar
oci-java-sdk-events-1.34.0.jar
oci-java-sdk-filestorage-1.34.0.jar
oci-java-sdk-full-1.34.0.jar
oci-java-sdk-functions-1.34.0.jar
oci-java-sdk-goldengate-1.34.0.jar
oci-java-sdk-healthchecks-1.34.0.jar
oci-java-sdk-identity-1.34.0.jar
oci-java-sdk-integration-1.34.0.jar
oci-java-sdk-keymanagement-1.34.0.jar
oci-java-sdk-limits-1.34.0.jar
oci-java-sdk-loadbalancer-1.34.0.jar
oci-java-sdk-loganalytics-1.34.0.jar
oci-java-sdk-logging-1.34.0.jar
oci-java-sdk-loggingingestion-1.34.0.jar
oci-java-sdk-loggingsearch-1.34.0.jar
oci-java-sdk-managementagent-1.34.0.jar
oci-java-sdk-managementdashboard-1.34.0.jar
oci-java-sdk-marketplace-1.34.0.jar
oci-java-sdk-monitoring-1.34.0.jar
oci-java-sdk-mysql-1.34.0.jar
oci-java-sdk-networkloadbalancer-1.34.0.jar
oci-java-sdk-nosql-1.34.0.jar
oci-java-sdk-objectstorage-1.34.0.jar
oci-java-sdk-objectstorage-extensions-1.34.0.jar
oci-java-sdk-objectstorage-generated-1.34.0.jar
oci-java-sdk-oce-1.34.0.jar
tbcampbe: oci-java-sdk-ocvp-1.34.0.jar
oci-java-sdk-oda-1.34.0.jar
oci-java-sdk-ons-1.34.0.jar
oci-java-sdk-opsi-1.34.0.jar
oci-java-sdk-optimizer-1.34.0.jar
oci-java-sdk-osmanagement-1.34.0.jar
oci-java-sdk-resourcemanager-1.34.0.jar
oci-java-sdk-resourcesearch-1.34.0.jar
oci-java-sdk-rover-1.34.0.jar
oci-java-sdk-sch-1.34.0.jar
oci-java-sdk-secrets-1.34.0.jar
oci-java-sdk-streaming-1.34.0.jar
oci-java-sdk-tenantmanagercontrolplane-1.34.0.jar
oci-java-sdk-usageapi-1.34.0.jar
oci-java-sdk-vault-1.34.0.jar
oci-java-sdk-waas-1.34.0.jar
oci-java-sdk-workrequests-1.34.0.jar
osgi-resource-locator-1.0.3.jar
resilience4j-circuitbreaker-1.2.0.jar
resilience4j-core-1.2.0.jar
slf4j-api-1.7.29.jar
vavr-0.10.0.jar
vavr-match-0.10.0.jar

Parent topic: OCI Dependencies