Using the Oracle Cloud Infrastructure Event Handler

9.1 Overview

The Oracle Cloud Infrastructure Object Storage service is an internet-scale, high-performance storage platform that offers reliable and cost-efficient data durability. The Object Storage service can store an unlimited amount of unstructured data of any content type, including analytic data and rich content, like images and videos, see https://cloud.oracle.com/en_US/cloud-infrastructure.

You can use any format handler that the File Writer Handler supports.

Parent topic: Using the Oracle Cloud Infrastructure Event Handler

9.2 Detailing the Functionality

The Oracle Cloud Infrastructure Event Handler requires the Oracle Cloud Infrastructure Java software development kit (SDK) to transfer files to Oracle Cloud Infrastructure Object Storage. Oracle GoldenGate for Big Data does not include the Oracle Cloud Infrastructure Java SDK, see https://docs.cloud.oracle.com/iaas/Content/API/Concepts/sdkconfig.htm.

You must download the Oracle Cloud Infrastructure Java SDK at:

https://docs.us-phoenix-1.oraclecloud.com/Content/API/SDKDocs/javasdk.htm

Extract the JAR files to a permanent directory. There are two directories required by the handler, the JAR library directory that has Oracle Cloud Infrastructure SDK JAR and a third-party JAR library. Both directories must be in the gg.classpath.

Specify the gg.classpath environment variable to include the JAR files of the Oracle Cloud Infrastructure Java SDK.

Example

gg.classpath=/usr/var/oci/lib/*:/usr/var/oci/third-party/lib/*

For more information, see OCI Dependencies.

Parent topic: Using the Oracle Cloud Infrastructure Event Handler

9.3 Configuring the Oracle Cloud Infrastructure Event Handler

You configure the Oracle Cloud Infrastructure Event Handler operation using the properties file. These properties are located in the Java Adapter properties file (not in the Replicat properties file).

The Oracle Cloud Infrastructure Event Handler works only in conjunction with the File Writer Handler.

To enable the selection of the Oracle Cloud Infrastructure Event Handler, you must first configure the handler type by specifying gg.eventhandler.name.type=oci and the other Oracle Cloud Infrastructure properties as follows:

Table 9-1 Oracle Cloud Infrastructure Event Handler Configuration Properties

Properties	Required/ Optional	Legal Values	Default	Explanation
`gg.eventhandler.name.type`	Required	`oci`	None	Selects the Oracle Cloud Infrastructure Event Handler.
`gg.eventhandler.name.contentType`	Optional	Valid content type value which is used to indicate the media type of the resource.	`application/octet-stream`	The content type of the object.
`gg.eventhandler.name.contentEncoding`	Optional	Valid values indicate which encoding to be applied.	utf-8	The content encoding of the object.
`gg.eventhandler.name.contentLanguage`	Optional	Valid language intended for the audience.	en	The content language of the object.
`gg.eventhandler.name.configFilePath`	Required	Path to the event handler `config` file.	None	The configuration file name and location.
`gg.eventhandler.name.profile`	Required	Valid string representing the profile name.	None	In the Oracle Cloud Infrastructure `config` file, the entries are identified by the profile name. The default profile is `DEFAULT`. You can have an additional profile like `ADMIN_USER`. Any value that isn't explicitly defined for the `ADMIN_USER` profile (or any other profiles that you add to the `config` file) is inherited from the `DEFAULT` profile.
`gg.eventhandler.name.namespace`	Required	Oracle Cloud Infrastructure namespace.	None	The namespace serves as a top-level container for all buckets and objects and allows you to control bucket naming within user’s tenancy. The Object Storage namespace is a system-generated string assigned during account creation. Your namespace string is listed in Object Storage Settings while using the Oracle Cloud Infrastructure Console.
`gg.eventhandler.name.region`	Required	Oracle Cloud Infrastructure region	None	Oracle Cloud Infrastructure Servers and Data is hosted in a region and is a localized geographic area. The valid Region Identifiers are listed at Oracle Cloud Infrastructure Documentation - Regions and Availability Domains.
`gg.eventhandler.name.compartmentID`	Required	Valid compartment id.	None	A compartment is a logical container to organize Oracle Cloud Infrastructure resources. The `compartmentID` is listed in Bucket Details while using the Oracle Cloud Infrastructure Console.
`gg.eventhandler.name.pathMappingTemplate`	Required	A string with resolvable keywords and constants used to dynamically generate the path in the Oracle Cloud Infrastructure bucket to write the file.	None	Use keywords interlaced with constants to dynamically generate unique Oracle Cloud Infrastructure path names at runtime.
`gg.eventhandler.name.fileNameMappingTemplate`	Optional	A string with resolvable keywords and constants used to dynamically generate the Oracle Cloud Infrastructure file name at runtime.	None	Use resolvable keywords and constants to dynamically generate the Oracle Cloud Infrastructure data file name at runtime. If not set, the upstream file name is used.
`gg.eventhandler.name.bucketMappingTemplate`	Required	A string with resolvable keywords and constants used to dynamically generate the path in the Oracle Cloud Infrastructure bucket to write the file.	None	Use resolvable keywords and constants used to dynamically generate the Oracle Cloud Infrastructure bucket name at runtime. The event handler attempts to create the Oracle Cloud Infrastructure bucket if it does not exist.
`gg.eventhandler.name.finalizeAction`	Optional	`none` \| `delete`	None	Set to `none` to leave the Oracle Cloud Infrastructure data file in place on the finalize action. Set to `delete` if you want to delete the Oracle Cloud Infrastructure data file with the finalize action.
`gg.eventhandler.name.eventHandler`	Optional	A unique string identifier cross referencing a child event handler.	No event handler is configured.	Sets the event handler that is invoked on the file roll event. Event handlers can do file roll event actions like loading files to S3, converting to Parquet or ORC format, loading files to HDFS, loading files to Oracle Cloud Infrastructure Storage Classic, or loading file to Oracle Cloud Infrastructure.
`gg.eventhandler.name.proxyServer`	Optional	The host name of your proxy server.	None	Set to the host name of the proxy server if OCI connectivity requires routing through a proxy server.
`gg.eventhandler.name.proxyPort`	Optional	The port number of the proxy server.	None	Set to the port number of the proxy server if OCI connectivity requires routing through a proxy server.
`gg.eventhandler.name.proxyProtocol`	Optional	`HTTP \| HTTPS`	`HTTP`	Sets the proxy protocol connection to the proxy server for additional level of security. The majority of proxy servers support HTTP. Only set this if the proxy server supports HTTPS and HTTPS is required.
`gg.eventhandler.name.proxyUsername`	Optional	The username for the proxy server.	None	Sets the username for connectivity to the proxy server if credentials are required. Most proxy servers do not require credentials.
`gg.eventhandler.name.proxyPassword`	Optional	The password for the proxy server.	None	Sets the password for connectivity to the proxy server if credentials are required. Most proxy servers do not require credentials.

Sample Configuration

gg.eventhandler.oci.type=oci
gg.eventhandler.oci.configFilePath=~/.oci/config
gg.eventhandler.oci.profile=DEFAULT
gg.eventhandler.oci.namespace=dwcsdemo
gg.eventhandler.oci.region=us-ashburn-1
gg.eventhandler.oci.compartmentID=ocid1.compartment.oc1..aaaaaaaajdg6iblwgqlyqpegf6kwdais2gyx3guspboa7fsi72tfihz2wrba
gg.eventhandler.oci.pathMappingTemplate=${schemaName}
gg.eventhandler.oci.bucketMappingTemplate=${schemaName}
gg.eventhandler.oci.fileNameMappingTemplate=${tableName}_${currentTimestamp}.txt
gg.eventhandler.oci.finalizeAction=NONE
goldengate.userexit.writers=javawriter

Parent topic: Using the Oracle Cloud Infrastructure Event Handler

9.4 Configuring Credentials for Oracle Cloud Infrastructure

Basic configuration information like user credentials and tenancy Oracle Cloud IDs (OCIDs) of Oracle Cloud Infrastructure is required for the Java SDKs to work, see https://docs.cloud.oracle.com/iaas/Content/General/Concepts/identifiers.htm.

The ideal configuration file include keys user, fingerprint, key_file, tenancy, and region with their respective values. The default configuration file name and location is ~/.oci/config.

Create the config file as follows:

Create a directory called .oci in the Oracle GoldenGate for Big Data home directory
Create a text file and name it config.
Obtain the values for these properties:
user
1. Login to the Oracle Cloud Infrastructure Console https://console.us-ashburn-1.oraclecloud.com.
2. Click Username.
3. Click User Settings.
  
  The User's OCID is displayed and is the value for the key user.
tenancy

The Tenancy ID is displayed at the bottom of the Console page.

region

The region is displayed with the header session drop-down menu in the Console.

fingerprint

To generate the fingerprint, use the How to Get the Key's Fingerprint instructions at:

https://docs.cloud.oracle.com/iaas/Content/API/Concepts/apisigningkey.htm

key_file

You need to share the public and private key to establish a connection with Oracle Cloud Infrastructure. To generate the keys, use the How to Generate an API Signing Keyat:

https://docs.cloud.oracle.com/iaas/Content/API/Concepts/apisigningkey.htm

Sample Configuration File

user=ocid1.user.oc1..aaaaaaaat5nvwcna5j6aqzqedqw3rynjq
fingerprint=20:3b:97:13::4e:c5:3a:34
key_file=~/.oci/oci_api_key.pem
tenancy=ocid1.tenancy.oc1..aaaaaaaaba3pv6wkcr44h25vqstifs

Parent topic: Using the Oracle Cloud Infrastructure Event Handler

9.5 Using Templated Strings

Templated strings can contain a combination of string constants and keywords that are dynamically resolved at runtime. This event handler makes extensive use of templated strings to generate the Oracle Cloud Infrastructure directory names, data file names, and Oracle Cloud Infrastructure bucket names. These strings give you the flexibility to select where to write data files and the names of those data files. You should exercise caution when choosing file and directory names to avoid file naming collisions that can result in an abend.

Supported Templated Strings

Keyword	Description
`${fullyQualifiedTableName}`	The fully qualified source table name delimited by a period (`.`). For example, `MYCATALOG.MYSCHEMA.MYTABLE`.
`${catalogName}`	The individual source catalog name. For example, `MYCATALOG`.
`${schemaName}`	The individual source schema name. For example, `MYSCHEMA`.
`${tableName}`	The individual source table name. For example, `MYTABLE`.
`${groupName}`	The name of the Replicat process (with the thread number appended if you’re using coordinated apply).
`${emptyString}`	Evaluates to an empty string. For example,`“”`
`${operationCount}`	The total count of operations in the data file. It must be used either on rename or by the event handlers or it will be zero (`0`) because nothing is written yet. For example, `“1024”`.
`${insertCount}`	The total count of insert operations in the data file. It must be used either on rename or by the event handlers or it will be zero (`0`) because nothing is written yet. For example, `“125”`.
`${updateCount}`	The total count of update operations in the data file. It must be used either on rename or by the event handlers or it will be zero (`0`) because nothing is written yet. For example, `“265”`.
`${deleteCount}`	The total count of delete operations in the data file. It must be used either on rename or by the event handlers or it will be zero (`0`) because nothing is written yet. For example, `“11”`.
`${truncateCount}`	The total count of truncate operations in the data file. It must be used either on rename or by the event handlers or it will be zero (`0`) because nothing is written yet. For example, `“5”`.
`${currentTimestamp}`	The current timestamp. The default output format for the date time is `yyyy-MM-dd_HH-mm-ss.SSS`. For example, `2017-07-05_04-31-23.123`. Alternatively, you can customize the format of the current timestamp by inserting the format inside square brackets like: `${currentTimestamp[MM-dd_HH]}` This format uses the syntax defined in the Java `SimpleDateFormat` class, see https://docs.oracle.com/javase/8/docs/api/java/text/SimpleDateFormat.html.
`${toUpperCase[]}`	Converts the contents inside the square brackets to uppercase. For example, `${toUpperCase[${fullyQualifiedTableName}]}`.
`${toLowerCase[]}`	Converts the contents inside the square brackets to lowercase. For example, `${toLowerCase[${fullyQualifiedTableName}]}`.

Configuration of template strings can use a mix of keywords and static strings to assemble path and data file names at runtime.

Path Configuration Example

/usr/local/${fullyQualifiedTableName}

Data File Configuration Example

${fullyQualifiedTableName}_${currentTimestamp}_${groupName}.txt

Requirements

The directory and file names generated using the templates must be legal on the system being written to. File names must be unique to avoid a file name collision. You can avoid a collision by adding a current timestamp using the ${currentTimestamp} keyword. If you are using coordinated apply, then adding ${groupName} into the data file name is recommended.

Parent topic: Using the Oracle Cloud Infrastructure Event Handler

9.6 Troubleshooting

Connectivity Issues

If the OCI Event Handler is unable to connect to the OCI object storage when running on premise, it’s likely your connectivity to the public internet is protected by a proxy server. Proxy servers act a gateway between the private network of a company and the public internet. Contact your network administrator to get the URL of your proxy server.

Oracle GoldenGate for Big Data connectivity to OCI can be routed through a proxy server by setting the following configuration properties:

gg.eventhandler.name.proxyServer={insert your proxy server name}
gg.eventhandler.name.proxyPort={insert your proxy server port number}

ClassNotFoundException Error

The most common initial error is an incorrect classpath that does not include all the required client libraries so results in a ClassNotFoundException error. Specify the gg.classpath variable to include all of the required JAR files for the Oracle Cloud Infrastructure Java SDK, see Detailing the Functionality.

Parent topic: Using the Oracle Cloud Infrastructure Event Handler