8.2.28 Oracle Cloud Infrastructure Object Storage
The Oracle Cloud Infrastructure Event Handler is used to load files generated by the File Writer Handler into an Oracle Cloud Infrastructure Object Store.
The Oracle Cloud Infrastructure Event Handler is used to load files generated by the Flat Files into an Oracle Cloud Infrastructure Object Store. This topic describes how to use the OCI Event Handler.
- Overview
- Detailing the Functionality
- Configuration
- Configuring Credentials for Oracle Cloud Infrastructure
- Troubleshooting
- OCI Dependencies
Parent topic: Target
8.2.28.1 Overview
The Oracle Cloud Infrastructure Object Storage service is an internet-scale, high-performance storage platform that offers reliable and cost-efficient data durability. The Object Storage service can store an unlimited amount of unstructured data of any content type, including analytic data and rich content, like images and videos, see https://cloud.oracle.com/en_US/cloud-infrastructure.
You can use any format handler that the File Writer Handler supports.
Parent topic: Oracle Cloud Infrastructure Object Storage
8.2.28.2 Detailing the Functionality
The Oracle Cloud Infrastructure Event Handler requires the Oracle Cloud Infrastructure Java software development kit (SDK) to transfer files to Oracle Cloud Infrastructure Object Storage. Oracle GoldenGate for Big Data does not include the Oracle Cloud Infrastructure Java SDK, see https://docs.cloud.oracle.com/iaas/Content/API/Concepts/sdkconfig.htm.
You must download the Oracle Cloud Infrastructure Java SDK at:
https://docs.us-phoenix-1.oraclecloud.com/Content/API/SDKDocs/javasdk.htm
Extract the JAR files to a permanent directory. There are two directories required by the handler, the JAR library directory that has Oracle Cloud Infrastructure SDK JAR and a third-party JAR library. Both directories must be in the gg.classpath
.
Specify the gg.classpath
environment variable to include the JAR files of the Oracle Cloud Infrastructure Java SDK.
Example
gg.classpath=/usr/var/oci/lib/*:/usr/var/oci/third-party/lib/*
Setting of the proxy server settings requires additional dependency libraries identified by the following Maven coordinates:
Group ID: com.oracle.oci.sdk
Artifact ID: oci-java-sdk-addons-apache
The best way to get all of the dependencies is to use the Dependency Downloading utility scripts. The OCI script downloads both the OCI Java SDK and the Apache Addons libraries.
For more information on this dependency, see OCI Documentation - README.
Parent topic: Oracle Cloud Infrastructure Object Storage
8.2.28.3 Configuration
You configure the Oracle Cloud Infrastructure Event Handler operation using the properties file. These properties are located in the Java Adapter properties file (and not in the Replicat properties file).
The Oracle Cloud Infrastructure Event Handler works only in conjunction with the File Writer Handler.
To enable the selection of the Oracle Cloud Infrastructure Event Handler, you must
first configure the handler type by specifying
gg.eventhandler.name.type=oci
and the other Oracle Cloud
Infrastructure properties as follows:
Table 8-34 Oracle Cloud Infrastructure Event Handler Configuration Properties
Properties | Required/ Optional | Legal Values | Default | Explanation |
---|---|---|---|---|
|
Required |
|
None |
Selects the Oracle Cloud Infrastructure Event Handler. |
gg.eventhandler.name.contentType |
Optional | Valid content type value which is used to indicate the media type of the resource. | application/octet-stream |
The content type of the object. |
gg.eventhandler.name.contentEncoding |
Optional | Valid values indicate which encoding to be applied. | utf-8 | The content encoding of the object. |
gg.eventhandler.name.contentLanguage |
Optional | Valid language intended for the audience. | en | The content language of the object. |
|
Optional |
Path to the event handler |
None |
The configuration file name and location. Ifgg.eventhandler.name.configFilePath
is not set, then the following authentication parameters are
required:
gg.eventhandler.name.configFilePath .
|
gg.eventhandler.name.userId |
Optional | Valid user ID | None | OCID of the user calling the API. To get the value,
see (Required Keys and OCIDs)https://docs.oracle.com/en-us/iaas/Content/API/Concepts/apisigningkey.htm#Required_Keys_and_OCIDs.
Example: ocid1.user.oc1..<unique_ID>
(shortened for brevity)
|
gg.eventhandler.name.tenancyId |
Optional | Valid tenancy ID | None | OCID of your tenancy. To get the value, see
(Required Keys and OCIDs)https://docs.oracle.com/en-us/iaas/Content/API/Concepts/apisigningkey.htm#Required_Keys_and_OCIDs.
in Oracle Cloud Infrastructure documentation. Example:
ocid1.tenancy.oc1..<unique_ID> |
gg.eventhandler.name.privateKeyFile |
Optional | A valid path to the file | None | Full path and filename of the private key.
Note: The key pair must be in PEM format. For more information about generating a key pair in PEM format, see (Required Keys and OCIDs)https://docs.oracle.com/en-us/iaas/Content/API/Concepts/apisigningkey.htm#Required_Keys_and_OCIDs in Oracle Cloud Infrastructure documentation. Example:
/home/opc/.oci/oci_api_key.pem |
gg.eventhandler.name.publicKeyFingerprint |
Optional | String | None | Fingerprint for the public key that was added to this user. To get the value, see (Required Keys and OCIDs) https://docs.oracle.com/en-us/iaas/Content/API/Concepts/apisigningkey.htm#Required_Keys_and_OCIDs in Oracle Cloud Infrastructure documentation. |
|
Required |
Valid string representing the profile name. |
|
In the Oracle Cloud Infrastructure |
|
Required |
Oracle Cloud Infrastructure region |
None |
Oracle Cloud Infrastructure Servers and Data is hosted in a region and is a localized geographic area. The valid Region Identifiers are listed at Oracle Cloud Infrastructure Documentation - Regions and Availability Domains. |
|
Required |
Valid compartment id. |
None |
A compartment is a logical container to organize Oracle Cloud Infrastructure resources. The |
|
Required |
A string with resolvable keywords and constants used to dynamically generate the path in the Oracle Cloud Infrastructure bucket to write the file. |
None |
Use keywords interlaced with constants to dynamically generate unique Oracle Cloud Infrastructure path names at runtime. See Template Keywords. |
|
Optional |
A string with resolvable keywords and constants used to dynamically generate the Oracle Cloud Infrastructure file name at runtime. |
None |
Use resolvable keywords and constants to dynamically generate the Oracle Cloud Infrastructure data file name at runtime. If not set, the upstream file name is used. See Template Keywords. |
|
Required |
A string with resolvable keywords and constants used to dynamically generate the path in the Oracle Cloud Infrastructure bucket to write the file. |
None |
Use resolvable keywords and constants used to dynamically generate the Oracle Cloud Infrastructure bucket name at runtime. The event handler attempts to create the Oracle Cloud Infrastructure bucket if it does not exist. See Template Keywords. |
|
Optional |
|
None |
Set to |
|
Optional |
A unique string identifier cross referencing a child event handler. |
No event handler is configured. |
Sets the event handler that is invoked on the file roll event. Event handlers can do file roll event actions like loading files to S3, converting to Parquet or ORC format, loading files to HDFS, loading files to Oracle Cloud Infrastructure Storage Classic, or loading file to Oracle Cloud Infrastructure. |
gg.eventhandler.name.proxyServer |
Optional | The host name of your proxy server. | None | Set to the host name of the proxy server if OCI connectivity requires routing through a proxy server. |
gg.eventhandler.name.proxyPort |
Optional | The port number of the proxy server. | None | Set to the port number of the proxy server if OCI connectivity requires routing through a proxy server. |
gg.eventhandler.name.proxyProtocol |
Optional |
HTTP | HTTPS |
HTTP |
Sets the proxy protocol connection to the proxy server for additional level of security. The majority of proxy servers support HTTP. Only set this if the proxy server supports HTTPS and HTTPS is required. |
gg.eventhandler.name.proxyUsername |
Optional | The username for the proxy server. | None | Sets the username for connectivity to the proxy server if credentials are required. Most proxy servers do not require credentials. |
gg.eventhandler.name.proxyPassword |
Optional | The password for the proxy server. | None | Sets the password for connectivity to the proxy server if credentials are required. Most proxy servers do not require credentials. |
gg.handler.name.SSEKey |
Optional | A legal Base64 encoded OCI server side encryption key. | None | Allows you to control the encryption of data files loaded to OCI. OCI encrypts by default. This property allows an additional level of control by supporting encryption with a specific key. That key must also be used to decrypt data files. |
Sample Configuration
gg.eventhandler.oci.type=oci gg.eventhandler.oci.configFilePath=~/.oci/config gg.eventhandler.oci.profile=DEFAULT gg.eventhandler.oci.namespace=dwcsdemo gg.eventhandler.oci.region=us-ashburn-1 gg.eventhandler.oci.compartmentID=ocid1.compartment.oc1..aaaaaaaajdg6iblwgqlyqpegf6kwdais2gyx3guspboa7fsi72tfihz2wrba gg.eventhandler.oci.pathMappingTemplate=${schemaName} gg.eventhandler.oci.bucketMappingTemplate=${schemaName} gg.eventhandler.oci.fileNameMappingTemplate=${tableName}_${currentTimestamp}.txt gg.eventhandler.oci.finalizeAction=NONE goldengate.userexit.writers=javawriter
8.2.28.3.1 Automatic Configuration
OCI Object storage replication involves configuring multiple components, such as the File Writer Handler, formatter, and the target OCI Object Storage Event Handler.
The Automatic Configuration functionality helps you to auto configure these components so that the manual configuration is minimal.
The properties modified by auto-configuration is also logged in the handler log file.
To enable auto configuration to replicate to the OCI Object Storage
target, set the parameter gg.target=oci
.
8.2.28.3.1.1 File Writer Handler Configuration
The File Writer Handler name is pre set to the value
oci
.
You can add or edit a property of the File Writer Handler. For example:
gg.handler.oci.pathMappingTemplate=./dirout
Parent topic: Automatic Configuration
8.2.28.3.1.2 Formatter Configuration
The json row formatter is set by default.
You can add or edit a property of the formatter. For example:
gg.handler.oci.format=json_row
Parent topic: Automatic Configuration
8.2.28.4 Configuring Credentials for Oracle Cloud Infrastructure
Basic configuration information like user credentials and tenancy Oracle Cloud IDs (OCIDs) of Oracle Cloud Infrastructure is required for the Java SDKs to work, see https://docs.cloud.oracle.com/iaas/Content/General/Concepts/identifiers.htm.
The ideal configuration file include keys user
, fingerprint
, key_file
, tenancy
, and region
with their respective values. The default configuration file name and location is ~/.oci/config
.
Create the config
file as follows:
- Create a directory called
.oci
in the Oracle GoldenGate for Big Data home directory - Create a text file and name it
config
. - Obtain the values for these properties:
-
user
-
- Login to the Oracle Cloud Infrastructure Console https://console.us-ashburn-1.oraclecloud.com.
- Click Username.
- Click User Settings.
The User's OCID is displayed and is the value for the key user.
-
tenancy
-
The Tenancy ID is displayed at the bottom of the Console page.
-
region
-
The region is displayed with the header session drop-down menu in the Console.
-
fingerprint
-
To generate the fingerprint, use the How to Get the Key's Fingerprint instructions at:
https://docs.cloud.oracle.com/iaas/Content/API/Concepts/apisigningkey.htm
-
key_file
-
You need to share the public and private key to establish a connection with Oracle Cloud Infrastructure. To generate the keys, use the How to Generate an API Signing Keyat:
https://docs.cloud.oracle.com/iaas/Content/API/Concepts/apisigningkey.htm
-
pass_phrase
- This is an optional property. It is used to configure
the passphrase if the private key in the pem file is protected with
a passphase. The following openssl command can be used to take an
unprotected private key pem file and add a passphrase.
The following command prompts the user for the passphrase:
openssl rsa -aes256 -in in.pem -out out.pem
-
Sample Configuration File
user=ocid1.user.oc1..aaaaaaaat5nvwcna5j6aqzqedqw3rynjq fingerprint=20:3b:97:13::4e:c5:3a:34 key_file=~/.oci/oci_api_key.pem tenancy=ocid1.tenancy.oc1..aaaaaaaaba3pv6wkcr44h25vqstifs
Parent topic: Oracle Cloud Infrastructure Object Storage
8.2.28.5 Troubleshooting
Connectivity Issues
If the OCI Event Handler is unable to connect to the OCI object storage when running on premise, it’s likely your connectivity to the public internet is protected by a proxy server. Proxy servers act a gateway between the private network of a company and the public internet. Contact your network administrator to get the URL of your proxy server.
Oracle GoldenGate for Big Data connectivity to OCI can be routed through a proxy server by setting the following configuration properties:
gg.eventhandler.name.proxyServer={insert your proxy server name} gg.eventhandler.name.proxyPort={insert your proxy server port number}
ClassNotFoundException Error
The most common initial error is an incorrect classpath that does not include all the required client libraries so results in a ClassNotFoundException
error. Specify the gg.classpath
variable to include all of the required JAR files for the Oracle Cloud Infrastructure Java SDK, see Detailing the Functionality.
Parent topic: Oracle Cloud Infrastructure Object Storage
8.2.28.6 OCI Dependencies
The maven coordinates for OCI are as follows:
Maven groupId: com.oracle.oci.sdk
Maven artifactId: oci-java-sdk-full
Version: 1.34.0
The following are the Apache add-ons to which, support routing through a proxy server:
Maven groupId: com.oracle.oci.sdk
Maven artifactId: oci-java-sdk-addons-apache
Version: 1.34.0
Parent topic: Oracle Cloud Infrastructure Object Storage
8.2.28.6.1 OCI 1.34.0
accessors-smart-1.2.jar aopalliance-repackaged-2.6.1.jar asm-5.0.4.jar bcpkix-jdk15on-1.68.jar bcprov-jdk15on-1.68.jar checker-qual-3.5.0.jar commons-codec-1.15.jar commons-io-2.8.0.jar commons-lang3-3.8.1.jar commons-logging-1.2.jar error_prone_annotations-2.3.4.jar failureaccess-1.0.1.jar guava-30.1-jre.jar hk2-api-2.6.1.jar hk2-locator-2.6.1.jar hk2-utils-2.6.1.jar httpclient-4.5.13.jar httpcore-4.4.13.jar j2objc-annotations-1.3.jar jackson-annotations-2.12.0.jar jackson-core-2.12.0.jar jackson-databind-2.12.0.jar jackson-datatype-jdk8-2.12.0.jar jackson-datatype-jsr310-2.12.0.jar jackson-module-jaxb-annotations-2.10.1.jar jakarta.activation-api-1.2.1.jar jakarta.annotation-api-1.3.5.jar jakarta.inject-2.6.1.jar jakarta.ws.rs-api-2.1.6.jar jakarta.xml.bind-api-2.3.2.jar javassist-3.25.0-GA.jar jcip-annotations-1.0-1.jar jersey-apache-connector-2.32.jar jersey-client-2.32.jar jersey-common-2.32.jar jersey-entity-filtering-2.32.jar jersey-hk2-2.32.jar jersey-media-json-jackson-2.32.jar json-smart-2.3.jar jsr305-3.0.2.jar listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar nimbus-jose-jwt-8.5.jar oci-java-sdk-addons-apache-1.34.0.jar oci-java-sdk-analytics-1.34.0.jar oci-java-sdk-announcementsservice-1.34.0.jar oci-java-sdk-apigateway-1.34.0.jar oci-java-sdk-apmcontrolplane-1.34.0.jar oci-java-sdk-apmsynthetics-1.34.0.jar oci-java-sdk-apmtraces-1.34.0.jar oci-java-sdk-applicationmigration-1.34.0.jar oci-java-sdk-artifacts-1.34.0.jar oci-java-sdk-audit-1.34.0.jar oci-java-sdk-autoscaling-1.34.0.jar oci-java-sdk-bds-1.34.0.jar oci-java-sdk-blockchain-1.34.0.jar oci-java-sdk-budget-1.34.0.jar oci-java-sdk-cims-1.34.0.jar oci-java-sdk-circuitbreaker-1.34.0.jar oci-java-sdk-cloudguard-1.34.0.jar oci-java-sdk-common-1.34.0.jar oci-java-sdk-computeinstanceagent-1.34.0.jar oci-java-sdk-containerengine-1.34.0.jar oci-java-sdk-core-1.34.0.jar oci-java-sdk-database-1.34.0.jar oci-java-sdk-databasemanagement-1.34.0.jar oci-java-sdk-datacatalog-1.34.0.jar oci-java-sdk-dataflow-1.34.0.jar oci-java-sdk-dataintegration-1.34.0.jar oci-java-sdk-datasafe-1.34.0.jar oci-java-sdk-datascience-1.34.0.jar oci-java-sdk-dns-1.34.0.jar oci-java-sdk-dts-1.34.0.jar oci-java-sdk-email-1.34.0.jar oci-java-sdk-events-1.34.0.jar oci-java-sdk-filestorage-1.34.0.jar oci-java-sdk-full-1.34.0.jar oci-java-sdk-functions-1.34.0.jar oci-java-sdk-goldengate-1.34.0.jar oci-java-sdk-healthchecks-1.34.0.jar oci-java-sdk-identity-1.34.0.jar oci-java-sdk-integration-1.34.0.jar oci-java-sdk-keymanagement-1.34.0.jar oci-java-sdk-limits-1.34.0.jar oci-java-sdk-loadbalancer-1.34.0.jar oci-java-sdk-loganalytics-1.34.0.jar oci-java-sdk-logging-1.34.0.jar oci-java-sdk-loggingingestion-1.34.0.jar oci-java-sdk-loggingsearch-1.34.0.jar oci-java-sdk-managementagent-1.34.0.jar oci-java-sdk-managementdashboard-1.34.0.jar oci-java-sdk-marketplace-1.34.0.jar oci-java-sdk-monitoring-1.34.0.jar oci-java-sdk-mysql-1.34.0.jar oci-java-sdk-networkloadbalancer-1.34.0.jar oci-java-sdk-nosql-1.34.0.jar oci-java-sdk-objectstorage-1.34.0.jar oci-java-sdk-objectstorage-extensions-1.34.0.jar oci-java-sdk-objectstorage-generated-1.34.0.jar oci-java-sdk-oce-1.34.0.jar tbcampbe: oci-java-sdk-ocvp-1.34.0.jar oci-java-sdk-oda-1.34.0.jar oci-java-sdk-ons-1.34.0.jar oci-java-sdk-opsi-1.34.0.jar oci-java-sdk-optimizer-1.34.0.jar oci-java-sdk-osmanagement-1.34.0.jar oci-java-sdk-resourcemanager-1.34.0.jar oci-java-sdk-resourcesearch-1.34.0.jar oci-java-sdk-rover-1.34.0.jar oci-java-sdk-sch-1.34.0.jar oci-java-sdk-secrets-1.34.0.jar oci-java-sdk-streaming-1.34.0.jar oci-java-sdk-tenantmanagercontrolplane-1.34.0.jar oci-java-sdk-usageapi-1.34.0.jar oci-java-sdk-vault-1.34.0.jar oci-java-sdk-waas-1.34.0.jar oci-java-sdk-workrequests-1.34.0.jar osgi-resource-locator-1.0.3.jar resilience4j-circuitbreaker-1.2.0.jar resilience4j-core-1.2.0.jar slf4j-api-1.7.29.jar vavr-0.10.0.jar vavr-match-0.10.0.jar
Parent topic: OCI Dependencies