Apache Iceberg

Apache Iceberg is a high-performance table format for extremely large analytic tables. Apache Iceberg brings the reliability and simplicity of SQL tables to GG for DAA, while making it possible for engines, such as Spark, Trino, Flink, Presto, Hive, and Impala to safely work with the same tables, at the same time.

Detailed Functionality

The Oracle GoldenGate Iceberg Replicat can replicate GoldenGate trail records to Iceberg tables.

The Iceberg open-table-format files could be written to local files, AWS Simple Storage Service(S3), Google Cloud Storage(GCS), or Azure DataLake Storage(ADLS).

Replication without a SQL Engine

Oracle GoldenGate Iceberg Replicat process does not require a SQL engine to replicate data to Iceberg tables.

It uses the Iceberg Java SDK along with object storage specific Java SDK to write data to Iceberg tables.

Iceberg File Format

The default file format for Iceberg data files and delete files is Parquet.

Oracle GoldenGate can be configured to write files in any of the following Iceberg supported file formats:

Iceberg Catalog

Oracle GoldenGate supports the following Iceberg catalogs:

Iceberg Specification

Oracle GoldenGate generates data files and delete files as per the Iceberg specification version 3 or version 2.

See https://iceberg.apache.org/spec/

Delete Files and Merge-On-Read (MoR)

Oracle GoldenGate generates Iceberg delete files for the UPDATE and DELETE operations.

Therefore, the Iceberg table property write.update.mode is always set to merge-on-read.

SQL engines should support merge-on-read to query tables replicated by Oracle GoldenGate.

Iceberg supports two types of delete files:

Operation Support

The Iceberg event handler supports the following operations:

Compressed Update Handling

A compressed update record in the Oracle GoldenGate trail file contains values for the key columns and the modified columns.

An uncompressed update record contains values for all the columns.

Oracle GoldenGate trails may contain compressed or uncompressed update records. The default extract configuration writes compressed updates to the trail files.

If there are missing column values in the update operations, then Replicat will ABEND.

This behavior can be overridden by setting the parameter gg.eventhandler.iceberg.abendOnMissingColumns=false in the Replicat properties file.

When the parameter is set to false, Replicat will handle compressed updates by querying the previous values of the missing columns from the Iceberg table.

Lookup Missing values in Sparse Updates

The lookup of the missing values is an expensive operation and may impact the performance of the Replicat process.

By default, Oracle GoldenGate writes records to Iceberg in micro batches every ten minutes.

Every micro-batch for a table can potentially contain millions of rows.

Micro batches will be processed for every target table in concurrent threads.

Therefore, it is critical that sufficient JVM heap memory is allocated to the Replicat process.

The lookup is performed only for such rows that contain at least one missing value in the update operation.

Oracle GoldenGate will automatically create target tables. During auto-creation of tables, Oracle GoldenGate Replicat will enable creation of Iceberg metrics (min/max values) for all the identifier (key) columns.

The metrics are stored in the Iceberg metadata files.

Iceberg metrics helps speed up the lookup of the missing values in the UPDATE operations.

INSERTALLRECORDS Support

Iceberg event handler supports INSERTALLRECORDS parameter. See:INSERTALLRECORDS. Set the INSERTALLRECORDS parameter in the Replicat parameter file (.prm).

Setting this property directs the Replicat process to generate Iceberg data files to append operation data into the Iceberg target table.

Operation Aggregation

Operation aggregation is the process of aggregating (merging/compressing) multiple operations on the same row into a single output operation based on a threshold.

Operation records are aggregated in-memory.

You can tune the frequency of apply interval using gg.handler.iceberg.fileRollInterval property, the default value is set to 15m (fifteen minutes).

The Replicat process will generate Iceberg data files and delete files for the aggregated operations.

Automatic Table Creation

Oracle GoldenGate Replicat will automatically create target tables if the target table does not exist.

Iceberg Metadata Provider

A new metadata provider for Iceberg is implemented to retrieve the Iceberg target table metadata.

Iceberg Metadata provider is auto configured and enabled by the Replicat process.

Iceberg Identifier Fields

The identifier fields in the Iceberg table are used to uniquely identify the rows in the Iceberg table.

During the automatic table creation, Oracle GoldenGate maps the key columns to Iceberg identifier fields.

Note: Iceberg tables without identifier fields are not supported in the current release.

Primary Key Updates and Truncates

Configuration

The configuration of the Iceberg replication properties is stored in the Replicat properties file.

Automatic Configuration

Iceberg replication involves configuring multiple components, such as the File Writer Handler, and the target Iceberg Event Handler.

The Automatic Configuration functionality helps you to autoconfigure these components so that the manual configuration is minimal.

The properties modified by autoconfiguration is also logged in the handler log file.

To enable autoconfiguration to replicate to the Iceberg target, set the parameter gg.target=iceberg.

File Writer Configuration

The File Writer Handler name is pre-set to the value iceberg and its properties are automatically set to the required values for Iceberg.

Iceberg Event Handler Configuration

The Iceberg Event Handler name is pre-set to the value iceberg.

This topic details the configuration properties available for the Iceberg Event handler, the required ones must be changed to match your Iceberg configuration.

Common Iceberg Properties

Iceberg can be configured to work with multiple catalogs and object stores.

The following are the common properties:

Properties Required/Optional Legal Values Default Explanation
gg.eventhandler.iceberg.warehouseLocation Optional String value. None Directory path to the Iceberg warehouse location excluding the object storage scheme. Example: /path/to/warehouse. This is a required property when using the hadoop catalog. For other Iceberg catalogs, warehouse location has a catalog specific requirement.
gg.eventhandler.iceberg.fileRollInterval Optional The default unit of measure is milliseconds. You can stipulate ms, s, m, h to signify milliseconds, seconds, minutes, or hours respectively. Examples of legal values include 10000, 10000ms, 10s, 10m, or    
1.5h. Values of 0 or less indicate that file rolling on time is turned off. 15m The parameter determines how often the data will be pushed into the Iceberg warehouse. Use with caution, the higher this value is the more data will need to be stored in the memory of the Replicat process. <div class="infoboxnote" id="GUID-6CAA2B8E-FE37-439E-AC6F-B468981AF46C__GUID-058F0204-EB6C-4D63-963A-0E318777DC5D"> <p class="notep1">Note:</p>Use the parameter with caution. Increasing its default value (15m) will increase the amount of data stored in the internal memory of the Replicat. This can cause out of memory errors and stop the Replicat if it runs out of memory. </div>    
gg.eventhandler.iceberg.fileSystemScheme Optional String value. file:// Warehouse scheme to indicate the Iceberg object storage location. Valid values are: file://, gs://, s3://, s3a://, abfss://. For more information, see File System Scheme.
gg.eventhandler.iceberg.catalogType Optional String value. hadoop Iceberg catalog type. Valid values are: hadoop, nessie, rest, glue, polaris.
gg.eventhandler.iceberg.fileFormat Optional parquet, orc, or avro. parquet Iceberg table file format to be used in target tables. Supported file formats: Parquet, Avro, and ORC.
gg.eventhandler.iceberg.icebergTableProperties Optional String value. None Path to a table properties file to specify additional Iceberg table properties to set to the target tables.
gg.eventhandler.iceberg.abendOnMissingColumns Optional true or false. true When set to true and the UPDATE operation contains a missing value, Replicat will ABEND. When set to false, Replicat will not ABEND if UPDATE operations have missing column values. The missing columns values will be read by querying the target tables. This lookup may impact the performance of the Replicat process.
gg.eventhandler.iceberg.abendOnSchemaChanges Optional true or false true When set to true and schema changes are detected, the replicat process will ABEND. User can manually update the target schema and set the configuration to false to proceed. When set to false, a warning message is logged for schema changes.
gg.validate.keyupdate Optional true or false false If set to true, Replicat will validate key update operations (optype 115) and correct to normal update if no key values have changed.
File System Scheme

The gg.eventhandler.iceberg.fileSystemScheme property is used to specify the object storage scheme.

The following are the supported object storage schemes:

Iceberg Common Dependencies

The following are the common Iceberg dependencies:

<dependencies>
       <!-- Common Iceberg dependencies START -->
       <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>3.4.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-mapreduce-client-core</artifactId>
            <version>3.4.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.iceberg</groupId>
            <artifactId>iceberg-arrow</artifactId>
            <version>1.6.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.iceberg</groupId>
            <artifactId>iceberg-core</artifactId>
            <version>1.6.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.iceberg</groupId>
            <artifactId>iceberg-data</artifactId>
            <version>1.6.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.iceberg</groupId>
            <artifactId>iceberg-parquet</artifactId>
            <version>1.6.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.iceberg</groupId>
            <artifactId>iceberg-gcp</artifactId>
            <version>1.6.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.iceberg</groupId>
            <artifactId>iceberg-aws</artifactId>
            <version>1.6.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.iceberg</groupId>
            <artifactId>iceberg-orc</artifactId>
            <version>1.6.1</version>
        </dependency>
    <dependency>
        <groupId>org.apache.iceberg</groupId>
        <artifactId>iceberg-nessie</artifactId>
        <version>1.6.1</version>
    </dependency>
    <!-- Common Iceberg dependencies END -->
</dependencies>

You can download the dependencies from maven central using the script download_dependencies.sh in the DependencyDownloader directory.

Follow these steps:

  1. Change directory to DependencyDownloader.

  2. Edit config_proxy.sh if proxy configuration is required.

  3. Run the script:

    ./download_dependencies.sh xmls/iceberg-common.xml

    This script will download the dependencies and store them in the iceberg-common directory. gg.classpath can be configured to include the dependencies from the iceberg-common directory as follows: gg.classpath=/path/to/DependencyDownloader/dependencies/iceberg-common/*

AWS Java SDK dependencies for Writing to AWS S3 (s3:// Scheme)

The following are the Iceberg dependencies to write to AWS S3 using the s3:// scheme:

<dependencies>
    <!-- s3:// scheme dependencies START -->
    <dependency>
        <groupId>software.amazon.awssdk</groupId>
        <artifactId>s3</artifactId>
        <version>2.28.6</version>
    </dependency>
    <dependency>
        <groupId>software.amazon.awssdk</groupId>
        <artifactId>sts</artifactId>
        <version>2.28.6</version>
    </dependency>
    <dependency>
        <groupId>software.amazon.awssdk</groupId>
        <artifactId>glue</artifactId>
        <version>2.28.6</version>
    </dependency>
    <dependency>
        <groupId>software.amazon.awssdk</groupId>
        <artifactId>url-connection-client</artifactId>
        <version>2.28.6</version>
    </dependency>
    <!-- s3:// scheme dependencies END -->
</dependencies>

The dependencies can be downloaded from maven central using the script download_dependencies.sh in the DependencyDownloader directory.

Follow these steps:

This script will download the dependencies and store them in the iceberg-aws-java-sdk directory.

gg.classpath: can be configured to include the dependencies as follows:

gg.classpath=/path/to/DependencyDownloader/dependencies/iceberg-aws-java-sdk/*:/path/to/DependencyDownloader/dependencies/iceberg-common/*
Hadoop AWS SDK Dependencies for Writing to AWS S3 (s3a:// Scheme)

The following are the Iceberg dependencies to write to AWS S3 using the s3a:// scheme:

<dependencies>
    <!-- s3a:// scheme dependencies START -->
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-aws</artifactId>
        <version>3.4.0</version>
    </dependency>
    <!-- s3a:// scheme dependencies END -->
</dependencies>

You can download the dependencies from maven central using the script download_dependencies.sh in the DependencyDownloader directory.

Follow these steps:

This script will download the dependencies and store them in the iceberg-hadoop-aws directory.

gg.classpath can be configured to include the dependencies as follows:

gg.classpath=/path/to/DependencyDownloader/dependencies/iceberg-hadoop-aws/*:/path/to/DependencyDownloader/dependencies/iceberg-common/*
Hadoop Google Cloud Storage SDK Dependencies for Writing to Google Cloud Storage (GCS)

The following are the Iceberg dependencies to write to GCS using the Hadoop GCS SDK:

<dependencies>
    <!-- gs:// scheme dependencies START -->
    <dependency>
        <groupId>com.google.cloud.bigdataoss</groupId>
        <artifactId>gcs-connector</artifactId>
        <version>hadoop3-2.2.22</version>
    </dependency>
    <!-- gs:// scheme dependencies END -->
</dependencies>

The dependencies can be downloaded from maven central using the script download_dependencies.sh in the DependencyDownloader directory.

Follow these steps:

This script will download the dependencies and store them in the iceberg-hadoop-gcs directory.

gg.classpath can be configured to include the dependencies as follows:

g.classpath=/path/to/DependencyDownloader/dependencies/iceberg-hadoop-gcs/*:/path/to/DependencyDownloader/dependencies/iceberg-common/*
Google Cloud Storage SDK Dependencies for Writing to Google Cloud Storage (GCS)

The following are the Iceberg dependencies to write to GCS using the Google Cloud Storage Java SDK:

<dependencies>
    <dependency>
        <groupId>com.google.cloud</groupId>
        <artifactId>google-cloud-storage</artifactId>
        <version>2.37.0</version>
    </dependency>
</dependencies>

The dependencies can be downloaded from maven central using the script download_dependencies.sh in the DependencyDownloader directory.

Follow these steps:

This script will download the dependencies and store them in the iceberg-gcs-java-sdk directory.

gg.classpath can be configured to include the dependencies as follows:

gg.classpath=/path/to/DependencyDownloader/dependencies/iceberg-hadoop-gcs/*:/path/to/DependencyDownloader/dependencies/iceberg-common/*
Hadoop Azure SDK Dependencies for Writing to Azure Data Lake (ADLS)

The following are the Iceberg dependencies to write to ADLS using the Hadoop Azure Java SDK:

<dependencies>
    <!-- abfss:// scheme dependencies START -->
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-azure</artifactId>
        <version>3.4.0</version>
    </dependency>
    <!-- abfss:// scheme dependencies END -->
</dependencies>

The dependencies can be downloaded from maven central using the script download_dependencies.sh in the DependencyDownloader directory.

Follow these steps:

This script will download the dependencies and store them in the iceberg-hadoop-azure directory.

gg.classpath: can be configured to include the dependencies as follows:

gg.classpath=/path/to/DependencyDownloader/dependencies/iceberg-hadoop-azure/*:/path/to/DependencyDownloader/dependencies/iceberg-common/*

Configuration for Iceberg Nessie Catalog

Configuration for Nessie Catalog and AWS S3 s3:// Scheme

The following are the configuration properties for the Nessie catalog and AWS S3 object store using s3:// scheme:

Properties Required/Optional Legal Values Default Explanation
gg.eventhandler.iceberg.catalogType Optional String value. hadoop nessie
gg.eventhandler.iceberg.nessieBranch Optional String value. main Nessie Catalog branch name where the Iceberg table metadata exists.
gg.eventhandler.iceberg.nessieUri Required String value. None Nessie Catalog endpoint URI. Example: 
http://.com:10001/api/v2
.
gg.eventhandler.iceberg.fileSystemScheme Optional String value. file:// File system scheme to indicate AWS S3 object storage location: s3://.
gg.eventhandler.iceberg.awsS3Region Required String value. None AWS S3 bucket region. Example: us-east-2.
gg.eventhandler.iceberg.awsS3Bucket Required String value. None AWS S3 bucket name that houses the Iceberg Warehouse.
gg.eventhandler.iceberg.awsAccessKeyId Optional String value. None AWS access key id for authentication.
gg.eventhandler.iceberg.awsSecretKey Optional String value. None AWS secret access key for authentication.
gg.eventhandler.iceberg.awsSessionToken Optional String value. None AWS session token for authentication.
gg.eventhandler.iceberg.awsRoleArn Optional String value. None AWS role ARN for authentication.
gg.eventhandler.iceberg.awsS3Endpoint Optional String value. None AWS S3 endpoint.
gg.eventhandler.iceberg.proxyServer Optional String value. None Proxy server to connect to the AWS S3 object storage.
gg.eventhandler.iceberg.proxyPort Optional String value. 80 Proxy server port to connect to the AWS S3 object storage.
gg.eventhandler.iceberg.awsS3PathStyleAccess Optional true or false. None Enables path-style access for S3-compatible storage, for example, domain.com/bucket instead of virtual-hosted style like bucket.domain.com). This is typically required when using non-AWS S3-compatible storage, such as Dell ECS or MinIO.
Classpath And Dependencies

The Java classpath (gg.classpath) should include the following dependencies:

Sample Configuration for Nessie Catalog and AWS S3 s3:// Scheme
gg.target=iceberg
gg.eventhandler.iceberg.warehouseLocation=/path/to/iceberg/tables
gg.classpath=DependencyDownloader/dependencies/iceberg-aws-java-sdk/*:DependencyDownloader/dependencies/iceberg-common/*
gg.eventhandler.iceberg.catalogType=nessie
gg.eventhandler.iceberg.nessieBranch=main
gg.eventhandler.iceberg.nessieUri=http://<nessie-server>:10001/api/v2
gg.eventhandler.iceberg.fileSystemScheme=s3://
gg.eventhandler.iceberg.awsS3Region=us-east-2
gg.eventhandler.iceberg.awsS3Bucket=<s3-bucket>
gg.eventhandler.iceberg.awsAccessKeyId=<access-key-id>
gg.eventhandler.iceberg.awsSecretKey=<secret-key>
gg.eventhandler.iceberg.proxyServer=<proxy-server>
gg.eventhandler.iceberg.proxyPort=<proxy-port>

Configuration for Nessie Catalog and AWS S3 s3a:// Scheme

The following are the configuration properties for the Nessie catalog and AWS S3 object store using s3a:// scheme:

Properties Required/Optional Legal Values Default Explanation
gg.eventhandler.iceberg.catalogType Optional String value. hadoop nessie.
gg.eventhandler.iceberg.nessieBranch Optional String value. main Nessie Catalog branch name where the Iceberg table metadata exists.
gg.eventhandler.iceberg.nessieUri Required String value. None Nessie Catalog endpoint URI. Example: http://<nessie-server>.com:10001/api/v2.
gg.eventhandler.iceberg.fileSystemScheme Optional String value. file:// File system scheme to indicate AWS S3 object storage location: s3a://.
gg.eventhandler.iceberg.awsS3Bucket Required String value. None AWS S3 bucket name that houses the Iceberg Warehouse.
gg.eventhandler.iceberg.awsAccessKeyId Required String value. None AWS access key id for authentication.
gg.eventhandler.iceberg.awsSecretKey Required String value. None AWS secret access key for authentication.
gg.eventhandler.iceberg.awsSessionToken Optional String value. None AWS session token for authentication.
gg.eventhandler.iceberg.proxyServer Optional String value. None Proxy server to connect to the AWS S3 object storage.
gg.eventhandler.iceberg.proxyPort Optional String value. 80 Proxy server port to connect to the AWS S3 object storage.
Classpath and Dependencies

The Java classpath (gg.classpath) should include the following dependencies:

Sample Configuration for Nessie Catalog and AWS S3 s3a:// scheme
gg.target=iceberg
gg.eventhandler.iceberg.warehouseLocation=/path/to/iceberg/tables
gg.classpath=DependencyDownloader/dependencies/iceberg-hadoop-aws/*:DependencyDownloader/dependencies/iceberg-common/*
gg.eventhandler.iceberg.catalogType=nessie
gg.eventhandler.iceberg.nessieBranch=main
gg.eventhandler.iceberg.nessieUri=http://<nessie-server>:10001/api/v2
gg.eventhandler.iceberg.fileSystemScheme=s3a://
gg.eventhandler.iceberg.awsS3Region=us-east-2
gg.eventhandler.iceberg.awsS3Bucket=<s3-bucket>
gg.eventhandler.iceberg.awsAccessKeyId=<access-key-id>
gg.eventhandler.iceberg.awsSecretKey=<secret-key>
gg.eventhandler.iceberg.proxyServer=<proxy-server>
gg.eventhandler.iceberg.proxyPort=<proxy-port>

Configuration for Nessie Catalog and GCS gs:// Scheme

The following are the configuration properties for the Nessie catalog and GCS object store using gs:// scheme:

Properties Required/Optional Legal Values Default Explanation
gg.eventhandler.iceberg.catalogType Optional String value. hadoop nessie.
gg.eventhandler.iceberg.nessieBranch Optional String value. main Nessie Catalog branch name where the Iceberg table metadata exists.
gg.eventhandler.iceberg.nessieUri Required String value. None Nessie Catalog endpoint URI. Example: http://<nessie-server>.com:10001/api/v2.
gg.eventhandler.iceberg.fileSystemScheme Optional String value. file:// File system scheme to indicate GCS object storage location: gs://.
gg.eventhandler.iceberg.gcpStorageBucket Required String value. None Google Cloud Storage bucket name that houses the Iceberg Warehouse.
gg.eventhandler.iceberg.gcpProjectId Required String value. None Sets the project-id of the Google Cloud project that houses the GCS bucket.
gg.eventhandler.iceberg.gcpServiceAccountJsonKeyFile Required String value. None Sets the path to the Google Service account key file.
gg.eventhandler.iceberg.proxyServer Optional String value. None Proxy server to connect to the GCS object storage.
gg.eventhandler.iceberg.proxyPort Optional String value. 80 Proxy server port to connect to the GCS object storage.
Classpath and Dependencies

The Java classpath (gg.classpath) should include the following dependencies:

Sample Configuration for Nessie Catalog and GCS gs:// Scheme
gg.target=iceberg
gg.eventhandler.iceberg.warehouseLocation=/path/to/iceberg/tables
gg.classpath=DependencyDownloader/dependencies/iceberg-hadoop-gcs/*:DependencyDownloader/dependencies/iceberg-common/*
gg.eventhandler.iceberg.catalogType=nessie
gg.eventhandler.iceberg.nessieBranch=main
gg.eventhandler.iceberg.nessieUri=http://<nessie-server>:10001/api/v2
gg.eventhandler.iceberg.fileSystemScheme=gs://
gg.eventhandler.iceberg.gcpStorageBucket=<gcs-bucket>
gg.eventhandler.iceberg.gcpProjectId=<gcp-project-id>
gg.eventhandler.iceberg.gcpServiceAccountJsonKeyFile=<gcp-service-account-key-file>
gg.eventhandler.iceberg.proxyServer=<proxy-server>
gg.eventhandler.iceberg.proxyPort=<proxy-port>

Configuration for Nessie Catalog and Azure Data Lake Storage abfss:// Scheme

The following are the configuration properties for the Nessie catalog and Azure Data Lake Storage using abfss:// scheme:

Properties Required/Optional Legal Values Default Explanation
gg.eventhandler.iceberg.catalogType Optional String value. hadoop nessie.
gg.eventhandler.iceberg.nessieBranch Optional String value. main Nessie Catalog branch name where the Iceberg table metadata exists.
gg.eventhandler.iceberg.nessieUri Required String value. None Nessie Catalog endpoint URI. Example:http://<nessie-server>.com:10001/api/v2.
gg.eventhandler.iceberg.fileSystemScheme Optional String value. file:// File system scheme to indicate Azure Data Lake Storage location: abfss://.
gg.eventhandler.iceberg.azureAccountName Required String value. None Azure storage account name that contains the container for the Iceberg Warehouse.
gg.eventhandler.iceberg.azureContainer Required String value. None Azure storage account container name that houses the Iceberg Warehouse.
gg.eventhandler.iceberg.azureAccountKey Required String value. None Azure storage account key.
gg.eventhandler.iceberg.azureBlobEndpoint Optional String value. <azureContainer>@<azureAccountName>.dfs.core.windows.net Azure Storage service endpoint.
gg.eventhandler.iceberg.proxyServer Optional String value. None Proxy server to connect to the Azure object storage.
gg.eventhandler.iceberg.proxyPort Optional String value. 80 Proxy server port to connect to the Azure object storage.
gg.eventhandler.iceberg.azureTenantId Optional String value. None Azure Tenant ID. Required only for EntraID auth (gg.eventhandler.iceberg.azureAccountKey has to be unset).
gg.eventhandler.iceberg.azureClientId Optional String value. None Azure EntraID Client ID. Required only for EntraID auth (gg.eventhandler.iceberg.azureAccountKey has to be unset)
gg.eventhandler.iceberg.azureClientSecret Optional String value. None Azure EntraID client secret. Required only for EntraID auth (gg.eventhandler.iceberg.azureAccountKey) has to be unset)
Classpath and Dependencies

The Java classpath (gg.classpath) should include the following dependencies:

Sample Configuration for Nessie Catalog and ADLS abfss:// Scheme
gg.target=iceberg
gg.eventhandler.iceberg.warehouseLocation=/path/to/iceberg/tables
gg.classpath=DependencyDownloader/dependencies/iceberg-hadoop-azure/*:DependencyDownloader/dependencies/iceberg-common/*
gg.eventhandler.iceberg.catalogType=nessie
gg.eventhandler.iceberg.nessieBranch=main
gg.eventhandler.iceberg.nessieUri=http://<nessie-server>:10001/api/v2
gg.eventhandler.iceberg.fileSystemScheme=abfss://
gg.eventhandler.iceberg.azureAccountName=<azure-storage-account-name>
gg.eventhandler.iceberg.azureContainer=<azure-storage-container>
gg.eventhandler.iceberg.azureAccountKey=<azure-storage-account-key>
gg.eventhandler.iceberg.proxyServer=<proxy-server>
gg.eventhandler.iceberg.proxyPort=<proxy-port>
Nessie Namespace

Nessie namespace is the top-level container for all the tables in the Nessie catalog.

Before starting the Replicat process, it is required to have existing namespaces before creating or writing to tables.

Nessie namespace can be created using the nessie command line program (nessie-cli-<version>.jar) as follows: create namespace QASOURCE;

The Nessie namespace is mapped to the GoldenGate schema in the MAP statement.

For example: MAP QASOURCE.TCUSTMER, TARGET QASOURCE.TCUSTMER;

Configuration for Iceberg AWS Glue Catalog

Configuration for Iceberg AWS Glue Catalog and AWS S3 s3:// OR s3a:// Scheme

The following are the configuration properties for the AWS Glue catalog and AWS S3 object store using s3:// or s3a:// scheme:

Properties Required/Optional Legal Values Default Explanation
gg.eventhandler.iceberg.catalogType Optional String value. hadoop glue.
gg.eventhandler.iceberg.awsGlueId Required String value. None The Glue catalog ID is your numeric AWS account ID.
gg.eventhandler.iceberg.fileSystemScheme Optional String value. file:// File system scheme to indicate AWS S3 object storage location: s3:// or s3a://.
gg.eventhandler.iceberg.awsS3Region Required String value. None AWS S3 bucket region. Example: us-east-2.
gg.eventhandler.iceberg.awsS3Bucket Required String value. None AWS S3 bucket name that houses the Iceberg Warehouse.
gg.eventhandler.iceberg.awsAccessKeyId Optional String value. None AWS access key id for authentication.
gg.eventhandler.iceberg.awsSecretKey Optional String value. None AWS secret access key for authentication.
gg.eventhandler.iceberg.awsSessionToken Optional String value. None AWS session token for authentication.
gg.eventhandler.iceberg.awsRoleArn Optional String value. None AWS role ARN for authentication.
gg.eventhandler.iceberg.awsS3Endpoint Optional String value. None AWS S3 endpoint.
gg.eventhandler.iceberg.proxyServer Optional String value. None Proxy server to connect to the AWS S3 object storage.
gg.eventhandler.iceberg.proxyPort Optional String Value. 80 Proxy server port to connect to the AWS S3 object storage.
gg.eventhandler.iceberg.awsS3PathStyleAccess Optional true or false None Enables path-style access for S3-compatible storage (for example, domain.com/bucket instead of virtual-hosted style like bucket.domain.com). This is typically required when using non-AWS S3-compatible storage, such as Dell ECS or MinIO.

Classpath and Dependencies

The Java classpath (gg.classpath) should include the following dependencies:

Sample Configuration for Iceberg AWS Glue Catalog and AWS S3 s3:// or s3a:// Scheme

gg.target=iceberg
gg.eventhandler.iceberg.warehouseLocation=/path/to/iceberg/tables
gg.classpath=DependencyDownloader/dependencies/iceberg-aws-java-sdk/*:DependencyDownloader/dependencies/iceberg-common/*
gg.eventhandler.iceberg.catalogType=glue
gg.eventhandler.iceberg.awsGlueId=<aws-acccount-id>
gg.eventhandler.iceberg.fileSystemScheme=s3://
#gg.eventhandler.iceberg.fileSystemScheme=s3a://
gg.eventhandler.iceberg.awsS3Region=us-east-2
gg.eventhandler.iceberg.awsS3Bucket=<s3-bucket>
gg.eventhandler.iceberg.awsAccessKeyId=<access-key-id>
gg.eventhandler.iceberg.awsSecretKey=<secret-key>
gg.eventhandler.iceberg.proxyServer=<proxy-server>
gg.eventhandler.iceberg.proxyPort=<proxy-port>

Table Names and Case Sensitivity

AWS Glue catalog supports only lower case names.

AWS Glue catalog supports only two-part table names.

The target table in the GGDAA Replicat MAP statement should be mapped to the Glue database and table names.

Example: MAP QASOURCE.TCUSTMER, TARGET "glue_database"."tcustmer";

In this example, glue_database is the Glue database name and tcustmer is the Glue table name.

Configuration for Iceberg Polaris Catalog

Apache Polaris is an open-source, fully-featured catalog for Apache Iceberg.

There are a few options to setup Polaris:

This topic contains the following:

Polaris Common Configuration

The following are the configuration properties for the Polaris catalog:

Properties Required/Optional Legal Values Default Explanation
gg.eventhandler.iceberg.catalogType Required String value. hadoop polaris.
gg.eventhandler.iceberg.polarisCatalogUri Required String value. None Polaris Catalog endpoint URI. Example: https://<polaris-account>.snowflakecomputing.com/polaris/api/catalog.
gg.eventhandler.iceberg.polarisCatalogName Required String value. None Polaris Catalog name. Catalog name is the entry point to the Polaris catalog namespace and tables.
gg.eventhandler.iceberg.polarisClientId Required String value. None Polaris principal’s client ID used for authentication and authorization to the respective Polaris catalog.
gg.eventhandler.iceberg.polarisClientSecret Required String value. None Polaris principal’s client secret used for authentication and authorization to the respective Polaris catalog.
gg.eventhandler.iceberg.polarisPrincipalRole Optional String value. ALL The role to be assumed by the Polaris principal.

Polaris Catalog with Google Cloud Storage (GCS)

The environment variable GOOGLE_APPLICATION_CREDENTIALS must be set to the path to the Google Service account key file. Add the following to the Replicat parameter file (.prm):

SETENV (GOOGLE_APPLICATION_CREDENTIALS = "/path/to/the/gcp-service-account-json-key.json")

Polaris Catalog with AWS S3 Storage

Properties Required/Optional Legal Values Default Explanation
gg.eventhandler.iceberg.awsS3Region Required String value. None Required only if the Polaris catalog points to AWS S3 Storage. AWS S3 bucket region. Example: us-east-2.
gg.eventhandler.iceberg.fileSystemScheme Optional String value. file:// Required only if the Polaris catalog points to AWS S3 Storage. File system scheme to indicate AWS S3 object storage location: s3://.
gg.eventhandler.iceberg.awsAccessKeyId Optional String value. None Required only if the Polaris catalog points to AWS S3 Storage. AWS access key id for authentication.
gg.eventhandler.iceberg.awsSecretKey Optional String value. None Required only if the Polaris catalog points to AWS S3 Storage. AWS secret access key for authentication.
gg.eventhandler.iceberg.awsSessionToken Optional String value. None Required only if the Polaris catalog points to AWS S3 Storage. AWS session token for authentication.
gg.eventhandler.iceberg.awsS3Endpoint Optional String value. None Required only if the Polaris catalog points to AWS S3 Storage. AWS S3 endpoint.
gg.eventhandler.iceberg.awsS3PathStyleAccess Optional true or false None Enables path-style access for S3-compatible storage (for example, domain.com/bucket instead of virtual-hosted style like bucket.domain.com). This is typically required when using non-AWS but an S3-compatible storage, such as Dell ECS or MinIO.

Polaris Catalog with Azure Data Lake Storage (ADLS)

Properties Required/Optional Legal Values Default Explanation
gg.eventhandler.iceberg.fileSystemScheme Optional String value. file:// Required only if the Polaris catalog points to Azure Data Lake Storage. Warehouse scheme to indicate Azure Data Lake Storage location: abfss://.
gg.eventhandler.iceberg.azureAccountName Required String value. None Required only if the Polaris catalog points to Azure Data Lake Storage. Azure storage account name that contains the container for the Iceberg Warehouse.
gg.eventhandler.iceberg.azureContainer Required String value. None Required only if the Polaris catalog points to Azure Data Lake Storage. Azure storage account container name that houses the Iceberg Warehouse.
gg.eventhandler.iceberg.azureAccountKey Required String value. None Required only if the Polaris catalog points to Azure Data Lake Storage. Azure storage account key.
gg.eventhandler.iceberg.azureBlobEndpoint Optional String value. <azureContainer>@<azureAccountName>.dfs.core.windows.net Required only if the Polaris catalog points to Azure Data Lake Storage. Azure Storage service endpoint.
gg.eventhandler.iceberg.azureTenantId Optional String value. None Azure Tenant ID. Required only for EntraID auth (gg.eventhandler.iceberg.azureAccountKey has to be unset).
gg.eventhandler.iceberg.azureClientId Optional String value. None Azure EntraID Client ID. Required only for EntraID auth (gg.eventhandler.iceberg.azureAccountKey has to be unset)
gg.eventhandler.iceberg.azureClientSecret Optional String value. None Azure EntraID client secret. Required only for EntraID auth (gg.eventhandler.iceberg.azureAccountKey) has to be unset)

Polaris Catalog and GCS Storage Classpath And Dependencies

If Polaris catalog is setup to write to GCS, then the Java classpath (gg.classpath) should include the following dependencies:

Polaris Catalog and AWS S3 storage Classpath and Dependencies

If Polaris catalog is setup to write to AWS S3, then the Java classpath (gg.classpath) should include the following dependencies:

Polaris Catalog and ADLS storage Classpath And Dependencies

If Polaris catalog is setup to write to ADLS, then the Java classpath (gg.classpath) should include the following dependencies:

Sample Configuration for Polaris Catalog

gg.target=iceberg
#For catalog using GCS
gg.classpath=DependencyDownloader/dependencies/iceberg-gcs-java-sdk/*:DependencyDownloader/dependencies/iceberg-common/*
#For catalog using S3
#gg.classpath=DependencyDownloader/dependencies/iceberg-aws-java-sdk/*:DependencyDownloader/dependencies/iceberg-common/*
#For catalog using ADLS
#gg.classpath=DependencyDownloader/dependencies/iceberg-hadoop-azure/*:DependencyDownloader/dependencies/iceberg-common/*
gg.eventhandler.iceberg.catalogType=polaris
gg.eventhandler.iceberg.polarisCatalogUri=https://<polaris-account>.snowflakecomputing.com/polaris/api/catalog
gg.eventhandler.iceberg.polarisCatalogName=<polaris_gcs_catalog>
gg.eventhandler.iceberg.polarisClientId=<clientId>
gg.eventhandler.iceberg.polarisClientSecret=<clientSecret>
gg.eventhandler.iceberg.polarisPrincipalRole=ALL

Polaris Namespace

Polaris namespace is the top-level container for all the tables in the Polaris catalog.

Before starting the Replicat process, the Polaris namespace should be created in the respective Polaris catalog.

The Polaris namespace is mapped to the GoldenGate schema in the MAP statement.

Example: MAP QASOURCE.TCUSTMER, TARGET "polaris_namespace"."tcustmer";

Configuration for Iceberg REST Catalog

Iceberg defines a REST specification (https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml) for catalog implementations.

Any REST server that implements the Iceberg REST API can be used as the Iceberg catalog.

For example, Polaris is an implementation of the Iceberg REST API.

Configuration for Iceberg REST Catalog

The following are the configuration properties for the Polaris catalog:

Properties Required/Optional Legal Values Default Explanation
gg.eventhandler.iceberg.catalogType Required String value. hadoop rest.
gg.eventhandler.iceberg.restCatalogUri Required String value. None REST Catalog endpoint URI. Example:https://<polaris-account>.snowflakecomputing.com/polaris/api/catalog.
gg.eventhandler.iceberg.restCatalogProperties Optional String value. None Properties file with additional configuration for the REST catalog.

Sample Configuration for REST Catalog based on Polaris

gg.target=iceberg
#For catalog using GCS
gg.classpath=DependencyDownloader/dependencies/iceberg-gcs-java-sdk/*:DependencyDownloader/dependencies/iceberg-common/*
#For catalog using S3
#gg.classpath=DependencyDownloader/dependencies/iceberg-s3/*:DependencyDownloader/dependencies/iceberg-common/*
#For catalog using ADLS
#gg.classpath=DependencyDownloader/dependencies/iceberg-hadoop-azure/*:DependencyDownloader/dependencies/iceberg-common/*
gg.eventhandler.iceberg.catalogType=rest
gg.eventhandler.iceberg.restCatalogUri=https://<polaris-account>.snowflakecomputing.com/polaris/api/catalog
gg.eventhandler.iceberg.restCatalogProperties=/path/to/rest/catalog.properties
# Optional configuration for authentication to the object storage.
# Some REST implementations do not require a separate authentication to the storage layer.
#gg.eventhandler.iceberg.fileSystemScheme=s3://
#gg.eventhandler.iceberg.awsS3Region=<s3-region>
#gg.eventhandler.iceberg.awsS3Bucket=<s3-bucket>
#gg.eventhandler.iceberg.awsAccessKeyId=<access-key-id>
#gg.eventhandler.iceberg.awsSecretKey=<secret-key>
#gg.eventhandler.iceberg.fileSystemScheme=abfss://
#gg.eventhandler.iceberg.azureAccountName=<azure-storage-account-name>
#gg.eventhandler.iceberg.azureContainer=<azure-storage-container>
#gg.eventhandler.iceberg.azureAccountKey=<azure-storage-account-key>
#gg.eventhandler.iceberg.fileSystemScheme=gs://
#gg.eventhandler.iceberg.gcpStorageBucket=<gcs-bucket>
#gg.eventhandler.iceberg.gcpProjectId=<gcp-project-id>
#gg.eventhandler.iceberg.gcpServiceAccountJsonKeyFile=<gcp-service-account-key-file>

Sample Rest Catalog Properties file (For Polaris)

warehouse=polaris_s3_catalog
credential=<ClientId>:<ClientSecret>
scope=PRINCIPAL_ROLE:ALL
token-refresh-enabled=true

Configuration and Table Names for REST Catalog based on AWS S3 Tables properties

gg.target=iceberg
gg.classpath=DependencyDownloader/dependencies/iceberg-aws-java-sdk/*:DependencyDownloader/dependencies/iceberg-common/*
gg.eventhandler.iceberg.catalogType=rest
gg.eventhandler.iceberg.restCatalogUri=https://s3tables.<aws-region>.amazonaws.com/iceberg
gg.eventhandler.iceberg.restCatalogProperties=/path/to/rest/catalog.properties
gg.eventhandler.iceberg.fileSystemScheme=s3://
gg.eventhandler.iceberg.awsS3Region=<aws-region>
gg.eventhandler.iceberg.awsAccessKeyId=<access-key-id>
gg.eventhandler.iceberg.awsSecretKey=<secret-key>

Rest Catalog Properties

warehouse=arn:aws:s3tables:<aws-region>:<aws-account-id>:bucket/<s3-table-bucket>
rest.sigv4-enabled=true
rest.signing-name=s3tables
rest.signing-region=<aws-region>
AWS Identity and Access Management Permissions for S3 Tables

This section provides a list of the specific Identity and Access Management (IAM) permissions required whenever first-class S3 tables are introduced.

To enable S3 Tables functionality, ensure the following AWS IAM actions are allowed in the policy:

Sample configuration for REST Catalog based on IBM watsonx.data

gg.target=iceberg
#For catalog using ADLS
gg.classpath=DependencyDownloader/dependencies/iceberg-hadoop-azure/*:DependencyDownloader/dependencies/iceberg-common/*
#For catalog using GCS
#gg.classpath=DependencyDownloader/dependencies/iceberg-gcs-java-sdk/*:DependencyDownloader/dependencies/iceberg-common/*
#For catalog using S3
#gg.classpath=DependencyDownloader/dependencies/iceberg-s3/*:DependencyDownloader/dependencies/iceberg-common/*
gg.eventhandler.iceberg.catalogType=rest
gg.eventhandler.iceberg.restCatalogUri=https://<watsonx-uri>/mds/iceberg
gg.eventhandler.iceberg.restCatalogProperties=/path/to/rest/watsonx-data-catalog.properties
# Optional configuration for authentication to the object storage.
gg.eventhandler.iceberg.fileSystemScheme=abfss://
gg.eventhandler.iceberg.azureAccountName=<azure-storage-account-name>
gg.eventhandler.iceberg.azureContainer=<azure-storage-container>
gg.eventhandler.iceberg.azureAccountKey=<azure-storage-account-key>
#gg.eventhandler.iceberg.fileSystemScheme=s3://
#gg.eventhandler.iceberg.awsS3Region=<s3-region>
#gg.eventhandler.iceberg.awsS3Bucket=<s3-bucket>
#gg.eventhandler.iceberg.awsAccessKeyId=<access-key-id>
#gg.eventhandler.iceberg.awsSecretKey=<secret-key>
#gg.eventhandler.iceberg.fileSystemScheme=gs://
#gg.eventhandler.iceberg.gcpStorageBucket=<gcs-bucket>
#gg.eventhandler.iceberg.gcpProjectId=<gcp-project-id>
#gg.eventhandler.iceberg.gcpServiceAccountJsonKeyFile=<gcp-service-account-key-file>

Sample Rest Catalog Properties file (For IBM watsonx.data)

warehouse=<wxd-catalog-name> # Replace with actual values
header.Authorization=Basic <Base64 encoded <username:password>> # Replace with actual values

Standalone Utility to Test Connection to Apache Iceberg REST Catalog

Oracle GoldenGate for DAA is compatible with any Apache Iceberg REST catalog implementation. For custom REST catalog implementations, Oracle recommends to test the REST catalog connectivity before configuration of the Replicat group. Note: Oracle has a standalone utility to investigate catalog connection issues. You can request access to this utility by using a service request.

To install the standalone utility perform the following steps:

  1. Download the standalone program ggicebergcatalogutil-<version>.jar.

  2. Ensure that you have Java in the class path; Java run time version should be >= 11.

  3. Run the program as follows:

    java -jar ggicebergcatalogutil-<version>.jar

    The following message gets displayed:

    Usage: Provide two arguments: <namespace> '</path/to/iceberg-rest-catalog.properties>'

    Sample property file for Polaris REST catalog

    warehouse=<polaris-catalog-name> # Replace with actual polaris catalog name
uri=https://<polaris-account>.snowflakecomputing.com/polaris/api/catalog # Replace with actual uri
credential=<polaris-client-id>:<polaris-client-secret> # Replace with actual credentials
scope=PRINCIPAL_ROLE:ALL # Replace with actual scope
token-refresh-enabled=true

    Sample property file for IBM Watsonx REST catalog

     uri=https://<watsonx-uri>/mds/iceberg # Replace with actual uri
 warehouse=<wxd-catalog-name> # Replace with actual values
 header.Authorization=Basic <Base64 encoded <username:password>> # Replace with actual values

    Sample property file for any custom REST catalog

    # Add properties required for the custom REST Catalog

  4. Create a catalog properties file for the REST catalog and run the following command:

java -jar ggicebergcatalogutil-<version>.jar <namespace> '/path/to/iceberg-catalog.properties'

  1. If the REST catalog properties and the namespace names are valid and the connection is successful, then the output should list tables in the namespace.

    Example

    Tables in namespace {QASOURCE}:
       - {QASOURCE.P_S3_SV_TCUSTMER} - {QASOURCE.P_S3_SV_TCUSTORD} -  {QASOURCE.TCUSTMER} - {QASOURCE.TCUSTORD}

Configuration for Iceberg Hadoop Catalog

Hadoop catalog is not recommended for production usage as it has no reliable locking mechanism and would impact concurrent reads and writes.

Hadoop catalog is used for testing purposes only.

Configuration for Iceberg Hadoop Catalog and file:// Scheme

The following are the configuration properties for the Hadoop catalog and the local file system as the Iceberg storage using file:// scheme:

Properties Required/Optional Legal Values Default Explanation
gg.eventhandler.iceberg.catalogType Optional String value. hadoop hadoop.
gg.eventhandler.iceberg.fileSystemScheme Optional String value. file:// File system scheme to indicate local file system as the storage: file://.
gg.eventhandler.iceberg.warehouseLocation Required String value. None Local directory path to the Iceberg warehouse.

Note: This configuration is typically used for testing purposes for storing the Iceberg tables on the local file system.

Classpath and Dependencies

The Java classpath (gg.classpath) should include the following dependencies:

Sample Configuration for Iceberg Hadoop Catalog and Local File Storage file:// Scheme
gg.target=iceberg
gg.eventhandler.iceberg.warehouseLocation=/path/to/iceberg/tables
gg.classpath=DependencyDownloader/dependencies/iceberg-common/*

Configuration for Iceberg Hadoop Catalog and s3a:// Scheme

The following are the configuration properties for the Hadoop catalog and AWS S3 object store using s3a:// scheme:

Properties Required/Optional Legal Values Default Explanation
gg.eventhandler.iceberg.catalogType Optional String value. hadoop hadoop.
gg.eventhandler.iceberg.fileSystemScheme Optional String value. file:// File system scheme to indicate AWS S3 object storage location: s3a://.
gg.eventhandler.iceberg.awsS3Bucket Required String value. None AWS S3 bucket name that houses the Iceberg Warehouse.
gg.eventhandler.iceberg.awsAccessKeyId Required String value. None AWS access key id for authentication.
gg.eventhandler.iceberg.awsSecretKey Required String value. None AWS secret access key for authentication.
gg.eventhandler.iceberg.awsSessionToken Optional String value. None AWS session token for authentication.
gg.eventhandler.iceberg.proxyServer Optional String value. None Proxy server to connect to the AWS S3 object storage.
gg.eventhandler.iceberg.proxyPort Optional String value. 80 Proxy server port to connect to the AWS S3 object storage.
gg.eventhandler.iceberg.awsS3Endpoint Optional Any valid URL string. None The AWS S3 endpoint URL. This is typically required when using non-AWS S3-compatible storage, such as Dell ECS or MinIO.
gg.eventhandler.iceberg.awsS3PathStyleAccess Optional true or false None Enables path-style access for S3-compatible storage (for example, domain.com/bucket instead of virtual-hosted style like bucket.domain.com). This is typically required when using non-AWS S3-compatible storage, such as Dell ECS or MinIO.
Classpath and Dependencies

The Java classpath (gg.classpath) should include the following dependencies:

Sample Configuration for Hadoop Catalog and AWS S3 s3a:// Scheme
gg.target=iceberg
gg.eventhandler.iceberg.warehouseLocation=/path/to/iceberg/tables
gg.classpath=DependencyDownloader/dependencies/iceberg-hadoop-aws/*:DependencyDownloader/dependencies/iceberg-common/
gg.eventhandler.iceberg.catalogType=hadoop
gg.eventhandler.iceberg.fileSystemScheme=s3a://
gg.eventhandler.iceberg.awsS3Region=us-east-2
gg.eventhandler.iceberg.awsS3Bucket=<s3-bucket>
gg.eventhandler.iceberg.awsAccessKeyId=<access-key-id>
gg.eventhandler.iceberg.awsSecretKey=<secret-key>
gg.eventhandler.iceberg.proxyServer=<proxy-server>
gg.eventhandler.iceberg.proxyPort=<proxy-port>

Configuration for Iceberg Hadoop Catalog and gs:// Scheme

The following are the configuration properties for the Hadoop catalog and GCS object store using gs:// scheme:

Properties Required/Optional Legal Values Default Explanation
gg.eventhandler.iceberg.catalogType Optional String value. hadoop hadoop.
gg.eventhandler.iceberg.fileSystemScheme Optional String value. file:// File system scheme to indicate GCS object storage location: gs://.
gg.eventhandler.iceberg.gcpStorageBucket Required String value. None Google Cloud Storage bucket name that houses the Iceberg Warehouse.
gg.eventhandler.iceberg.gcpProjectId Required String value. None Sets the project-id of the Google Cloud project that houses the GCS bucket.
gg.eventhandler.iceberg.gcpServiceAccountJsonKeyFile Required String value. None Sets the path to the Google Service account key file.
gg.eventhandler.iceberg.proxyServer Optional String value. None Proxy server to connect to the GCS object storage.
gg.eventhandler.iceberg.proxyPort Optional String value. 80 Proxy server port to connect to the GCS object storage.
Classpath and Dependencies

The Java classpath (gg.classpath) should include the following dependencies:

Sample Configuration for Hadoop Catalog and GCS gs:// Scheme
gg.target=iceberg
gg.eventhandler.iceberg.warehouseLocation=/path/to/iceberg/tables
gg.classpath=DependencyDownloader/dependencies/iceberg-hadoop-gcs/*:DependencyDownloader/dependencies/iceberg-common/*
gg.eventhandler.iceberg.catalogType=hadoop
gg.eventhandler.iceberg.fileSystemScheme=gs://
gg.eventhandler.iceberg.gcpStorageBucket=<gcs-bucket>
gg.eventhandler.iceberg.gcpProjectId=<gcp-project-id>
gg.eventhandler.iceberg.gcpServiceAccountJsonKeyFile=<gcp-service-account-key-file>
gg.eventhandler.iceberg.proxyServer=<proxy-server>
gg.eventhandler.iceberg.proxyPort=<proxy-port>

Configuration for Iceberg Hadoop Catalog and abfss:// Scheme

The following are the configuration properties for the Hadoop catalog and Azure Data Lake Storage using abfss:// scheme:

Properties Required/Optional Legal Values Default Explanation
gg.eventhandler.iceberg.catalogType Optional String value hadoop hadoop.
gg.eventhandler.iceberg.fileSystemScheme Optional String value file:// File system scheme to indicate Azure Data Lake Storage location:abfss://.
gg.eventhandler.iceberg.azureAccountName Required String value None Azure storage account name that contains the container for the Iceberg Warehouse.
gg.eventhandler.iceberg.azureContainer Required String value None Azure storage account container name that houses the Iceberg Warehouse.
gg.eventhandler.iceberg.azureAccountKey Required String value. None Azure storage account key.
gg.eventhandler.iceberg.azureBlobEndpoint Optional String value. \ Azure Storage service endpoint.
gg.eventhandler.iceberg.proxyServer Optional String value. None Proxy server to connect to the Azure object storage.
gg.eventhandler.iceberg.proxyPort Optional String value. 80 Proxy server port to connect to the Azure object storage.
gg.eventhandler.iceberg.azureTenantId Optional String value. None Azure Tenant ID. Required only for EntraID auth (gg.eventhandler.iceberg.azureAccountKey has to be unset).
gg.eventhandler.iceberg.azureClientId Optional String value. None Azure EntraID Client ID. Required only for EntraID auth (gg.eventhandler.iceberg.azureAccountKey has to be unset)
gg.eventhandler.iceberg.azureClientSecret Optional String value. None Azure EntraID client secret. Required only for EntraID auth (gg.eventhandler.iceberg.azureAccountKey) has to be unset)
Classpath and Dependencies

The Java classpath (gg.classpath) should include the following dependencies:

Sample Configuration for Hadoop Catalog and ADLS abfss:// Scheme
gg.target=iceberg
gg.eventhandler.iceberg.warehouseLocation=/path/to/iceberg/tables
gg.classpath=DependencyDownloader/dependencies/iceberg-hadoop-azure/*:DependencyDownloader/dependencies/iceberg-common/*
gg.eventhandler.iceberg.catalogType=hadoop
gg.eventhandler.iceberg.fileSystemScheme=abfss://
gg.eventhandler.iceberg.azureAccountName=<azure-storage-account-name>
gg.eventhandler.iceberg.azureContainer=<azure-storage-container>
gg.eventhandler.iceberg.azureAccountKey=<azure-storage-account-key>
gg.eventhandler.iceberg.proxyServer=<proxy-server>
gg.eventhandler.iceberg.proxyPort=<proxy-port>

Configuration for Oracle AI Data Catalog

Oracle AI Data Catalog is Oracle’s fully featured REST catalog implementation for Apache Iceberg.

Apache Iceberg Oracle AI Data Catalog

The following are the configuration properties for the replicat.

Table 14 Apache Iceberg Oracle

Properties Required/Optional Legal Values Default Explanation
gg.eventhandler.iceberg.catalogType Required String value. oadc The iceberg catalog type.
gg.eventhandler.iceberg.oadcCatalogUri Required String value. None REST Catalog endpoint URI. Example: https://<catalog-uri>/catalog.
gg.eventhandler.iceberg.oadcCatalogProperties Optional String value. None Path to properties file with additional configuration for the catalog.
gg.eventhandler.iceberg.oadcAuthType Optional String value. oauth2 Catalog authentication type
gg.eventhandler.iceberg.oadcClientId Required when auth type = oauth2 String value. None Client ID of the DB that is related to the catalog.
gg.eventhandler.iceberg.oadcClientSecret Required when auth type = oauth2 String value. None Secret of the DB that is related to the catalog.
gg.eventhandler.iceberg.oadcTokenUri Required when auth type = oauth2 String value. None The Uri of the authorization entity.
gg.eventhandler.iceberg.oadcScope Optional when auth type = oauth2 String value. PRINCIPAL_ROLE:ALL The oauth2 scope.

The following are some of the Oracle AI Data Catalog configuration properties:

Table 15 Apache Iceberg Oracle

Properties Required/Optional Legal Values Default Explanation
credential Required String value None The catalog credentials in the form: username:password. |
oauth2-server-uri Optional String value v1/auth/token The catalog authentication uri
scope Optional String value PRINCIPAL_ROLE:ALL The catalog scope

Note: For a comprehensive list of all the catalog properties, refer to the Oracle AI Data Catalog official documentation.

Configuration for Oracle AI Data Catalog (OCI Object Storage)

Table 16 Properties for Oracle

Properties Required/Optional Legal Values Default Explanation
gg.eventhandler.iceberg.fileSystemScheme Optional String value None File system scheme for S3-compatible storage (typically s3:// or s3a://). Uses3a:// for Hadoop-compatible access.
gg.eventhandler.iceberg.awsS3Region Required String value None AWS S3 bucket region. Example: us-east-2.
gg.eventhandler.iceberg.awsS3Bucket Required String value None AWS S3 bucket name that houses the Iceberg Warehouse.
gg.eventhandler.iceberg.awsAccessKeyId Optional String value None AWS access key id for authentication.
gg.eventhandler.iceberg.awsSecretKey Optional String value None AWS secret access key for authentication.
gg.eventhandler.iceberg.awsS3Endpoint Optional String value None AWS S3 endpoint.

Properties

gg.target=iceberg
gg.eventhandler.iceberg.catalogType=oadc
gg.eventhandler.iceberg.oadcCatalogUri=https://<catalog-uri>/catalog
gg.eventhandler.iceberg.oadcCatalogProperties=/path/to/oadc/catalog.properties

Object Storage Properties

gg.eventhandler.iceberg.fileSystemScheme=s3://
gg.eventhandler.iceberg.awsS3Region=<aws-region>
gg.eventhandler.iceberg.awsS3Bucket=<aws-bucket>
gg.eventhandler.iceberg.awsAccessKeyId=<access-key>
gg.eventhandler.iceberg.awsSecretKey=<secret-key>
gg.eventhandler.iceberg.awsS3Endpoint=https://<s3-endpoint>

Configuration for Oracle AI Data Catalog (Azure Storage)

Table 17 Properties for AI Data Catalog based on Azure

Properties Required/Optional Legal Values Default Explanation
gg.eventhandler.iceberg.fileSystemScheme Required String value None File system scheme to indicate Azure
gg.eventhandler.iceberg.azureAccountName Required String value None Azure storage account name that contains
gg.eventhandler.iceberg.azureContainer Required String value None Azure storage account container name
gg.eventhandler.iceberg.azureAccountKey Required String value None Azure storage account key.

Properties

gg.target=iceberg
 gg.eventhandler.iceberg.catalogType=oadc
 gg.eventhandler.iceberg.oadcCatalogUri=https://<catalog-uri>/catalog
 gg.eventhandler.iceberg.oadcCatalogProperties=/path/to/oadc/catalog.properties

Object Storage Properties

gg.eventhandler.iceberg.fileSystemScheme=abfss://
gg.eventhandler.iceberg.azureAccountName=<azure-storage-account-name>
gg.eventhandler.iceberg.azureContainer=<azure-storage-container>
gg.eventhandler.iceberg.azureAccountKey=<azure-storage-account-key>

OADC Catalog Properties

credential=<client-id>:<client-secret>
scope=<scope-key:scope-value>

Testing the Connection to Iceberg Oracle AI Data Catalog

Oracle GoldenGate for Distributed Applications and Analytics (GG for DAA) supports any implementation of the Iceberg REST Catalog specification. Oracle recommends verifying connectivity to the REST catalog before configuring a Replicat group.

Note: Oracle provides a standalone utility to help diagnose Iceberg REST catalog connectivity issues. You can request access to this utility by opening a Service Request (SR).

Using the Standalone Utility

  1. Download the standalone utility:

    ggicebergcatalogutil-\<version>.jar

  2. Ensure that Java 11 or later is installed and available in your PATH.

  3. Run the utility without arguments to display the usage information:

    java -jar ggicebergcatalogutil-\<version>.jar

    The utility displays the following usage message:

    Usage:
java -jar ggicebergcatalogutil-\<version>.jar \<namespace> \<path-to-iceberg-rest-catalog.properties>

  4. Create a properties file for your Iceberg REST catalog.

    warehouse=\<catalog-name> uri=https://\<catalog-uri>/catalog\
    credential=\<client-id>:\<client-secret> scope=PRINCIPAL_ROLE:ALL\
    token-refresh-enabled=true

  5. Run the utility by specifying the namespace and the properties file:

    java -jar ggicebergcatalogutil-\<version>.jar \<namespace> /path/to/iceberg-rest-catalog.properties

  6. Verify the output.

    If the connection succeeds and the namespace exists, the utility lists the tables in the specified namespace.

    Example:

    Tables in namespace {QASOURCE}:

       - {QASOURCE.P_S3_SV_TCUSTMER}

       - {QASOURCE.P_S3_SV_TCUSTORD}

       - {QASOURCE.TCUSTMER}

       - {QASOURCE.TCUSTORD}


**Troubleshooting**
       - Verify that the `uri`, `warehouse`, `credential`, and `scope` properties are valid.

       - Ensure that the specified namespace exists in the Iceberg REST catalog.

       - Confirm that the client has network connectivity to the REST catalog endpoint.

       - Review any authentication or authorization errors returned by the utility before configuring the Replicat group.


## Configuration Templates {#GUID-AC4C3C23-917A-4969-A583-A3EA492EF67C}

Iceberg configuration templates are available in the directory `/path/to/AdapterExamples/bigdata/iceberg`.

The following template properties files are packaged with Oracle GoldenGate:


       - `iceberg-glue-s3.properties`

       - `iceberg-hadoop-adls.properties`

       - `iceberg-hadoop-gcs.properties`

       - `iceberg-hadoop-localfile.properties`

       - `iceberg-hadoop-s3.properties`

       - `iceberg-nessie-adls.properties`

       - `iceberg-nessie-gcs.properties`

       - `iceberg-nessie-s3.properties`

       - `iceberg-nessie-s3a.properties`

       - `iceberg-polaris-adls.properties`

       - `iceberg-polaris-gcs.properties`

       - `iceberg-polaris-s3.properties`

       - `iceberg-rest.properties`


## Limitations {#GUID-6CFDFB38-16B9-4BC8-8386-772E9E0938AA}


       - Oracle GoldenGate does not support configuration of partition columns during automatic table creation.

If partitioned tables are required, the Iceberg table should be created manually with the required partition columns.

       - Altering the partitioning schema of a table is not supported after starting the Replication process.

If the partitioning schema of a table needs to be changed, the table should be dropped and recreated manually in the target database.

The data in the table will need to be reloaded.


**Note:**  Contact Oracle Support for assistance with this process.

  1. Wait until the initial load Replicat group has completed apply of the initial load trail files.
  2. Stop the change synchronization group for Extract.
  3. Configure a change synchronization Replicat group.
  4. Add the parameter UPDATEINSERTS to the change synchronization Replicat group.
  5. Start the change synchronization Replicat group.
  6. Wait until the change synchronization Replicat group has processed all the trails generated by change synchronization Extract group. The last record's end offset in the last trail file must match the targetCheckpoint value in the JSON checkpoint file of the change synchronization Replicat group. Example:

    • Run ls -l on the last trail file.</code></pre>

      -rw-r–r– 1 username dba 5660 Feb 22 2024 /path/to/trail/tr000000003

      
     - Here the last record's end offset is `5660`, and the trail sequence is `3`.

     - Open JSON checkpoint file for the change synchronization Replicat group
This should have the following attribute:

      “targetCheckpoint” : { “trailSequence” : 3, “trailOffset” : 5660 }

      This `targetCheckpoint` must match the last record's end offset.
  7. Shutdown change synchronization Replicat group and remove the parameter UPDATEINSERTS.
  8. Initial load is complete now. Start change synchronization Extract and Replicat groups.

    Iceberg Change Synchronization Replicat Behavior During Instantiation

    • Execute [DELETE+INSERT] for all the INSERT operations, irrespective of whether the base row exists on the target or not.
    • Run [DELETE+INSERT] for all the UPDATE operations, irrespective of whether the base row exists on the target or not.
    • Run DELETE for all the DELETE operations, irrespective of whether the base row exists on the target or not. Note: No collisions will be logged in the Iceberg Replicat report file.

Configuration for Third-Party S3-Compatible Storage

Third-party S3-compatible storage refers to object storage systems that emulate the AWS S3 API but are provided by vendors other than AWS, such as Dell ECS or MinIO. These can be used as the backing storage for Iceberg tables in Oracle GoldenGate, often in on-premises, hybrid, or custom cloud environments. To access third-party S3-compatible storage, you typically need to configure the gg.eventhandler.iceberg.awsS3Endpoint property to point to your storage endpoint (for example, a custom URL such as, https://your-ecs-host:9021). Additionally, enable path-style access by setting gg.eventhandler.iceberg.awsS3PathStyleAccess to true, as virtual-hosted style (for example, bucket.domain.com) is often not supported by non-AWS systems. These properties can be used with various Iceberg catalogs (for example, Hadoop, Nessie, or REST) and integrate with authentication mechanisms like access keys. Note: This configuration is typically required for non-AWS storage to avoid DNS resolution issues (for example, UnknownHostException). Test connectivity before deployment, and ensure your storage system fully supports the S3 API for Iceberg operations.



**Note:**  Before you migrate to Version 3, verify that your query engine is compatible with the underlying Iceberg library. You cannot downgrade from Version 3 to an earlier version; this upgrade is one-way.
{: .infoboxnote}



- [New Datatypes Supported in V3](apache-iceberg.html#GUID-1F7AC30B-B70B-441E-83BE-63462D8EE335)

- [Limitations](apache-iceberg.html#GUID-16E4AECB-E178-44A5-8087-69E118B612D8)


### New Datatypes Supported in V3 {#GUID-1F7AC30B-B70B-441E-83BE-63462D8EE335}

**Variant**

Oracle GoldenGate trail file columns that contain JSON or XML values map to the Iceberg `variant` type. Oracle GoldenGate identifies:
- JSON columns in the trail file with sub data type value `15`.

- XML columns in the trail file with sub data type value `10`.


**Note:**  The variant type is not supported when the Iceberg file format is set to ORC, for example, `gg.eventhandler.iceberg.fileFormat=orc`. This limitation exists because ORC does not support complex types in the Iceberg Version 3 Java SDK.
{: .infoboxnote}


### Limitations {#GUID-16E4AECB-E178-44A5-8087-69E118B612D8}

When you use Iceberg Version 3 with Oracle GoldenGate, note the following limitations:
- Iceberg Version 3-specific data types such as `timestamp_ns`, `timestamptz_ns`, `geometry`, and `geography` are not supported.

- You can upgrade an existing Iceberg table from Version 2 to Version 3, but you must plan and test the upgrade carefully. For upgrade procedures, refer to the *Apache Iceberg documentation*.


## Troubleshooting and Diagnostics {#GUID-2A5A08BB-E64B-4B38-A035-8596B90C3678}


- Oracle GoldenGate replicat supports the Iceberg data types as per the version 2 specification.

- Iceberg identifier(key) fields cannot be null. Therefore, the Replicat process will ABEND if the key column value is null.

- Schema changes to the table such as `ADD/ALTER/DROP` columns is not supported while Replicat process is running.

  There are steps to quiesce the replication process, apply the schema changes and resume the replication process.

  **Note:**  Contact Oracle Support for assistance with this process.


  The Replicat process will ABEND if there are unmapped columns in the target table.

- Replicat ABEND with the following message:

ICEBERGEH-00060 Operation record at position ‘00000000030000003318’ for the table ‘hadoop.oggdb1.types_tab’ has missing column values in an UPDATE. Replicat will ABEND. To override this behavior set ‘gg.eventhandler.iceberg.abendOnMissingColumns=false’and restart the Replicat process. Setting this property to false will instruct Replicat to lookup missing columns from the target table and therefore may impact performance.



By default, the Iceberg Replicat process expects trails files without missing column value in the UPDATE operations. Replicat can be configured to process compressed trails files with missing column values in the UPDATE operations by setting the property `gg.eventhandler.iceberg.abendOnMissingColumns=false`.

- Replicat ABEND with the following message:

ICEBERGEH-00057 Detected changes in the partition columns for the table ‘hadoop.oggdb1.types_tab’. Partition columns in the previous run: ‘', partition columns in this run: ''. GoldenGate does not support changing partition columns. Alter the table manually to match the partition columns in the previous run and restart the replicat process.


The Iceberg Replicat process does not support changing partition columns.

- Replicat ABEND with the following message:

ICEBERGEH-00067 Invalid state. The column ‘' in the target table '' is not mapped. The following are the mapped columns: ''. Iceberg Replicat requires all the columns in the target table to be mapped. Please map the column ''' and restart the Replicat process.


The Iceberg Replicat process requires all the columns in the target table to be mapped.


- Replicat ABEND with the following message:

ICEBERGEH-00068 Key column ‘' in the table '<table name>' is of type float or double. Iceberg does not support float or double type as identifier (key) fields. Initiating Replicat process shutdown. Please modify the table schema to exclude double/float types as key columns and restart the Replicat process.



As per the current Iceberg specification (version 2), the column types `double` and `float` cannot be used as identifier (key) columns.

- Replicat ABEND with the following message:

ICEBERGEH-00070 Table ‘' contains a key column '' of '<binary/fixed/uuid>' type that is not supported by GoldenGate. The following column types are not supported as key: 'binary, fixed, uuid'. To proceed, either use a supported Iceberg key column type by altering the 'KEYCOLS' clause in the Replicat 'MAP' statement as per the following example: 'MAP ., TARGET ., KEYCOLS("key1", "key2");' or alter the Iceberg target tables's identifier fields to exclude the key column types that are not supported by GoldenGate. You can use the following Iceberg SQL statement to alter the table schema: 'ALTER TABLE prod.db.sample SET IDENTIFIER FIELDS key1, key2'.

The Iceberg types `binary`, `fixed` and `uuid` cannot be used as identifier (key) columns.

- Replicat ABEND with the following message:

ICEBERGEH-00071=Table ‘' does not define an Iceberg identifier column. Identifier columns are used as key columns by GoldenGate. Initiating Replicat process shutdown. Please alter the Iceberg target tables's schema to add identifier columns. You can use the following Iceberg SQL statement to alter the table schema: 'ALTER TABLE prod.db.sample SET IDENTIFIER FIELDS key1, key2'.

The Iceberg target table should have identifier columns (key columns) in the schema.

- Replicat ABEND with the following message:

Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: Operation failed: “This endpoint does not support BlobStorageEvents or SoftDelete. Please disable these account features if you would like to use this endpoint.”, 409, PUT, https://azpoctest.dfs.core.windows.net/azpoctestcont?resource=filesystem, rId: c8a048cc-b01f-0033-0741-52c5d4000000, EndpointUnsupportedAccountFeatures, “This endpoint does not support BlobStorageEvents or SoftDelete. Please disable these account features if you would like to use this endpoint. RequestId:c8a048cc-b01f-0033-0741-52c5d4000000 Time:2026-10-02T07:38:41.5930618Z” ```

This is a known Azure response that can be misleading. It occurs when Hierarchical Namespace (HNS) is disabled on the target Storage Account, which prevents ADLS Gen2 filesystem operations. Enable HNS on the target Azure Storage Account, then restart Replicat.

- `gg.eventhandler.iceberg.gcpProjectId`, `gg.eventhandler.iceberg.gcpServiceAccountJsonKeyFile`.

Connectivity to third-party S3 compatible Storage: