Microsoft Fabric OneLake

Microsoft Fabric is an end-to-end analytics and data platform designed for enterprises that require a unified solution. OneLake is built into the Fabric platform and provides a unified location to store all organizational data where the workloads operate. See https://learn.microsoft.com/en-us/fabric/onelake/. You can use the OneLake Event handler to load files that contain operations records into the following targets:

OneLake Event Handler Prerequisites

OneLake Mappings to Azure Data Lake Gen2

OneLake Event Handler Configuration

OneLake Event Handler Automatic Configuration

OneLake replication involves configuring multiple components, such as the File Writer Handler, Avro formatter, Parquet Event Handler, ORC Event Handler, and the OneLake Event Handler. The Automatic Configuration functionality will autoconfigure these components so that the user configuration is minimal. The properties modified by auto configuration would be logged in the handler log file.

To enable autoconfiguration to replicate data to the Lakehouse target, set the parameter gg.target=fabric_lakehouse.

To enable autoconfiguration to replicate data to the mirrored database target, set the parameter ` gg.target=fabric_mirrored_database`.

File Writer Handler Configuration

The File Writer Handler name is pre set based on the gg.target configuration. For example, if gg.target=fabric_lakehouse, then the File Writer Handler name is set to the value fabric_lakehouse and its properties are automatically set to the required values for Lakehouse. As per this example, you can add or edit a property of the File Writer Handler as follows: gg.handler.fabric_lakehouse.inactivityRollInterval=1m.

Autoconfiguration of Parquet/ORC Event Handler

Event Handler name is pre-set to the value parquet or orc based on the file format configuration.

OneLake Event Handler File Format Configuration for Parquet/ORC

Note: For the Open Mirroring target (gg.target=fabric_mirrored_database), the file format configuration is internal and cannot be modified.

OneLake Event Handler Configuration

OneLake Event Handler name is pre set to the value onelake.

gg.target must be set to one of the following values:

Properties Required/Optional Legal Values Default Explanation
gg.eventhandler.onelake.workspace Required String None Sets the Microsoft Fabric workspace name.
gg.eventhandler.onelake.lakehouse Required String None Applicable only to the Lakehouse target. Sets the Microsoft Fabric lakehouse name.
gg.eventhandler.onelake.mirror Required String None Applicable only to the mirrored database target. Sets the mirrored database name in Fabric.
gg.eventhandler.onelake.tenantId Optional String None Sets the Azure tenant ID of the application.
gg.eventhandler.onelake.clientId Optional String None Sets the Azure client ID of the application.
gg.eventhandler.onelake.clientSecret Optional String None Sets the Azure client secret for the authentication.
gg.eventhandler.onelake.pathMappingTemplate Optional A string with resolvable keywords and constants used to dynamically generate the landing path for data files into OneLake. If gg.target is set to fabric_mirrored_database, then the default value is ${catalogname}.MountedRelationalDatabase/Files/LandingZone/${schemaname}.schema/${tablename}. This cannot be modified. If gg.target=fabric_lakehouse, then the default value is ${catalogname}.lakehouse/Files/ogg/${groupName}/${schemaname}.schema/${tablename}, this can be modified. Use keywords interlaced with constants to dynamically generate a path names at runtime. Example path name would be: ogg/data/${fullyQualifiedTableName}. For more information about the supported keywords, see Template Keywords.
gg.eventhandler.onelake.fileNameMappingTemplate Optional A string with resolvable keywords and constants used to dynamically generate the data file names at runtime. If gg.format is set to fabric_mirrored_database, then this value is set to ${custom[]} and cannot be edited. If gg.target=fabric_lakehouse, then the default value is based on the upstream handler, and can be modified. Use keywords interlaced with constants to dynamically generate a unique file name at runtime. Typically, file names follow the format, ${fullyQualifiedTableName}_${groupName}_${currentTimestamp}.txt.
gg.eventhandler.onelake.endpoint Optional String https://onelake.dfs.fabric.microsoft.com Sets the Fabric OneLake endpoint.
gg.format Optional parquet, orc, or one of the GG for DAA pluggable formatter name. parquet Applicable only to the Lakehouse target. Sets the Fabric OneLake file format. For more information, see File Format for the Lakehouse target.
gg.eventhandler.onelake.replicatGroupPartitionName Optional String ogg_group_name Applicable only If gg.target is set to fabric_mirrored_database. A string to specify the Fabric partition column name. Value should be a string between 1 and 20 characters in length.
gg.eventhandler.onelake.maxRetries Optional Integer 3 The maximum number of retries to attempt if the file upload to OneLake fails. Value should be between 1 and
20.        
gg.eventhandler.onelake.maxRetries Optional Integer 10 The initial delay between retries (in seconds). The retry delay is initially set to the specified value and doubles with each retry. Value should be between 1 and 120.
gg.keyupdate.threadrange.behavior Optional ABEND, WARN If gg.target is set to fabric_mirrored_database, then the default value is ABEND. For all the other targets, the default value is not set. Controls Replicat’s behavior when it encounters a key update operation for a table configured with THREADRANGE clause. ABEND aborts processing, while WARN logs a warning.
gg.eventhandler.onelake.renameUploadEnabled Optional Boolean, True Enables uploading to a temporary file, validating the uploaded file size, and renaming it to the final target file. Set to false to use the legacy direct final-path upload behavior.  

File Format for the Lakehouse target

The parameter gg.format can be configured to set the file format.

It can be set to one of the following values:

OneLake Event Handler Classpath Configuration

Ensure that the classpath includes the path to the following dependencies:

Ensure that the classpath includes the path to the following dependencies in the given order: 1. OneLake Event Handler dependencies.

  1. Parquet Event Handler dependencies, including Hadoop dependencies.

  2. Azure Storage File DataLake Java SDK.

OneLake Event Handler Dependencies

The dependency downloader script onelake.sh can be used to download the OneLake dependencies. Alternatively, you can manually download the OneLake dependencies using the following maven co-ordinates:

<dependencies>
    <dependency>
        <groupId>com.azure</groupId>
        <artifactId>azure-storage-file-datalake</artifactId>
        <version>12.20.0</version>
    </dependency>
    <dependency>
        <groupId>com.azure</groupId>
        <artifactId>azure-identity</artifactId>
        <version>1.13.1</version>
    </dependency>
</dependencies>

Edit the gg.classpath configuration parameter to include the path to the Azure Storage File Data Lake SDK.

OneLake Event Handler Authentication

You can authenticate the Azure Storage device by configuring following:

Azure Tenant ID, Client ID, and Client Secret

To obtain your Azure tenant ID:

OneLake Event Handler Proxy Configuration

When the process is run behind a proxy server, the property jvm.bootoptions can be used to set proxy server configuration using well-known Java proxy properties. For example:

jvm.bootoptions=-Dhttps.proxyHost=some-proxy-address.com -Dhttps.proxyPort=80 -Djava.net.useSystemProxies=true

Sample Configuration for Lakehouse Target

gg.target=fabric_lakehouse
#TODO: format can be 'parquet' or 'orc' or one of the pluggable formatter types. Default is 'parquet'.
#gg.format=parquet
#TODO: Edit the Fabric workspace name.
gg.eventhandler.onelake.workspace=<workspace-name>
#TODO: Edit the Fabric lakehouse name.
gg.eventhandler.onelake.lakehouse=<lakehouse-name>
#TODO: Edit the tenant ID of the application.
gg.eventhandler.onelake.tenantId=<azure-tenant-id>
#TODO: Edit the client ID of the application.
gg.eventhandler.onelake.clientId=<azure-client-id>
#TODO: Edit the client secret for the authentication.
gg.eventhandler.onelake.clientSecret=<azure-client-secret>
#TODO: Edit the classpath to include Hadoop, Parquet, and Azure DataLake SDK dependencies.
gg.classpath=$THIRD_PARTY_DIR/onelake/*:$THIRD_PARTY_DIR/hadoop/*:$THIRD_PARTY_DIR/parquet/
#TODO: Edit the proxy configuration.
#jvm.bootoptions=-Dhttps.proxyHost=some-proxy-address.com -Dhttps.proxyPort=80 -Djava.net.useSystemProxies=true

Sample Configuration for Mirrored Database Target

gg.target=fabric_mirrored_database
#TODO: Edit the Fabric workspace name.
gg.eventhandler.onelake.workspace=<workspace-name>
#TODO: Edit the Fabric mirror Database name.
gg.eventhandler.onelake.mirror=<mirror-name>
#TODO: Edit the tenant ID of the application.
gg.eventhandler.onelake.tenantId=<azure-tenant-id>
#TODO: Edit the client ID of the application.
gg.eventhandler.onelake.clientId=<azure-client-id>
#TODO: Edit the client secret for the authentication.
gg.eventhandler.onelake.clientSecret=<azure-client-secret>#TODO: Edit the classpath to include Hadoop, Parquet, and Azure DataLake SDK dependencies.
gg.classpath=$THIRD_PARTY_DIR/hadoop/*:$THIRD_PARTY_DIR/parquet/*:$THIRD_PARTY_DIR/onelake/*
#TODO: Edit the proxy configuration.
#jvm.bootoptions=-Dhttps.proxyHost=some-proxy-address.com -Dhttps.proxyPort=80 -Djava.net.useSystemProxies=true

Performance Considerations

You can set GROUPTRANSOPS to a higher value (recommended up to 20000). By default, the value is set to 1000.

For initial load replicats, you can create multiple replicats with the classic mode and configure each replicat to handle one individual table or a discrete subset of tables.

For real-time replicats, you can use coordinated replicat mode to improve performance. While configuring coordinated replicat, tables should be mapped to individual threads.

Upload Reliability and Temporary Files

By default, the OneLake Event Handler uploads files using a temporary file followed by an atomic rename operation. This approach prevents zero-byte or partially uploaded files from appearing in the OneLake landing path with their final target names.

For each upload, the handler first writes the file to a temporary location under the _scratchPad subdirectory of the target directory, using the following naming convention:

_scratchPad/.gg-upload-<replicatGroup>-<uuid>-<targetFileName>.tmp

After the upload completes, the handler verifies that the size of the uploaded temporary file matches the local source file. Only after this validation succeeds does the handler rename the temporary file to its final target name.

If the temporary upload is incomplete, zero bytes, or the validation fails for any reason, the handler attempts to delete the temporary file and retries the upload according to the values of gg.eventhandler.onelake.maxRetries and gg.eventhandler.onelake.initialRetryDelay. Temporary file cleanup is performed on a best-effort basis because the same network conditions that caused the upload failure may also prevent cleanup.

This reliability mechanism adds a single file-size validation and a single rename operation for each successfully uploaded file. The file contents are uploaded only once. For typical Parquet and ORC workloads, the additional overhead is minimal. However, workloads that generate large numbers of very small files may experience a slight increase in per-file latency.

To disable this behavior, set:

gg.eventhandler.onelake.renameUploadEnabled=false

When this property is disabled, the handler uploads files directly to their final destination, restoring the legacy upload behavior. Although this eliminates the validation and rename operations, it may expose zero-byte or partially uploaded files if an upload is interrupted. Note: Temporary files matching the pattern _scratchPad/.gg-upload-* may appear while an upload is in progress. These are temporary staging files and are not final target files. If Replicat abends or a network interruption occurs, some temporary files may remain because cleanup is performed on a best-effort basis and can be affected by the same failure. If these files persist after confirming that no upload is active, they can be retained for troubleshooting or safely deleted.

OneLake Event Handler Primary Key Update

Primary key UPDATE behavior depends on the file-format configuration.

Mirrored Database in Microsoft Fabric

When file format is set to gg.format=fabric_mirroring, then primary key UPDATE operations will be split into a DELETE operation followed by an INSERT operation. This behavior cannot be modified.

Microsoft Fabric OneLake Initial Load

Transitioning from Initial Load to CDC Replication

You can follow the provided sequence to safely transition from Initial Load replication to Change Data Capture (CDC) replication.

  1. Run the Initial Load Replicat.

  2. Allow the Initial Load Replicat to complete loading all table data into the target OneLake location.

  3. Validate Initial Load completion:

    • Verify that all expected records are successfully replicated to the target.

    • Confirm row counts and data consistency between the source and target before you proceed.

  4. Stop and remove the Initial Load Replicat after successful validation, if it is no longer required.

  5. Delete the OneLake scratch pad directories:

    • Delete the scratchPad sub directory for all tables loaded through the Initial Load Replicat.

    • You can delete the scratchPad directory by using Azure Storage Explorer or REST API commands.

  6. Start the CDC Replicat.

  7. Start the CDC Replicat (CDREP1) only after the scratch pad cleanup is complete. CDC replication should then successfully acquire ownership of the table directory.

Lakehouse in Microsoft Fabric

If gg.target=fabric_lakehouse is set, then by default primary key UPDATE operations will result in a Replicat ABEND.

This behavior can be modified by configuration of the formatter property gg.handler.onelake.format.pkUpdateHandling

The property gg.handler.onelake.format.pkUpdateHandling can accept one of the following input:

Ingesting into Fabric Mirrored Tables using Partitioning

The existing solution for ingesting data into Fabric mirrored tables uses a single landing folder for a specific table and a metadata file to specify key columns. However, this approach does not scale well for large tables as only a single replicat thread is used. To improve scalability, we can leverage Fabric’s partitioning feature and Oracle GoldenGate’s coordinated apply feature to distribute the workload for specific tables.

Replicat MAP Statement

For tables that require partitioning, the Oracle GoldenGate coordinated apply feature is used to distribute the workload. For example:

MAP scott.orders, TARGET scott.orders, THREADRANGE (1-2, OID);

Th data files are stored into the following folders:

${catalogname}.MountedRelationalDatabase/Files/LandingZone/${schemaname}.schema/${tablename}/ogg_group_name=REP001/
${catalogname}.MountedRelationalDatabase/Files/LandingZone/${schemaname}.schema/${tablename}/ogg_group_name=REP002/

Metadata File Updates

For tables configured with THREADRANGE, the Fabric metadata file is updated as follows:

{ "keyColumns": ["ogg_group_name", "KEY_1", "KEY_2"], "IsPartitioned": "true"
        }

The metadata file is placed in the base table folder as:

${catalogname}.MountedRelationalDatabase/Files/LandingZone/${schemaname}.schema/${tablename}.

For tables that do not require partitioning, coordinated apply or THREADRANGE configuration is not required.

The Fabric metadata file for non-partitioned tables is as follows:

{ "keyColumns": ["KEY_1", "KEY_2"], "IsPartitioned": "false" }

Limitation

When partitioning is enabled for a table, that is, a table configured with THREADRANGE clause, in-place primary key update operations may be applied out of order.

To manage this behavior, use the gg.keyupdate.threadrange.behavior parameter, which allows you to control how Replicat handles such operations.

Managing the partnerEvents.json File for Fabric Mirrored Databases

Oracle GoldenGate generates a partnerEvents.json file in the landing zone when you configure a Fabric Mirrored Database with partner event tagging enabled.

Note: Oracle recommends implementing this file as part of the standard Oracle GoldenGate integration, although Microsoft considers it optional.

File Location

Oracle GoldenGate writes the partnerEvents.json file at the mirrored database level in the landing zone, rather than at the individual table level.

Behavior

Considerations

OneLake Event Handler Troubleshooting and Diagnostics