Create pipeline resources

Learn to create resources necessary to use Pipelines.

Create connections

OCI GoldenGate Pipelines support Oracle AI Database connections. Before you create a pipeline, ensure that you first prepare your database for GoldenGate, and then create connections to your source and target databases.

Note: If replicating from an Autonomous AI Database source to a target that isn’t an Autonomous AI Database, then you must unzip the source Autonomous AI Database wallet to /u01/targetwallet on target database. DBLINK is created from the target database to the source database.

You can also create connection source and target connections in the Create pipeline workflow.

Create pipelines

Before you begin:

Ensure that you:

Check your source and target database versions. The target database version must the same or a higher version than the source database.
Review and configure row uniqueness across database tables to avoid primary key issues with ZeroETL mirroring. Primary keys uniquely identifies a record or row in a database table. See Ensuring row uniqueness in source and target tables.

As a user with admin privileges for the target database, grant the following privileges to the GGADMIN user for ZeroETL mirroring:

GRANT DWROLE TO GGADMIN;
GRANT DATAPUMP_CLOUD_EXP TO GGADMIN;
GRANT DATAPUMP_CLOUD_IMP TO GGADMIN;
GRANT EXECUTE ON DBMS_CLOUD_ADMIN TO GGADMIN;
GRANT CREATE DATABASE LINK TO GGADMIN;

Additionally, for non-Autonomous AI Databases, add the following for Initial Load:

GRANT DATAPUMP_EXP_FULL_DATABASE TO GGADMIN CONTAINER=CURRENT;
GRANT DATAPUMP_IMP_FULL_DATABASE TO GGADMIN CONTAINER=CURRENT;
GRANT CREATE DATABASE LINK TO GGADMIN CONTAINER=CURRENT;

For Capture and Apply in non-Autonomous AI Databases, review Prepare database users and privileges for Oracle.

Check DBA_GOLDENGATE_SUPPORT_MODE view on the source database that displays information about the level of Oracle GoldenGate capture process support for the tables in the database.
Add the minimum required policies for GoldenGate pipelines.

Note:

If replicating data from Autonomous AI Database to a non-Autonomous AI Database, you must turn off GLOBAL_NAMES on your non-Autonomous AI Database. Run the following command:

ALTER SYSTEM SET GLOBAL_NAMES = FALSE;

See step 4 of Create Database Links from Other Databases to Autonomous AI Database to learn more.

To create a pipeline:

On the OCI GoldenGate Overview page, in the GoldenGate menu, select Pipelines.

Alternatively, you can select Data Fabric, and then Create pipeline.
On the Pipelines page, select Create pipeline.
In the Create pipeline panel, complete the fields as follows, and then select Create pipeline:
1. For Name, enter a name for the pipeline.
2. (Optional) For Description, enter a description to help distinguish this pipeline from others.
3. For Compartment, select the compartment in which to create the pipeline.
4. Select a license type.
5. Select your source and target connections from the dropdowns.
  Note:
  - If you haven't yet created connections, select Create connection to get started.
  - If using a connection with a Shared endpoint as your source or target, you must also select a subnet.
6. (Optional) Expand Advanced options to configure Process options:
  - Copy existing data before ongoing replication: Select this option to perform an initial load of the data. For existing tables, you can choose to:
    - Truncate: Deletes existing rows and then loads rows from the source.
    - Replace: Drops the existing table and then creates and loads it from the source.
    - Append: Leaves existing rows unchanged and loads rows from the source.
    - Skip: Leaves the table as is and moves on to the next object.
  - Initial Load type: Select the method to transfer a complete dataset from source to target before starting ongoing replication:
    - Database Link: Transfers data directly from the source database to the target database using an established database link, without intermediate storage. Works best for databases located in the same network or connected through a stable, high speed link.
    - Object Storage: Stages initial load data in Oracle Object Storage before importing it into the target database. Works best for migrations that involve large volumes of data across different regions or environments.
      
      Learn more about Initial Load types.
  - Replace schema changes (DDL): When selected, choose what action to take upon existing schemas when the process encounters an error:
    - Terminate: Roll back the transaction and stop processing.
    - Discard: Log the error to the discard file and continue processing.
    - Ignore: Ignore the error and continue processing.
  - Action upon DML error: Select the action to take upon existing data when the process encounters a DML error:
    - Terminate
    - Discard
    - Ignore
  - Restart after failure: Select this option to enable autorestart in the event the process stops for any reason.
  - Start pipeline using default mapping: Starts the pipeline immediately after creation with the default 1:1 mapping rules. If not selected, you can configure the rule mappings after creation and then manually start the pipeline.

Add mapping rules

To preview, add, or remove mapping rules:

On the Pipeline information page, select Mapping rules.
In the Mapping rules list you can review or edit the current mapping rules and:
- Preview: Review how your mapping rules affect the schemas and tables included in the replication.
- Add: Identify tables and schemas to include or exclude. You can use the fully qualified object name or wildcards in place of any part of the object name. For more information, see Using Wildcards in Database Object Names.
- Remove: Remove mapping rules.

Enable pipeline logs

On the Pipelines page, select the pipeline for which you want to enable logging.
On the Pipelines details page, select Logs.
In list of pipeline logs, select Enable log from the Critical Events Actions menu.
In the Enable log panel:
1. For Log name, enter a name.
2. From the Compartment dropdown, select a compartment.
3. For Log group, you can:
  - Select a group from the dropdown
  - Create a new group
  - Leave it blank, and a default group is automatically assigned
4. For Log retention, select the number of months from the dropdown.
5. Select Enable log.

Wait for the status to become Active.

Start the pipeline

To start the pipeline:

From the Actions menu on the pipeline's details page, select Start.
On the pipeline's details page, select Initialization.

The Initialization steps displays the current status of the pipeline steps. For each step, you can select View details from its Actions menu and review corresponding messages.
After the Pipeline's status is Active (Running), select Runtime.

Runtime processes display the state and latency of the Capture and Apply processes.

Known issues

Oracle Data Pump fails if the target database time zone file version is lower than the source database’s

The pipeline option, “Copy existing data to target database before starting replication” uses Oracle Data Pump. Oracle Data Pump fails if the target database’s time zone file version is lower than that of the source database.

Workaround: To work around this, you must upgrade the target database’s time zone version. See Manage time zone file updates on Autonomous AI Database.