Create pipeline resources

Learn to create resources necessary to use Pipelines.

Related Topics

Connect to Oracle Autonomous Databases

Create connections

OCI GoldenGate Pipelines support Oracle Database connections. Before you create a pipeline, ensure that you first prepare your database for GoldenGate, and then create connections to your source and target databases.

Note:

If replicating from an Autonomous AI Database source to a target that isn't an Autonomous AI Database, then you must unzip the source Autonomous AI Database wallet to /u01/targetwallet on target database. DBLINK is created from the target database to the source database.

You can also create connection source and target connections in the Create pipeline workflow.

Create pipelines

Before you begin:

Ensure that you:

Check your source and target database versions. The target database version must be a higher version than the source database.
Review and configure row uniqueness across database tables to avoid primary key issues with ZeroETL mirroring. Primary keys uniquely identifies a record or row in a database table. See Ensuring row uniqueness in source and target tables.
As a user with admin privileges for the target database, grant the following privileges to the GGADMIN user for ZeroETL mirroring:
```
GRANT DWROLE TO GGADMIN;
GRANT DATAPUMP_CLOUD_EXP TO GGADMIN;
GRANT DATAPUMP_CLOUD_IMP TO GGADMIN;
GRANT EXECUTE ON DBMS_CLOUD_ADMIN TO GGADMIN;
GRANT CREATE DATABASE LINK TO GGADMIN;
```
Additionally, for non-Autonomous AI Databases, add the following for Initial Load:
```
GRANT DATAPUMP_EXP_FULL_DATABASE TO GGADMIN CONTAINER=CURRENT;
GRANT DATAPUMP_IMP_FULL_DATABASE TO GGADMIN CONTAINER=CURRENT;
GRANT CREATE DATABASE LINK TO GGADMIN CONTAINER=CURRENT;
```
For Capture and Apply in non-Autonomous AI Databases, review Prepare database users and privileges for Oracle.
Check DBA_GOLDENGATE_SUPPORT_MODE view on the source database that displays information about the level of Oracle GoldenGate capture process support for the tables in the database.
Add the minimum required policies for GoldenGate pipelines.

Note:

If replicating data from Autonomous AI Database to a non-Autonomous AI Database, you must turn off GLOBAL_NAMES on your non-Autonomous AI Database. Run the following command:

ALTER SYSTEM SET GLOBAL_NAMES = FALSE;

See step 4 of Create Database Links from Other Databases to Autonomous AI Database to learn more.

To create a pipeline:

On the OCI GoldenGate Overview page, in the GoldenGate menu, click Pipelines.

Alternatively, you can click Data Fabric, and then Create pipeline.
On the Pipelines page, click Create pipeline.
In the Create pipeline panel, complete the fields as follows, and then click Create pipeline:
1. For Name, enter a name for the pipeline.
2. (Optional) For Description, enter a description to help distinguish this pipeline from others.
3. For Compartment, select the compartment in which to create the pipeline.
4. Select a license type.
5. Select your source and target connections from the dropdowns.
  Note:
  - If you haven't yet created connections, click Create connection to get started.
  - If using a connection with a Shared endpoint as your source or target, you must also select a subnet.
6. (Optional) Click Show advanced options to configure Process options:
  - Copy existing data before ongoing replication: Select this option to perform an initial load of the data. For existing tables, you can choose to:
    - Truncate: Deletes existing rows and then loads rows from the source.
    - Replace: Drops the existing table and then creates and loads it from the source.
    - Append: Leaves existing rows unchanged and loads rows from the source.
    - Skip: Leaves the table as is and moves on to the next object.
  - Replace schema changes (DDL): When selected, choose what action to take upon existing schemas when the process encounters an error:
    - Terminate: Roll back the transaction and stop processing.
    - Discard: Log the error to the discard file and continue processing.
    - Ignore: Ignore the error and continue processing.
  - Action upon DML error: Select the action to take upon existing data when the process encounters a DML error:
    - Terminate
    - Discard
    - Ignore
  - Restart after failure: Select this option to enable autorestart in the event the process stops for any reason.
  - Start pipeline using default mapping: Starts the pipeline immediately after creation with the default 1:1 mapping rules. If not selected, you can configure the rule mappings after creation and then manually start the pipeline.

Add mapping rules

To preview, add, or remove mapping rules:

On the Pipeline information page, select Mapping rules.
In the Mapping rules list you can review or edit the current mapping rules and:
- Preview: Review how your mapping rules affect the schemas and tables included in the replication.
- Add: Identify tables and schemas to include or exclude. You can use the fully qualified object name or wildcards in place of any part of the object name. For more information, see Using Wildcards in Database Object Names.
- Remove: Remove mapping rules.

Enable pipeline logs

On the Pipelines page, select the pipeline for which you want to enable logging.
On the Pipelines details page, select Logs.
In list of pipeline logs, select Enable log from the Critical Events Actions menu.
In the Enable log panel:
1. For Log name, enter a name.
2. From the Compartment dropdown, select a compartment.
3. For Log group, you can:
  - Select a group from the dropdown
  - Create a new group
  - Leave it blank, and a default group is automatically assigned
4. For Log retention, select the number of months from the dropdown.
5. Click Enable log.

Wait for the status to become Active.

Start the pipeline

To start the pipeline:

From the Actions menu on the pipeline's details page, select Start.
On the pipeline's details page, select Initialization.

The Initialization steps displays the current status of the pipeline steps. For each step, you can select View details from its Actions menu and review corresponding messages.
After the Pipeline's status is Active (Running), select Runtime.

Runtime processes display the state and latency of the Capture and Apply processes.

Known issues

Oracle Data Pump fails if the target database time zone file version is lower than the source database's

The pipeline option, "Copy existing data to target database before starting replication" uses Oracle Data Pump. Oracle Data Pump fails if the target database's time zone file version is lower than that of the source database.

Workaround: To work around this, you must upgrade the target database's time zone version. See Manage time zone file updates on Autonomous Database.

Title and Copyright Information

Oracle and/or its affiliates.