Manage pipelines

Explore pipeline management tasks essential for working with OCI GoldenGate Pipelines.

View pipeline details

Select a pipeline from the Pipelines page to view its details. On the pipeline details page, you can view the pipeline’s status, which can be one of the following:

• Creating

• Active (Stopped)

• Active (Running)

• Updating

• Needs attention

• Failed

You can also edit the pipeline or use the Actions menu to perform other actions such as:

• Edit

• Start or Stop

• Collect diagnostics

• Move resource

• Add tags

• Delete

Switch to another tab to view other related pipeline information.

Pipeline information

The Pipeline information page displays general information about this resource such as its OCID, compartment in which it resides, when it was created and last updated, and so on.

Mapping rules

Add, remove, or preview mapping rules that change the default one-to-one mapping of source tables and schemas to the target.

Initialization

View initialization steps that display the current status of pipeline steps. You can select any steps to view its details.

Runtime

View runtime status and latency.

Monitoring

View pipeline metrics, enable or disable critical event logs, and create and view notifications.

Work requests

View the status of pipeline work requests. Pipeline operations that create work requests include:

• Create

• Start

• Stop

• Update

• Delete

• Move

• Collect diagnostics

Tags

Tags help you locate resources within your tenancy. You can add and view a pipeline’s tags from the Pipelines page and from the pipeline details page. See manage tags.

Edit a pipeline

To edit a pipeline, select Edit on the pipeline’s details page. In the edit panel, you can edit:

• Name

• Description

• Process options:

  Note: You must restart the pipeline for any changes to take effect.

  • Copy existing data before ongoing replication: Select this option to perform an initial load of the data. For existing tables, you can choose to:

    • Truncate: Deletes existing rows and then loads rows from the source.

    • Replace: Drops the existing table and then creates and loads it from the source.

    • Append: Leaves existing rows unchanged and loads rows from the source.

    • Skip: Leaves the table as is and moves on to the next object.

  • Replace schema changes (DDL): When selected, choose what action to take upon existing schemas when the process encounters an error:

    • Terminate: Roll back the transaction and stop processing.

    • Discard: Log the error to the discard file and continue processing.

    • Ignore: Ignore the error and continue processing.

  • Action upon DML error: Select the action to take upon existing data when the process encounters a DML error:

    • Terminate

    • Discard

    • Ignore

  • Restart after failure: Select this option to enable autorestart in the event the process stops for any reason.

  • Start pipeline using default mapping: Starts the pipeline immediately after creation with the default 1:1 mapping rules. If not selected, you can configure the rule mappings after creation and then manually start the pipeline.

Start a pipeline

To start a pipeline, select Start from an inactive pipeline’s Actions menu on its details page. Select Start in the Start pipeline dialog to confirm this action. The pipeline then runs through the initialization steps based on the process options selected upon creation (or when updated if you edited the pipeline) before starting the Capture and Apply processes.

Pause a pipeline

You can pause a pipeline to temporarily stop Capture and Delivery processes. Pausing a pipeline does not pause billing for these resources. While a pipeline is paused, you can edit the pipeline, its mapping rules, and preview mapping rules.

Note: To stop billing, you must stop the pipeline.

To pause a pipeline, select Pause from the pipeline’s Actions menu.

When you’re ready for the pipeline to continue, select Start from the pipeline’s Actions menu.

If you want to stop the pipeline completely, select Stop.

Stop a pipeline

To stop a pipeline, select Stop from an Active pipeline’s Actions menu on its details page. Select Stop in the Stop pipeline dialog to confirm this action.

Stopping a pipeline also stops billing for these resources. You cannot edit the pipeline, its mappings, or preview the mappings when a pipeline is stopped and in an Inactive state.

To restart the pipeline, select Start from the pipeline’s Actions menu.

Clone a pipeline

You can clone a pipeline whose state is Active, Inactive, or Needs attention.

To clone a pipeline:

• On the Pipelines page, select Clone from a pipeline’s Actions menu, then follow the steps below.

• On a Pipeline’s details page, select Clone from the Actions menu.

  1. In the Clone pipeline panel, enter a Name for the pipeline clone.

  2. (Optional) Enter a description for the pipeline clone.

  3. Select the Compartment in which to create the pipeline clone.

  4. (Optional) Complete the Advanced options as needed.

  5. Select Clone.

Collect diagnostics

Collect diagnostics to analyze or share information about your OCI GoldenGate pipeline. The diagnostics file contains discarded records, if any, for the Capture and Apply processes.

To collect diagnostics about your pipeline:

1. On the pipeline details page, from the Actions menu, select Collect diagnostics.

2. In the Collect diagnostics panel, select the compartment and bucket in which to save the diagnostics file.

3. Enter a name prefix for the diagnostics file.

4. Select Collect diagnostics.

After the collection process completes, you can download the diagnostics file from the pipeline details page.

Monitor pipeline metrics

On the pipeline details page, select Monitoring to view pipeline metrics such as CPU utilization, OCPU consumption, Memory utilization, and Overall pipeline health. For more information about available metrics, see:

• Monitor performance

• OCI GoldenGate Metrics

Manage mapping rules

To preview, add, or remove mapping rules:

1. On the Pipeline information page, select Mapping rules.

2. In the Mapping rules list you can review or edit the current mapping rules and:

   • Preview: Review how your mapping rules affect the schemas and tables included in the replication.

   • Add: Identify tables and schemas to include or exclude. You can use the fully qualified object name or wildcards in place of any part of the object name. For more information, see Using Wildcards in Database Object Names.

   • Remove: Remove mapping rules.

View initialization steps

If you’re familiar with GoldenGate processes, then a pipeline is an abstraction of a data replication process. When you start the pipeline, initialization steps are performed in a linear fashion, creating all the necessary resources and prepares databases for data replication based on information collected upon creation. Initialization steps include:

• Check source and target database

• Prepare the database objects for extraction

• Creates a Heartbeat Table

• Creates and runs the Extract (Capture)

• Check that target schemas and tables exist

• Creates the Checkpoint Table

• Creates and runs the Replicat (Apply)

You can select View details from the Actions menu of each initialization step to review the status and system messages corresponding to each step.

Runtime

When your pipeline is Active and running, you can view the status and latency of the Capture and Apply processes on the Runtime section of your pipeline details page. You can use this information to observe the state of these processes and take action when needed.

Set up notifications

You can set up notifications to be kept informed of pipeline events.

To set up pipeline event notifications:

1. On your pipeline details page, select Monitoring.

2. On the Monitoring page, scroll down to Notifications, and then select Create notification.

3. On the Create notification page, you can select:

   • Quickstart, and then select from a template of preconfigured notification setups.

   • Advanced event notification, and then select Event rules that trigger actions for different event conditions.

   • Advanced alarm notification, and then select the metrics to track and trigger rules for the alarm to fire.

4. For Topic and subscriptions:

   1. Create a new topic or select an existing one.

   2. Confirm subscriptions to receive messages.

5. (Optional) For Advanced alarm notification, expand Advanced options, and configure the Alarm body (message body), and the notification’s message format.

6. Select Submit.

Manage pipeline logs

Enable or disable critical events logs for OCI GoldenGate pipelines.

To enable critical events logs:

1. On the pipeline details page, select Monitoring.

2. On the Monitoring page, scroll down to Logs.

3. In the Actions menu for Critical Events, select Enable log.

4. In the Enable log panel, complete the following fields, and then select Enable log:

   1. For Log name, enter a name for the log.

   2. From the Compartment dropdown, select the compartment in which to create the log.

   3. Select a Log group or select Create new group.

   4. Select a Log retention time.

Move a pipeline

You can move a pipeline from one compartment to another compartment.

To move a pipeline:

1. From the Pipelines page, select the pipeline you want to move.

2. From the Actions menu on the pipeline's details page, select Move resource.

3. In the Move Resource to a Different Compartment dialog, select the compartment to which you want to move the pipeline.

4. Select Move resource.

Manage tags for a pipeline

You can add and view a pipeline’s tags from the Pipelines page and from the pipeline’s details page.

On the Pipelines page, from a pipeline’s Actions menu, select Add Tags or View Tags.

On the pipeline details page, you can either select Tags from the Actions menu, or select the Tags tab to view and manage tags.

Learn more about tagging.

Delete a pipeline

Ensure that your pipeline is stopped and in an inactive state before you delete the pipeline.

To delete a pipeline:

1. On the Pipelines page, select the pipeline you want to delete.

2. From the Actions menu on the pipeline's details page, select Delete.

3. In the Delete pipeline dialog, confirm that you want to delete this pipeline, and then select Delete.

This action cannot be undone.