Manage pipelines

Explore pipeline management tasks essential for working with OCI GoldenGate Pipelines.

View pipeline details

Select a pipeline from the Pipelines page to view its details. On the pipeline details page, you can view the pipeline's status, which can be one of the following:
  • Creating
  • Active (Stopped)
  • Active (Running)
  • Updating
  • Needs attention
  • Failed
You can also edit the pipeline or use the Actions menu to perform other actions such as: Switch to another tab to view other related pipeline information.

Pipeline information

The Pipeline information page displays general information about this resource such as its OCID, compartment in which it resides, when it was created and last updated, and so on.

Mapping rules

Add, remove, or preview mapping rules that change the default one-to-one mapping of source tables and schemas to the target.

Initialization

View initialization steps that display the current status of pipeline steps. You can select any steps to view its details.

Runtime

View runtime status and latency.

Monitoring

View pipeline metrics, enable or disable critical event logs, and create and view notifications.

Work requests

View the status of pipeline work requests. Pipeline operations that create work requests include:

  • Create
  • Start
  • Stop
  • Update
  • Delete
  • Move
  • Collect diagnostics

Tags

Tags help you locate resources within your tenancy. You can add and view a pipeline's tags from the Pipelines page and from the pipeline details page. See manage tags.

Edit a pipeline

To edit a pipeline, click Edit on the pipeline's details page. In the edit panel, you can edit:

  • Name
  • Description
  • Process options:

    Note:

    You must restart the pipeline for any changes to take effect.
    • Copy existing data before ongoing replication: Select this option to perform an initial load of the data. For existing tables, you can choose to:
      • Truncate: Deletes existing rows and then loads rows from the source.
      • Replace: Drops the existing table and then creates and loads it from the source.
      • Append: Leaves existing rows unchanged and loads rows from the source.
      • Skip: Leaves the table as is and moves on to the next object.
    • Replace schema changes (DDL): When selected, choose what action to take upon existing schemas when the process encounters an error:
      • Terminate: Roll back the transaction and stop processing.
      • Discard: Log the error to the discard file and continue processing.
      • Ignore: Ignore the error and continue processing.
    • Action upon DML error: Select the action to take upon existing data when the process encounters a DML error:
      • Terminate
      • Discard
      • Ignore
    • Restart after failure: Select this option to enable autorestart in the event the process stops for any reason.
    • Start pipeline using default mapping: Starts the pipeline immediately after creation with the default 1:1 mapping rules. If not selected, you can configure the rule mappings after creation and then manually start the pipeline.

Start a pipeline

To start a pipeline, select Start from an inactive pipeline's Actions menu on its details page. Click Start in the Start pipeline dialog to confirm this action. The pipeline then runs through the initialization steps based on the process options selected upon creation (or when updated if you edited the pipeline) before starting the Capture and Apply processes.

Stop a pipeline

To stop a pipeline, select Stop from an Active pipeline's Actions menu on its details page. Click Stop in the Stop pipeline dialog to confirm this action. Once the pipeline stops and is in inactive state, you can click Start to restart the pipeline.

Note:

When you stop a Pipeline, you're still billed for resources used because a deployment still runs in the background. To avoid recurring costs, you must delete the pipeline.

Collect diagnostics

Collect diagnostics to analyze or share information about your OCI GoldenGate pipeline. The diagnostics file contains discarded records, if any, for the Capture and Apply processes.

To collect diagnostics about your pipeline:
  1. On the pipeline details page, from the Actions menu, select Collect diagnostics.
  2. In the Collect diagnostics panel, select the compartment and bucket in which to save the diagnostics file.
  3. Enter a name prefix for the diagnostics file.
  4. Click Collect diagnostics.
After the collection process completes, you can download the diagnostics file from the pipeline details page.

Monitor pipeline metrics

On the pipeline details page, select Monitoring to view pipeline metrics such as CPU utilization, OCPU consumption, Memory utilization, and Overall pipeline health. For more information about available metrics, see

Manage mapping rules

To preview, add, or remove mapping rules:
  1. On the Pipeline information page, select Mapping rules.
  2. In the Mapping rules list you can review or edit the current mapping rules and:
    • Preview: Review how your mapping rules affect the schemas and tables included in the replication.
    • Add: Identify tables and schemas to include or exclude. You can use the fully qualified object name or wildcards in place of any part of the object name. For more information, see Using Wildcards in Database Object Names.
    • Remove: Remove mapping rules.

View initialization steps

If you're familiar with GoldenGate processes, then a pipeline is an abstraction of a data replication process. When you start the pipeline, initialization steps are performed in a linear fashion, creating all the necessary resources and prepares databases for data replication based on information collected upon creation. Initialization steps include:

  • Check source and target database
  • Prepare the database objects for extraction
  • Creates a Heartbeat Table
  • Creates and runs the Extract (Capture)
  • Check that target schemas and tables exist
  • Creates the Checkpoint Table
  • Creates and runs the Replicat (Apply)

You can select View details from the Actions menu of each initialization step to review the status and system messages corresponding to each step.

Runtime

When your pipeline is Active and running, you can view the status and latency of the Capture and Apply processes on the Runtime section of your pipeline details page. You can use this information to observe the state of these processes and take action when needed.

Set up notifications

You can set up notifications to be kept informed of pipeline events.

To set up pipeline event notifications:
  1. On your pipeline details page, select Monitoring.
  2. On the Monitoring page, scroll down to Notifications, and then click Create notification.
  3. On the Create notification page, you can select:
    • Quickstart, and then select from a template of preconfigured notification setups.
    • Advanced event notification, and then select Event rules that trigger actions for different event conditions.
    • Advanced alarm notification, and then select the metrics to track and trigger rules for the alarm to fire.
  4. For Topic and subscriptions:
    1. Create a new topic or select an existing one.
    2. Confirm subscriptions to receive messages.
  5. (Optional) For Advanced alarm notification, expand Advanced options, and configure the Alarm body (message body), and the notification’s message format.
  6. Click Submit.

Manage pipeline logs

Enable or disable critical events logs for OCI GoldenGate pipelines.

To enable critical events logs:
  1. On the pipeline details page, select Monitoring.
  2. On the Monitoring page, scroll down to Logs.
  3. In the Actions menu for Critical Events, select Enable log.
  4. In the Enable log panel, complete the following fields, and then click Enable log:
    1. For Log name, enter a name for the log.
    2. From the Compartment dropdown, select the compartment in which to create the log.
    3. Select a Log group or click Create new group.
    4. Select a Log retention time.

Move a pipeline

You can move a pipeline from one compartment to another compartment.

To move a pipeline:
  1. From the Pipelines page, select the pipeline you want to move.
  2. From the Actions menu on the pipeline's details page, click Move resource.
  3. In the Move Resource to a Different Compartment dialog, select the compartment to which you want to move the pipeline.
  4. Click Move resource.

Manage tags for a pipeline

You can add and view a pipeline's tags from the Pipelines page and from the pipeline's details page.

On the Pipelines page, from a pipeline's Actions menu, select Add Tags or View Tags.

On the pipeline details page, you can either select Tags from the Actions menu, or select the Tags tab to view and manage tags.

Learn more about tagging.

Delete a pipeline

Ensure that your pipeline is stopped and in an inactive state before you delete the pipeline.

To delete a pipeline:
  1. On the Pipelines page, select the pipeline you want to delete.
  2. From the Actions menu on the pipeline's details page, select Delete.
  3. In the Delete pipeline dialog, confirm that you want to delete this pipeline, and then click Delete.
This action cannot be undone.