Control Pipelines (Start, Stop, Drop, or Reset a Pipeline)

After you create and test a pipeline, you control a pipeline by starting, stopping, or dropping the pipeline. You can also reset a pipeline.

• Start a Pipeline
  After you create a pipeline, you can start the pipeline.
• Stop a Pipeline
  Use STOP_PIPELINE to stop a pipeline. When a pipeline is stopped, no future jobs are scheduled for the pipeline.
• Drop a Pipeline
  The procedure DROP_PIPELINE drops an existing pipeline.
• Reset a Pipeline
  Use the reset pipeline operation to clear the record of the pipeline to the initial state.

Start a Pipeline

After you create a pipeline, you can start the pipeline.

When a pipeline is started the pipeline runs continuously in a scheduled job. The pipeline's scheduled job repeats, either by default every 15 minutes or at the interval you set with the interval attribute.

1. Start a pipeline.

   BEGIN
     DBMS_CLOUD_PIPELINE.START_PIPELINE(
         pipeline_name => 'EMPLOYEE_PIPELINE'
     );
   END;
   /

   By default, a pipeline job begins immediately, as soon as the pipeline is started. To start a pipeline job at a later time, specify a valid future date or timestamp using the start_date parameter.

   See START_PIPELINE Procedure for more information.

2. Verify that the pipeline is started.

   For example:

   
   SELECT pipeline_name, status from USER_CLOUD_PIPELINES
      WHERE pipeline_name = 'EMPLOYEE_PIPELINE';
   
   PIPELINE_NAME            STATUS  
   ------------------------ ------- 
   EMPLOYEE_PIPELINE        STARTED

Stop a Pipeline

Use STOP_PIPELINE to stop a pipeline. When a pipeline is stopped, no future jobs are scheduled for the pipeline.

By default currently running jobs complete when you stop a pipeline. Set the force parameter to TRUE to terminate any running jobs and stop the pipeline immediately.

1. Stop a pipeline.

   BEGIN
     DBMS_CLOUD_PIPELINE.STOP_PIPELINE(
         pipeline_name => 'EMPLOYEE_PIPELINE'
     );
   END;
   /

   See STOP_PIPELINE Procedure for more information.

2. Verify that the pipeline is stopped.

   
   SELECT pipeline_name, status from USER_CLOUD_PIPELINES
      WHERE pipeline_name = 'EMPLOYEE_PIPELINE';
   
   PIPELINE_NAME            STATUS  
   ------------------------ ------- 
   EMPLOYEE_PIPELINE        STOPPED

See STOP_PIPELINE Procedure for more information.

Drop a Pipeline

The procedure DROP_PIPELINE drops an existing pipeline.

If a pipeline has been started, it must be stopped before the pipeline can be dropped. See STOP_PIPELINE Procedure for more information.

In order to drop a pipeline that is started, set the force parameter to TRUE to terminate any running jobs and drop the pipeline immediately

1. Drop a pipeline.

   BEGIN
     DBMS_CLOUD_PIPELINE.DROP_PIPELINE(
         pipeline_name => 'EMPLOYEE_PIPELINE'
     );
   END;
   /

2. Verify the pipeline is dropped.

   SELECT pipeline_name, status from USER_CLOUD_PIPELINES
      WHERE pipeline_name = 'EMPLOYEE_PIPELINE';
   
   No rows selected

See DROP_PIPELINE Procedure for more information.

Reset a Pipeline

Use the reset pipeline operation to clear the record of the pipeline to the initial state.

Note:

You can optionally use reset pipeline to purge data in the database table associated with a load pipeline or to remove files in object store for an export pipeline. Usually this option is used when you are testing a pipeline during pipeline development.

Reset pipeline operates as follows:

• Load Pipeline: For a load pipeline, resetting the pipeline clears the record of the files being loaded by the pipeline. When you call either START_PIPELINE or RUN_PIPELINE_ONCE after resetting a load pipeline, the pipeline repeats the data load and includes all the files present in the object store location.

  When purge_data is set to TRUE, the procedure truncates the data in the database table.

• Export Pipeline: For an export pipeline, resetting the pipeline clears the last tracked data in the database table. When you call either START_PIPELINE or RUN_PIPELINE_ONCE after resetting an export pipeline, the pipeline repeats exporting data from the table or query.

  When purge_data set to TRUE, the procedure deletes existing files in the object store location specified with the location attribute.

To reset a pipeline:

1. Stop the pipeline you want to reset.

   A data pipeline must be in stopped state to reset it. See STOP_PIPELINE Procedure for more information.

2. Reset the pipeline.

   BEGIN
        DBMS_CLOUD_PIPELINE.RESET_PIPELINE(
           pipeline_name => 'EMPLOYEE_PIPELINE',
           purge_data => TRUE);
   END;
   /

   Only use the purge_data parameter with value TRUE if you want to clear data in your database table, for a load pipeline, or clear files in object store for an export pipeline. Usually this option is used when you are testing a pipeline during pipeline development.

   See RESET_PIPELINE Procedure for more information.