Adding a Data Flow Application Pipeline Step During Pipeline Creation

Use a Data Flow application as a pipeline step in a new Data Science pipeline.

Important

Review prerequisites at Before You Begin on the pipeline creation page.

This task documents the steps for adding a pipeline step from a Data Flow application when creating a pipeline by using the Oracle Cloud Console.

When creating and managing pipelines, we recommend that you use the ADS SDK. Alternatives to the Console or ADS include API and CLI. To use the API to create a pipeline, see CreatePipeline in the API Reference. To use the CLI to create a pipeline, see oci data-science pipeline create in the OCI CLI Command Reference.

  1. Follow the steps at Creating a Pipeline to open the Create pipeline page.
  2. On the Create pipeline page, select Add pipeline steps.
  3. Select From Data Flow applications and enter the following information.
    • Step name: Enter a unique name for the step. You can't repeat a step name in a pipeline.
    • Step description (Optional): Enter a step description, which can help you find step dependencies.
    • Depends on (Optional): If this step depends on another step, select one or more steps to run before this step.
    • Select a dataflow application compartment
    • Select a dataflow application
    • Default configuration (Optional):
      • Custom environment variable key (Optional): The environment variables for this pipeline step.
      • Value (Optional): The key's value.
      • Command line arguments (Optional): Enter the command line arguments that you want to use for running the pipeline step.
      • Maximum runtime (in minutes) (Optional): The maximum number of minutes that the pipeline step is allowed to run. The service cancels a pipeline run if its runtime exceeds the specified value. The maximum runtime is 30 days (43,200 minutes). We recommend that you configure a maximum runtime on all pipeline runs to prevent runaway pipeline runs.
    • Data Flow configuration: Select Configure to open the Configure Data Flow configuration panel and then enter the following information.
      • Driver shape
      • Driver OCPUs
      • Driver Memory (GB)
      • Executor shape
      • Executor OCPUs
      • Executor Memory (GB)
      • Number of executors
      • Compute shape parameterized
        • Driver shape parameterized
        • Driver OCPUs parameterized
        • Driver memory (GB) parameterized
        • Executor shape parameterized
        • Executor OCPUs parameterized
        • Executor memory (GB) parameterized
      • Enter the bucket path manually / Logs bucket URI
      • Object storage bucket name compartment
      • Object storage bucket name
      • Configuration
        • Key
        • Value
        • Warehouse bucket URI
      • Configure: Select to save entered information and go back to the Add pipeline step page.
    • Save: Select to save the step.

      The Create pipeline page reopens with the step added.

  4. Select Save.
The Create pipeline page reopens with the step added.
Continue following steps at Creating a Pipeline to define and create the new pipeline.