Creating a Pipeline

Create a Data Science pipeline to run a task. You can create pipelines by using the ADS SDK, OCI Console, or the OCI SDK. Using ADS for creating pipelines can make developing the pipeline, the steps, and the dependencies easier. ADS supports reading and writing the pipeline to and from a YAML file. You can use ADS to view a visual representation of the pipeline. We recommend that you use ADS to create and manage pipeline using code.

Before You Begin

    1. On the Projects list page, select the project that contains the pipelines that you want to work with. If you need help finding the list page or the project, see Listing Projects.
    2. On the project details page, select Pipelines.
    3. Select Create pipeline.
    4. On the Create pipeline page, enter the following information.
      • Compartment: Select the compartment to store the pipeline in.
      • Name (Optional): Enter a name for the pipeline (limit of 255 characters). If you don't provide a name, a name is automatically generated. Example: pipeline2022808222435
      • Description (Optional): Enter a description for the pipeline.
      • Pipeline steps: For each pipeline step that you want to add to the pipeline, select Add pipeline steps to open the Add pipeline step panel and then follow the procedure for the type of pipeline step that you want.
      • Default configuration (Optional): Enter one or more custom environment variables with their key values.
        • Custom environment variable key (Optional): Enter a custom environment variable key.
        • Value (Optional): Enter the value for the key.
        • Command line arguments (Optional): Enter the command line arguments that you want to use for running the pipeline step.
        • Maximum runtime (in minutes) (Optional): The maximum number of minutes that steps in this pipeline are allowed to run. The service cancels a pipeline run if its runtime exceeds the specified value. The maximum runtime is 30 days (43,200 minutes). We recommend that you configure a maximum runtime on all pipeline runs to prevent runaway pipeline runs.
      • Parameters (Optional)
        • Custom parameter key
        • Value
      • Compute shape (Optional)
        • Change the Compute shape by selecting Change shape. Then, follow these steps in the Select compute shape panel.
          Note

          For the AMD shape, you can use the default or set the number of OCPUs and memory.
          • Select an instance type.
          • Select a shape series.
          • Select one of the supported Compute shapes in the series. Select the shape that best suits how you want to use the resource.
          • Expand the selected shape to configure OCPUs and memory.
            • Number of OCPUs
            • Amount of memory (GB): For each OCPU, select up to 64 GB of memory and a maximum total of 512 GB. The minimum amount of memory allowed is either 1 GB or a value matching the number of OCPUs, whichever is greater.
            • Enable Burstable Shape: Select if using burstable VMs, and then for Baseline utilization per OCPU, select the percentage of OCPUs that you usually want to use. The supported values are 12.5% and 50%. (For model deployments, only the value of 50% is supported.)
          • Select Select shape.
      • Compute shape parameterized (Optional)
        • Shape parameterized
        • Ocpus parameterized
        • MemoryInGBs parameterized
      • Block Storage: Enter the amount of storage that you want to use between 50 GB and 10, 240 GB (10 TB). You can change the value by 1 GB increments. The default value is 100 GB.
      • Networking resources (available for Standard compute infrastructure type): Select the relevant option.
        • Default Networking: Restricts traffic to Oracle services only. The system uses the existing service-managed network. The workload is attached by using a secondary VNIC to a preconfigured, service-managed VCN, and subnet. This provided subnet lets egress to the public internet through a NAT gateway, and access to other Oracle Cloud services through a service gateway.

          If you need access only to the public internet and OCI services, we recommend using this option. It doesn't require you to create networking resources or write policies for networking permissions.

        • Custom Networking: Select the VCN and subnet (by compartment) that you want to use.

          For egress access to the public internet, use a private subnet with a route to a NAT gateway.

          Note

          • Custom networking must be used to use a file storage mount.
          • Switching from custom networking to managed networking isn't supported after creation.
          • If you see the banner The specified subnet is not accessible. Select a different subnet., then create policies for custom networking.
      • Logging (Optional)
        • Enable logging: Log messages.
        • Log group compartment: Select the compartment that contains the log group.
        • Log group: Select the log group.
        • Select automatic log creation or a specified log:
          • Enable automatic log creation
          • Select a log
      • Storage mounts (Optional):
        • File storage mounts (Optional): Select Add file storage mount and enter the following information.
          • Compartment: Select the compartment that contains the target that you want to mount.
          • Mount target: The mount target that you want to use.
          • Export path: The export path that you want to use.
          • Destination path and directory: Enter the path to use for mounting the storage.

            The path must start with an alphanumeric character. The destination directory must be unique across the storage mounts provided. The allowed characters are alphanumerics, hyphen ( - ) and underscore ( _ ).

            You can specify the full path, such as /opc/storage-directory. If only a directory is specified, such as /storage-directory, then it's mounted under the default /mnt directory. You can't specify OS specific directories, such as /bin or /etc.

        • Object storage mounts (Optional): Select Add object storage mount and enter the following information.
          • Compartment: Select the compartment that contains the bucket that you want to mount.
          • Bucket: Select the bucket that you want to use.
          • Object name prefix (Optional): Object name prefix. The prefix must start with an alphanumeric character. The allowed characters are alphanumerics, slash ( / ), hyphen ( - ) and underscore ( _ ).
          • Destination path and directory: Enter the path to use for mounting the storage.

            The path must start with an alphanumeric character. The destination directory must be unique across the storage mounts provided. The allowed characters are alphanumerics, hyphen ( - ) and underscore ( _ ).

            You can specify the full path, such as /opc/storage-directory. If only a directory is specified, such as /storage-directory, then it's mounted under the default /mnt directory. You can't specify OS specific directories, such as /bin or /etc.

          Note

          If using custom networking:
          1. Create the service gateway in the VCN.
          2. For the route table configurations in the private subnet, add the service gateway.
          3. Change the egress rules of security list of the required subnet to let traffic to all services in the network.
      • Retention configuration (Optional): Specify number of days to retain pipeline run. Default value: 90
      • Advanced options
        • Tags: Add tags to the pipeline. If you have permissions to create a resource, then you also have permissions to apply free-form tags to that resource. To apply a defined tag, you must have permissions to use the tag namespace. For more information about tagging, see Resource Tags. If you're not sure whether to apply tags, skip this option or ask an administrator. You can apply tags later.
    5. Select Create pipeline.

      After the pipeline is in an active state, you can use pipeline runs to repeatedly run the pipeline.

  • These environment variables control the pipeline run.

    You can use the OCI CLI to create a pipeline as in this Python example:

    1. Create a pipeline:

      The following parameters are available to use in the payload:

      Parameter name Required Description
      Pipeline (top level)
      projectId Required The project OCID to create the pipeline in.
      compartmentId Required The compartment OCID to the create the pipeline in.
      displayName Optional The name of the pipeline.
      infrastructureConfigurationDetails Optional

      Default infrastructure (compute) configuration to use for all the pipeline steps, see infrastructureConfigurationDetails for details on the supported parameters.

      Can be overridden by the pipeline run configuration.

      logConfigurationDetails Optional

      Default log to use for the all the pipeline steps, see logConfigurationDetails for details on the supported parameters.

      Can be overridden by the pipeline run configuration.

      configurationDetails Optional

      Default configuration for the pipeline run, see configurationDetails for details on supported parameters.

      Can be overridden by the pipeline run configuration.

      freeformTags Optional Tags to add to the pipeline resource.
      stepDetails
      stepName Required Name of the step. Must be unique in the pipeline.
      description Optional Free text description for the step.
      stepType Required CUSTOM_SCRIPT or ML_JOB
      jobId Required* For ML_JOB steps, this is the job OCID to use for the step run.
      stepInfrastructureConfigurationDetails Optional*

      Default infrastructure (Compute) configuration to use for this step, see infrastructureConfigurationDetails for details on the supported parameters.

      Can be overridden by the pipeline run configuration.

      *Must be defined on at least one level (precedence based on priority, 1 being highest):

      1 pipeline run and/or

      2 step and/or

      3 pipeline

      stepConfigurationDetails Optional*

      Default configuration for the step run, see configurationDetails for details on supported parameters.

      Can be overridden by the pipeline run configuration.

      *Must be defined on at least one level (precedence based on priority, 1 being highest):

      1 pipeline run and/or

      2 step and/or

      3 pipeline

      dependsOn Optional List of steps that must be completed before this step begins. This creates the pipeline workflow dependencies graph.
      infrastructureConfigurationDetails
      shapeName Required Name of the Compute shape to use. For example, VM.Standard2.4.
      blockStorageSizeInGBs Required Number of GBs to use as the attached storage for the VM.
      logConfigurationDetails
      enableLogging Required Define to use logging.
      logGroupId Required Log group OCID to use for the logs. The log group must be created and available when the pipeline runs
      logId Optional* Log OCID to use for the logs when not using the enableAutoLogCreation parameter.
      enableAutoLogCreation Optional If set to True, a log for each pipeline run is created.
      configurationDetails
      type Required Only DEFAULT is supported.
      maximumRuntimeInMinutes Optional Time limit in minutes for the pipeline to run.
      environmentVariables Optional

      Environment variables to provide for the pipeline step runs.

      For example:

      "environmentVariables": {
      
       "CONDA_ENV_TYPE": "service"
      
      }

      Review the list of service supported environment variables.

      pipeline_payload = {
          "projectId": "<project_id>",
          "compartmentId": "<compartment_id>",
          "displayName": "<pipeline_name>",
          "pipelineInfrastructureConfigurationDetails": {
              "shapeName": "VM.Standard2.1",
              "blockStorageSizeInGBs": "50"
          },
          "pipelineLogConfigurationDetails": {
              "enableLogging": True,
              "logGroupId": "<log_group_id>",
              "logId": "<log_id>"
          },
          "pipelineDefaultConfigurationDetails": {
              "type": "DEFAULT",
              "maximumRuntimeInMinutes": 30,
              "environmentVariables": {
                  "CONDA_ENV_TYPE": "service",
                  "CONDA_ENV_SLUG": "classic_cpu"
              }
          },
          "stepDetails": [
              {
                  "stepName": "preprocess",
                  "description": "Preprocess step",
                  "stepType": "CUSTOM_SCRIPT",
                  "stepInfrastructureConfigurationDetails": {
                      "shapeName": "VM.Standard2.4",
                      "blockStorageSizeInGBs": "100"
                  },
                  "stepConfigurationDetails": {
                      "type": "DEFAULT",
                      "maximumRuntimeInMinutes": 90
                      "environmentVariables": {
                          "STEP_RUN_ENTRYPOINT": "preprocess.py",
                          "CONDA_ENV_TYPE": "service",
                          "CONDA_ENV_SLUG": "onnx110_p37_cpu_v1"
                  }
              },
              {
                  "stepName": "postprocess",
                  "description": "Postprocess step",
                  "stepType": "CUSTOM_SCRIPT",
                  "stepInfrastructureConfigurationDetails": {
                      "shapeName": "VM.Standard2.1",
                      "blockStorageSizeInGBs": "80"
                  },
                  "stepConfigurationDetails": {
                      "type": "DEFAULT",
                      "maximumRuntimeInMinutes": 60
                  },
                  "dependsOn": ["preprocess"]
              },
          ],
          "freeformTags": {
              "freeTags": "cost center"
          }
      }
      pipeline_res = dsc.create_pipeline(pipeline_payload)
      pipeline_id = pipeline_res.data.id

      Until all pipeline steps artifacts are uploaded, the pipeline is in the CREATING state.

    2. Upload a step artifact:

      After an artifact is uploaded, it can't be changed.

      fstream = open(<file_name>, "rb")
      dsc.create_step_artifact(pipeline_id, step_name, fstream, content_disposition=f"attachment; filename={<file_name>}")
    3. Update a pipeline:

      You can only update a pipeline when it's in an ACTIVE state.

      update_pipeline_details = {
      "displayName": "pipeline-updated"
      }
      self.dsc.update_pipeline(<pipeline_id>, <update_pipeline_details>)
    4. Start pipeline run:
      pipeline_run_payload = {
      "projectId": project_id,
      "displayName": "pipeline-run",
      "pipelineId": <pipeline_id>,
      "compartmentId": <compartment_id>,
      }
      dsc.create_pipeline_run(pipeline_run_payload)
  • The ADS SDK is also a publicly available Python library that you can install with this command:

    pip install oracle-ads

    You can use the ADS SDK to create and run pipelines.

Creating Pipelines with Custom Networking Using APIs

You can select custom networking when creating a pipeline. Use a custom network that you've already created in the pipeline to give you extra flexibility on the network.

Provide subnet-id in the infrastructure-configuration-details to use a custom subnet on the pipeline level. For example:

"infrastructure-configuration-details": {
      "block-storage-size-in-gbs": 50,
      "shape-config-details": {
        "memory-in-gbs": 16.0,
        "ocpus": 1.0
      },
      "shape-name": "VM.Standard.E4.Flex",
      "subnet-id": "ocid1.subnet.oc1.iad.aaaaaaaa5lzzq3fyypo6x5t5egplbfyxf2are6k6boop3vky5t4h7g35xkoa"
}

Or in the step-container-configuration-details to use a custom subnet for a particular step. For example:

"step-infrastructure-configuration-details": {
          "block-storage-size-in-gbs": 50,
          "shape-config-details": {
            "memory-in-gbs": 16.0,
            "ocpus": 1.0
          },
          "shape-name": "VM.Standard.E4.Flex",
          "subnet-id": "ocid1.subnet.oc1.iad.aaaaaaaa5lzzq3fyypo6x5t5egplbfyxf2are6k6boop3vky5t4h7g35xkoa"
},