Adding a Container Pipeline Step During Pipeline Creation

Use a container as a pipeline step in a new Data Science pipeline.

Important

Review prerequisites at Before You Begin on the pipeline creation page.

This task documents the steps for adding a pipeline step from a container when creating a pipeline by using the Oracle Cloud Console.

When creating and managing pipelines, we recommend that you use the ADS SDK. Alternatives to the Console or ADS include API and CLI. To use the API to create a pipeline, see CreatePipeline in the API Reference. To use the CLI to create a pipeline, see oci data-science pipeline create in the OCI CLI Command Reference.

  1. Follow the steps at Creating a Pipeline to open the Create pipeline page.
  2. On the Create pipeline page, select Add pipeline steps.
  3. Select From container and enter the following information.

    Optionally, when defining pipeline steps, you can select to use Bring Your Own Container. For more information, see Bring Your Own Container (BYOC) for Pipelines.

    • Step name: Enter a unique name for the step. You can't repeat a step name in a pipeline.
    • Step description (Optional): Enter a step description, which can help you find step dependencies.
    • Depends on (Optional): If this step depends on another step, select one or more steps to run before this step.
    • Container configuration
      • Configure container environment: Select Configure to open the Configure container environment panel and then enter the following information.
        • Repository compartment
        • Repository
        • Image
        • Entrypoint (Optional)
        • CMD (Optional): Use CMD as arguments to the ENTRYPOINT or the only command to run in the absence of an ENTRYPOINT.
        • Image digest (Optional)
        • Signature ID (Optional): If using signature verification, enter the OCID of the image signature. Example: ocid1.containerimagesignature.oc1.iad.aaaaaaaaab....
    • Upload job artifact: Drag a step artifact into the box, or select the box to navigate to the file for selection.

      This step is optional only if BYOC is configured.

    • Default configuration (Optional):
      Note

      The step needs to ensure that the specified file (for example, /home/datascience/output.json) is populated with a valid JSON defining the specified variables. For example:
      { "message":"Hello John!", "ocpu": 2, "memory": 10 }
      • Custom environment variable key (Optional): The environment variables for this pipeline step.
      • Value (Optional): The key's value.
      • Command line arguments (Optional): Enter the command line arguments that you want to use for running the pipeline step.
      • Maximum runtime (in minutes) (Optional): The maximum number of minutes that the pipeline step is allowed to run. The service cancels a pipeline run if its runtime exceeds the specified value. The maximum runtime is 30 days (43,200 minutes). We recommend that you configure a maximum runtime on all pipeline runs to prevent runaway pipeline runs.
    • Output parameters (Optional):
      • Output parameter type: Select JSON.
      • Parameter name: Enter a parameter name.
      • Output file name: Select the output file name in which the step stores the output parameters. For example: /home/datascience/output.json.
    • Change the Compute shape by selecting Change shape. Then, follow these steps in the Select compute shape panel.
      Note

      For the AMD shape, you can use the default or set the number of OCPUs and memory.
      • Select an instance type.
      • Select a shape series.
      • Select one of the supported Compute shapes in the series. Select the shape that best suits how you want to use the resource.
      • Expand the selected shape to configure OCPUs and memory.
        • Number of OCPUs
        • Amount of memory (GB): For each OCPU, select up to 64 GB of memory and a maximum total of 512 GB. The minimum amount of memory allowed is either 1 GB or a value matching the number of OCPUs, whichever is greater.
        • Enable Burstable Shape: Select if using burstable VMs, and then for Baseline utilization per OCPU, select the percentage of OCPUs that you usually want to use. The supported values are 12.5% and 50%. (For model deployments, only the value of 50% is supported.)
      • Select Select shape.
    • Compute shape parameterized (Optional)
      • Shape parameterized
      • Ocpus parameterized
      • MemoryInGBs parameterized
    • Block Storage: Enter the amount of storage that you want to use between 50 GB and 10, 240 GB (10 TB). You can change the value by 1 GB increments. The default value is 100 GB.
    • Networking resources (available for Standard compute infrastructure type): Select the relevant option.
      • Default Networking: Restricts traffic to Oracle services only. The system uses the existing service-managed network. The workload is attached by using a secondary VNIC to a preconfigured, service-managed VCN, and subnet. This provided subnet lets egress to the public internet through a NAT gateway, and access to other Oracle Cloud services through a service gateway.

        If you need access only to the public internet and OCI services, we recommend using this option. It doesn't require you to create networking resources or write policies for networking permissions.

      • Custom Networking: Select the VCN and subnet (by compartment) that you want to use.

        For egress access to the public internet, use a private subnet with a route to a NAT gateway.

        Note

        • Custom networking must be used to use a file storage mount.
        • Switching from custom networking to managed networking isn't supported after creation.
        • If you see the banner The specified subnet is not accessible. Select a different subnet., then create policies for custom networking.
    • Storage mounts (Optional):
      • File storage mounts (Optional): Select Add file storage mount and enter the following information.
        • Compartment: Select the compartment that contains the target that you want to mount.
        • Mount target: The mount target that you want to use.
        • Export path: The export path that you want to use.
        • Destination path and directory: Enter the path to use for mounting the storage.

          The path must start with an alphanumeric character. The destination directory must be unique across the storage mounts provided. The allowed characters are alphanumerics, hyphen ( - ) and underscore ( _ ).

          You can specify the full path, such as /opc/storage-directory. If only a directory is specified, such as /storage-directory, then it's mounted under the default /mnt directory. You can't specify OS specific directories, such as /bin or /etc.

      • Object storage mounts (Optional): Select Add object storage mount and enter the following information.
        • Compartment: Select the compartment that contains the bucket that you want to mount.
        • Bucket: Select the bucket that you want to use.
        • Object name prefix (Optional): Object name prefix. The prefix must start with an alphanumeric character. The allowed characters are alphanumerics, slash ( / ), hyphen ( - ) and underscore ( _ ).
        • Destination path and directory: Enter the path to use for mounting the storage.

          The path must start with an alphanumeric character. The destination directory must be unique across the storage mounts provided. The allowed characters are alphanumerics, hyphen ( - ) and underscore ( _ ).

          You can specify the full path, such as /opc/storage-directory. If only a directory is specified, such as /storage-directory, then it's mounted under the default /mnt directory. You can't specify OS specific directories, such as /bin or /etc.

        Note

        If using custom networking:
        1. Create the service gateway in the VCN.
        2. For the route table configurations in the private subnet, add the service gateway.
        3. Change the egress rules of security list of the required subnet to let traffic to all services in the network.
    • Save: Select to save the step.

      The Create pipeline page reopens with the step added.

  4. Select Save.
The Create pipeline page reopens with the step added.
Continue following steps at Creating a Pipeline to define and create the new pipeline.