Adding Pipeline Parameters During Creation

Add pipeline parameters to a new Data Science pipeline.

Important

Review prerequisites at Before You Begin on the pipeline creation page. For limits and other important information about parameters, see Pipeline and Step Parameters.

This task documents the steps for adding pipeline parameters when creating a pipeline by using the Oracle Cloud Console.

When creating and managing pipelines, we recommend that you use the ADS SDK. Alternatives to the Console or ADS include API and CLI. To use the API to create a pipeline, see CreatePipeline in the API Reference. To use the CLI to create a pipeline, see oci data-science pipeline create in the OCI CLI Command Reference. See also Example JSON File for Use with CLI.

  1. Follow the steps at Creating a Pipeline to open the Create pipeline page.
  2. On the Create pipeline page, under Parameters, define pipeline parameters:
    • Custom environment variable key (Optional): The environment variables for this pipeline step.
    • Value (Optional): The key's value.
  3. If creating a step, under Output parameters, define output step parameters:

    This step is only for creating a pipeline step which is at a different level than pipeline.

    • Output parameter type: Select JSON.
    • Parameter name: Enter a parameter name.
    • Output file name: Select the output file name in which the step stores the output parameters. For example: /home/datascience/output.json.
  4. To define the compute shape with parameters, select Compute shape parameterized.
  5. Update other fields as needed.
    For field descriptions, see Creating a Pipeline.
  6. Select Create pipeline.

Example JSON File for Use with CLI

Use the oci data-science pipeline create command and required parameters to create a pipeline:

oci data-science pipeline create --from-json file://create.pipeline.json

For a complete list of parameters and values for CLI commands, see the CLI Command Reference.

The following example, create.pipeline.json, shows how to define a basic pipeline that takes two pipeline parameters, uses them to print a personalized greeting, and produces three output parameters that are used by the second step:

{
  "displayName": "Parameter Demo",
  "description": "Hello world pipeline with parameters",
  "compartmentId": "ocid1.compartment.oc1......",
  "projectId": "ocid1.datascienceproject.oc1......",
                            
  # defining 2 parameters that users can use to influence behavior of this pipeline
  "parameters": {
  "greeting": "Hello",
  "name": "John"
  },
  "infrastructureConfigurationDetails": {
   "blockStorageSizeInGBs": 50,
   "shapeConfigDetails": {
    "memoryInGBs": 16.0,
    "ocpus": 3.0
   },
   "shapeName": "VM.Standard.E5.Flex",
   "subnetId": null
  },
  "logConfigurationDetails": {
   "enableAutoLogCreation": true,
   "enableLogging": true,
   "logGroupId": "ocid1.loggroup......",
   "logId": null
  },
  "stepDetails": [
   {
    "stepName": "say_hello",
    "stepType": "CUSTOM_SCRIPT",
    "stepConfigurationDetails": {
     "type": "DEFAULT",
                
     # command line arguments and environment variables can refer to parameters (both pipeline and step output parameters)
     "commandLineArguments": "--person {{pipeline.parameters.name}}",
     "environmentVariables": {
      "GREETING": "{{pipeline.parameters.greeting}}"
     }
    },
    "stepParameters": {
     "parameterType": "DEFAULT",
     "output": {
      "outputParameterType": "JSON",
      "parameterNames": [
       "message",
       "ocpu",
       "memory"
      ],
      "outputFile": "/home/datascience/output.json"
     }
    }
   },
   {
    "stepName": "echo_message",
    "stepType": "CUSTOM_SCRIPT",
                            
    # to use output parameters of a step, the step must be included in the dependencies
    "dependsOn": [
     "say_hello"
    ],
    "stepConfigurationDetails": {
     "type": "DEFAULT",
                            
     # step output parameters can be used to configure next steps with on the fly provided values in a particular pipeline run
     "commandLineArguments": "--msg {{stepParameters.say_hello.output.message}}"
    },
    "stepInfrastructureConfigurationDetails": {
     "blockStorageSizeInGBs": 50,
     "shapeName": "VM.Standard.E5.Flex",
     "shapeConfigDetails": {
      "ocpus": 1,
      "memoryInGBs": 4,
                            
      # even infrastructure can be configured on the fly by refering to parameters
      "ocpusParameterized": "{{stepParameters.say_hello.output.ocpu}}",
      "memoryInGBsParameterized": "{{stepParameters.say_hello.output.memory}}"
     }
    }
   }
  ]
 }