Adding Pipeline Parameters During Creation
Add pipeline parameters to a new Data Science pipeline.
Review prerequisites at Before You Begin on the pipeline creation page. For limits and other important information about parameters, see Pipeline and Step Parameters.
This task documents the steps for adding pipeline parameters when creating a pipeline by using the Oracle Cloud Console.
When creating and managing pipelines, we recommend that you use the ADS SDK. Alternatives to the Console or ADS include API and CLI. To use the API to create a pipeline, see CreatePipeline in the API Reference. To use the CLI to create a pipeline, see oci data-science pipeline create in the OCI CLI Command Reference. See also Example JSON File for Use with CLI.
Example JSON File for Use with CLI
Use the oci data-science pipeline create command and required parameters to create a pipeline:
oci data-science pipeline create --from-json file://create.pipeline.jsonFor a complete list of parameters and values for CLI commands, see the CLI Command Reference.
The following example, create.pipeline.json, shows how to define a basic pipeline that takes two pipeline parameters, uses them to print a personalized greeting, and produces three output parameters that are used by the second step:
{
"displayName": "Parameter Demo",
"description": "Hello world pipeline with parameters",
"compartmentId": "ocid1.compartment.oc1......",
"projectId": "ocid1.datascienceproject.oc1......",
# defining 2 parameters that users can use to influence behavior of this pipeline
"parameters": {
"greeting": "Hello",
"name": "John"
},
"infrastructureConfigurationDetails": {
"blockStorageSizeInGBs": 50,
"shapeConfigDetails": {
"memoryInGBs": 16.0,
"ocpus": 3.0
},
"shapeName": "VM.Standard.E5.Flex",
"subnetId": null
},
"logConfigurationDetails": {
"enableAutoLogCreation": true,
"enableLogging": true,
"logGroupId": "ocid1.loggroup......",
"logId": null
},
"stepDetails": [
{
"stepName": "say_hello",
"stepType": "CUSTOM_SCRIPT",
"stepConfigurationDetails": {
"type": "DEFAULT",
# command line arguments and environment variables can refer to parameters (both pipeline and step output parameters)
"commandLineArguments": "--person {{pipeline.parameters.name}}",
"environmentVariables": {
"GREETING": "{{pipeline.parameters.greeting}}"
}
},
"stepParameters": {
"parameterType": "DEFAULT",
"output": {
"outputParameterType": "JSON",
"parameterNames": [
"message",
"ocpu",
"memory"
],
"outputFile": "/home/datascience/output.json"
}
}
},
{
"stepName": "echo_message",
"stepType": "CUSTOM_SCRIPT",
# to use output parameters of a step, the step must be included in the dependencies
"dependsOn": [
"say_hello"
],
"stepConfigurationDetails": {
"type": "DEFAULT",
# step output parameters can be used to configure next steps with on the fly provided values in a particular pipeline run
"commandLineArguments": "--msg {{stepParameters.say_hello.output.message}}"
},
"stepInfrastructureConfigurationDetails": {
"blockStorageSizeInGBs": 50,
"shapeName": "VM.Standard.E5.Flex",
"shapeConfigDetails": {
"ocpus": 1,
"memoryInGBs": 4,
# even infrastructure can be configured on the fly by refering to parameters
"ocpusParameterized": "{{stepParameters.say_hello.output.ocpu}}",
"memoryInGBsParameterized": "{{stepParameters.say_hello.output.memory}}"
}
}
}
]
}