Pipeline and Step Parameters
Add parameters to pipelines and steps.
ML Pipelines introduces parameterization for pipeline execution, enabling dynamic configuration of step inputs, infrastructure, and behavior without changing underlying code or container images. Parameters let you define a clear interface for pipelines, similar to a function that has a signature that defines its parameters. Pipelines become reusable components whose behavior can be precisely controlled at runtime without changing their implementation.
- Pipeline Parameters
- Defined at pipeline creation and provided at pipeline run start, they can be referenced across steps.
- Step Output Parameters
- Produced by steps at runtime (written to JSON files) and consumed by later steps.
With pipeline parameters, you can:
- Dynamically set environment variables, command line arguments, and infrastructure properties.
- Pass values from one step to another without hard-coding.
- Reference implicit parameters (for example,
pipeline.idorpipeline.region_identifier) automatically provided by the service. - Reuse step implementations across workflows. Parameters make it easier to reuse existing step implementations without modification, letting you combine them into both linear and parallel flows. You can adapt steps to fit into new, unforeseen workflows
Working with pipeline and step output parameters is easy. You can think of pipeline parameters as name-value pairs. You can refer to them from environment variables, command line arguments, and infrastructure configurations.
Parameter Types
Pipeline Parameters
-
Declared in pipeline definition (
parametersobject). -
Passed at pipeline run creation (
parametersOverride). -
Accessible in steps by using:
{{pipeline.parameters.<name>}} -
Support for implicit parameters such as:
-
pipeline.id -
pipeline.region_identifier -
pipeline.time_accepted
-
Step Output Parameters
For Data Flow integration, only the following items can be referenced in a Data Flow step configuration: pipeline-level parameters, implicit parameters, and output parameters from dependent Data Science job steps (
ML_JOB, CUSTOM_SCRIPT, or CONTAINER).-
Declared in
stepParameterswithoutputParameterType(JSON). -
Produced by writing values to files inside the step.
-
Accessed in later steps as:
{{stepParameters.<stepName>.output.<paramName>}} -
When a step refers to the step output parameters of another step, the step needs to state it directly in the dependencies (
dependsOnfield) or indirectly (transitively) by using dependencies of the step it depends on. That is, when a step is in the tree of steps that are depended on, its parameters can be used.
Parameterized Fields
Parameters can be referenced in:
-
Environment variables
-
Command line arguments
-
Infrastructure settings:
Note
For Data Flow integration, only the following items can be parameterized: environment variables, command-line arguments, driver shapes, driver OCPUs, driver memory, executor shapes, executor OCPUs, executor memory.-
ocpusParameterized -
memoryInGBsParameterized -
blockStorageSizeInGBsParameterized shapeName
-
-
Step run names
Limits
-
No parameterization of:
-
OCID fields (for example,
subnetId). - Storage mounts
-
-
Maximum number of placeholders:
-
Five per environment variable value
-
Ten in command line arguments
-
One in other fields
-
- Output step parameters can only be referenced in the configuration by steps that explicitly depend on the producing step or indirectly (transitively) by using dependencies of the step it depends on. That is, when a step is in the tree of steps that are depended on, its parameters can be used.
-
Step parameters aren't supported for Data Flow steps.
Tasks
Following are tasks related to pipelines and step parameters.