Pipeline and Step Parameters

Add parameters to pipelines and steps.

ML Pipelines introduces parameterization for pipeline execution, enabling dynamic configuration of step inputs, infrastructure, and behavior without changing underlying code or container images. Parameters let you define a clear interface for pipelines, similar to a function that has a signature that defines its parameters. Pipelines become reusable components whose behavior can be precisely controlled at runtime without changing their implementation.

Two new parameter types are introduced:
Pipeline Parameters
Defined at pipeline creation and provided at pipeline run start, they can be referenced across steps.
Step Output Parameters
Produced by steps at runtime (written to JSON files) and consumed by later steps.

With pipeline parameters, you can:

  • Dynamically set environment variables, command line arguments, and infrastructure properties.
  • Pass values from one step to another without hard-coding.
  • Reference implicit parameters (for example, pipeline.id or pipeline.region_identifier) automatically provided by the service.
  • Reuse step implementations across workflows. Parameters make it easier to reuse existing step implementations without modification, letting you combine them into both linear and parallel flows. You can adapt steps to fit into new, unforeseen workflows

Working with pipeline and step output parameters is easy. You can think of pipeline parameters as name-value pairs. You can refer to them from environment variables, command line arguments, and infrastructure configurations.

Parameter Types

Pipeline Parameters

  • Declared in pipeline definition (parameters object).

  • Passed at pipeline run creation (parametersOverride).

  • Accessible in steps by using:

    {{pipeline.parameters.<name>}}
  • Support for implicit parameters such as:

    • pipeline.id

    • pipeline.region_identifier

    • pipeline.time_accepted

Step Output Parameters

Note

For Data Flow integration, only the following items can be referenced in a Data Flow step configuration: pipeline-level parameters, implicit parameters, and output parameters from dependent Data Science job steps (ML_JOB, CUSTOM_SCRIPT, or CONTAINER).
  • Declared in stepParameters with outputParameterType (JSON).

  • Produced by writing values to files inside the step.

  • Accessed in later steps as:

    {{stepParameters.<stepName>.output.<paramName>}}
  • When a step refers to the step output parameters of another step, the step needs to state it directly in the dependencies (dependsOn field) or indirectly (transitively) by using dependencies of the step it depends on. That is, when a step is in the tree of steps that are depended on, its parameters can be used.

Parameterized Fields

Parameters can be referenced in:

  • Environment variables

  • Command line arguments

  • Infrastructure settings:

    Note

    For Data Flow integration, only the following items can be parameterized: environment variables, command-line arguments, driver shapes, driver OCPUs, driver memory, executor shapes, executor OCPUs, executor memory.
    • ocpusParameterized

    • memoryInGBsParameterized

    • blockStorageSizeInGBsParameterized

    • shapeName
  • Step run names

Limits

  • No parameterization of:

    • OCID fields (for example, subnetId).

    • Storage mounts
  • Maximum number of placeholders:

    • Five per environment variable value

    • Ten in command line arguments

    • One in other fields

  • Output step parameters can only be referenced in the configuration by steps that explicitly depend on the producing step or indirectly (transitively) by using dependencies of the step it depends on. That is, when a step is in the tree of steps that are depended on, its parameters can be used.
  • Step parameters aren't supported for Data Flow steps.