Parameterization

This section covers parameters and how they are used in jobs and tasks.

Topics:

• About Parameters
• Add Parameters to a Job
• Delete Parameters from a Job
• Add Parameters to a Task
• Delete Parameters from a Task
• Run a Job with Different Parameter Values

About Parameters

You can customize job runs by passing parameters that alter the behavior of the job, task, or job run.

Parameters can be provided at three different levels of a workflow: job level, task level, and job run level. Parameters take the following precedence in the case of conflicts: Job Run > Task > Job.

1. Job Run Level Parameters: Job run level parameters specified in the job run payload take precedence over task-level parameters defined in the job configuration. This means that any parameters specified during job run override the task-specific defaults.
2. Task Level Parameters: If job run parameters for a specific task are not provided, the task uses the parameters defined at the task level in the job configuration. Job-level parameters with the same name override task-level parameters for that specific task. So if you want a task specific parameter, you should name it differently then the job parameter.
3. Job Level Parameters: If neither task-level nor runtime parameters are provided, the default values set at the job level apply. Job-level parameters are considered defaults that can be used if no more specific parameters are available.

Note:

Job parameters are immutable in task contexts. This means if there is a job with parameter JobParamA with resolved value JobParamRuntimeValueA then TaskA execution cannot change the value of JobParamA. The value of JobParamA remains JobParamRuntimeValueA for all tasks and the entire job run. As a result, if you wanted to share information between tasks you could use intermediate storage or output parameters to achieve that.

When task names, task value keys, or job parameter names contain special characters (such as !@$%), you must surround those identifiers with backticks (` `). Only alphanumeric and underscore characters are usable without surrounding the identifier in backticks.

For example:

{
  "VariableWithSpecialChars": "{{job.parameters.`param$@`}}"
}

System Parameters are templated parameters whose value is provided by the system as part of workflow runs and subsequent task runs. You don't have to provide any value, default or otherwise, for these templated parameters. AI Data Platform has a fixed list of valid templated parameters / dynamic value references that are supported in workflow. System parameters are entered by surrounding them with two curly brackets. For, example {{job.id}}.

Table 12-1 Supported System Parameters

Parameter	Description
{{hub.id}}	The unique identifier assigned to the hub
{{hub.region}}	The region of the hub
{{workspace.id}}	The unique identifier assigned to the workspace
{{workspace.url}}	The URL of the workspace
{{job.id}}	The unique identifier assigned to the job
{{job.name}}	The name of the job at the time of the job run
{{job.run_id}}	The unique identifier assigned to the job run
{{job.repair_count}}	The number of repair attempts on the current job run
{{job.start_time.[argument]}}	A value based on the time (in UTC timezone) that the job run started. The return value is based on the argument option. See Options for date and time values.
{{job.parameters.[name]}}	The value of the job-level parameter with the key [name]
{{job.trigger.type}}	The trigger type of the job run. The possible values are Manual and Scheduled.
{{job.trigger.file_arrival.location}}	If a file arrival trigger is configured for this job, the value of the storage location
{{job.trigger.time.[argument]}}	A value based on the time (in UTC timezone) that the job run was triggered, rounded down to the closest minute for jobs with a cron schedule. The return value is based on the argument option. See Options for date and time values.
{{task.name}}	The name of the current task
{{task.run_id}}	The unique identifier of the current task run
{{task.execution_count}}	The number of times the current task was run (including retries and repairs)
{{task.notebook_path}}	The notebook path of the current notebook task
{{tasks.[task_name].run_id}}	The unique identifier assigned to the task run for [task_name]
{{tasks.[task_name].result_state}}	The result state of task [task_name]. The possible values are success, failed, excluded, canceled, skipped, timed out, upstream_canceled, and upstream_failed.
{{tasks.[task_name].error_code}}	The error code for task [task_name] if an error occurred running the task. Examples of possible values are RunExecutionError, ResourceNotFound, and UnauthorizedError. For successful tasks, this evaluates to an empty string.
{{tasks.[task_name].execution_count}}	The number of times the task [task_name] was run (including retries and repairs)
{{tasks.[task_name].notebook_path}}	The path to the notebook for the notebook task [task_name]
{{tasks.[task_name].values.[value_name]}}	The task value with the key [value_name] that was set by task [task_name]

Table 12-2 Options for Date and Time

Argument	Description
iso_weekday	Returns a digit from 1 to 7, representing the day of the week in the time stamp
is_weekday	Returns true if the timestamp is on a weekday
iso_date	Returns the date in ISO format
iso_datetime	Returns the date and time in ISO format
year	Returns the year part of the timestamp
month	Returns the month part of the timestamp
day	Returns the day part of the timestamp
hour	Returns the hour part of the timestamp
minute	Returns the minute part of the timestamp
second	Returns the second part of the timestamp
timestamp_ms	Returns the timestamp in milliseconds

Pass Parameters Between Tasks and Notebook

You can pass parameters from a task to a notebook and vice versa. This enables dynamic workflow behavior, allowing notebooks to adjust their processing based on runtime values.

The oidlUtils.parameters package provides the necessary functionality to handle these parameter operations. The oidlUtils package is a utility library in AI Data Platform that simplifies tasks such as parameter management, task value passing, and other workflow operations. It is commonly used in notebooks and tasks to get and set parameters across workflow stages.

Example Workflow: Passing Parameters

In this scenario, we have two notebooks in a workflow. Notebook 1 receives parameters from a task, processes them, and sets output parameters that are passed to Notebook 2 in the next task.

Notebook 1: Get and Set Parameters

# Get parameter if already set in the task
param_key = "param1"
param_value = oidlUtils.parameters.getParameter(param_key, "defaultValue")
print(param_value)
print("Param {} value is {}".format(param_key, param_value))
 
# Set parameter value in the task
oidlUtils.parameters.setTaskValue("output_param_2", "1234")
param_key = "param2"
param_value = oidlUtils.parameters.getParameter(param_key, "defaultValue2")
print("Param {} value is {}".format(param_key, param_value))

The first notebook retrieves a parameter (param1) passed from the task and then sets a new parameter (output_param_2), which will be used in the next task.

Notebook 2: Read the Output Parameter

output_param_2= "output_param_2"
param_value = oidlUtils.parameters.getParameter(output_param_2, "defaultValue")
print("Param {} value is {}".format(output_param_2, param_value)

The second notebook receives the output_param_2 from Notebook 1 via the workflow task and processes it.

1. Task 1: Notebook 1
   • In the first task, parameters can be passed to Notebook 1 from the job or task itself.
   • Notebook 1 processes the parameters as input parameters param1 and sets new output parameters (e.g., output_param_2).
2. Task 2: Notebook 2
   • In the second task, Notebook 2 receives the output parameter from Task 1 by referencing it with {{tasks.PassParameter.values.output_param_2}}.
   • The value of output_param_2 is passed into Notebook 2 where it can be used for further processing.

This approach makes it easy to pass values dynamically between tasks and notebooks, allowing your workflows to be more flexible and adaptive.

Add Parameters to a Job

You can provide your jobs different parameters to track.

1. On the Home page, click Workflow.
2. Click the job you want add parameters to and click the Details tab.
3. Under Job Parameters, provide the key and value to track. To add multiple parameters, click Add Parameter.
   Changes you made are saved automatically.

Delete Parameters from a Job

You update your jobs to remove parameters that are no longer needed.

1. On the Home page, click Workflow.
2. Click the job you want remove parameters from and click the Details tab.
3. Under Job Parameters, click Delete next to the parameter you want to remove.
   Changes you made are saved automatically.

Add Parameters to a Task

You can add parameters to tasks to alter their behavior.

1. On the Home page, click Workflow.
2. Click the job that contains the tasks you want add parameters to and click the Tasks tab.
3. Click the task you want to add parameters to.
4. Under Parameters, enter the key and value for your parameter. To add multiple parameters, click Add Parameter.
   Changes you made are saved automatically.

Delete Parameters from a Task

You can remove parameters you don't need from your tasks.

1. On the Home page, click Workflow.
2. Click the job that contains the tasks you want delete parameters from and click the Tasks tab.
3. Click the task you want to delete parameters from.
4. Click Delete next to each parameter you want removed.
   Changes you made are saved automatically.

Run a Job with Different Parameter Values

You can choose to run a job immediately with modified parameter values.

1. On the Home page, click Workflow.
2. Click on the job you want to run.
3. Click the down arrow next to Run Now. Click Run now with different parameters.
   
   

   

   
   
4. Enter new values. These parameter values only apply to this job run.
   
   

   

   
   
5. Click Run.