Oracle Cloud Infrastructure Documentation

Triggers

Triggers let ML Applications providers specify the triggering mechanism for their ML jobs or pipelines, easing the implementation of fully automated MLOps.

Triggers are the entry points for the runs of the ML workflows and are defined by YAML files within the ML Applications package as instance components. Triggers are automatically created when a new ML Application Instance is created but only when all other instance components are created. So, when a trigger is created, it can refer to other instance components created before. The jobs or ML Pipelines that are part of your application can be triggered by defining triggers as instance components in your ML Application. Triggers let providers and consumers start runs of some workflows.

Within the trigger definition, providers can specify:
Trigger target
Defines what is run. For example, a new pipeline or job run is created when the trigger is activated or invoked.
Trigger condition
Defines when the trigger is run. You can define which HTTP endpoints (WebHooks) activate or invoke the trigger or events such as instance was created.
Trigger parameters
Define which parameters can be passed to the trigger upon its activation (invocation). You can pass the parameter values further to the trigger target. For example, you can pass a reference to a container image that's started in your pipeline or job.
Triggers can be activated or invoked by:
HTTP-based triggering
Triggers can be fired in response to HTTP requests. Two endpoints that enable users to make HTTP requests for firing triggers.
  • Provider endpoint: Available on the ML Application Instance View resource, it's meant to be used by providers.
  • Consumer endpoint: Available on the ML Application Instance resource, it's meant to be used by consumers.
  • ML Applications providers have the option to enable these endpoints in any combination as follows:
    • Only provider endpoint.
    • Only consumer endpoint.
    • Both provider and consumer endpoints.
    • None
Event-based triggering
Triggers are fired in response to a lifecycle event on a particular resource. The only supported resource is the ML Application Instance. The supported lifecycle events are:
  • Creating an ML Application Instance: let providers implement ML Applications with a one-time training run.
  • Upgrading of ML Application Instance: let providers add the ability to retrain the model when a new version of the ML Application Implementation (for example, with a new training algorithm) is deployed.

The trigger target is a create operation payload for OCI resources which are created when the trigger condition is met. The supported target types are DataScienceJobRun and DataSciencePipelineRun.

Within the trigger targets, you can refer to implicit variables, for example, 
${app_instance.instance_components.oci_objectstorage_bucket.data_storage_bucket.name}
or trigger parameters, for example, 
${parameters.docker_image_tag}
Implicit variable references are replaced with actual values when the instance is created, updated, or upgraded. Parameter references are replaced with values when the trigger is activated. For more information see Implicit Variables for ML Applications Packages or Parameterized Triggers.

Defining Triggers

Triggers are defined as YAML files under the instance_components directory in the application package. Triggers files use the extension .trigger.yaml and must follow the schema as follows:

  • apiVersion
    • description: the version of the schema for this configuration file.
    • required: true
    • type: string
  • kind
    • description: the kind of resource (only ml-application-trigger supported).
    • required: true
    • type: string
  • metadata
    • description: the metadata for a particular resource.
    • required: true
    • type: object (map)
    • properties (supported metadata):
      • name
        • description: the name as an identifier for a particular instance of the resource.
        • required: true
        • type: string
  • spec
    • description: the specification of a particular resource.
    • required: true
    • type: object
    • properties:
      • parameters
        • description: A map of parameters that can be passed to the trigger upon its activation (invocation).
        • required: false
        • type: the map (parameter name maps to parameter properties)
          • the parameter name must match "\w+" regexp (at least one alphanumeric character).
        • parameter properties:
          • mandatory
            • type: Boolean (true or false)
            • required: false (default is false)
            • Description: Whether the particular argument is mandatory or not.
              Note

              Mandatory trigger parameters aren't allowed for triggers that have consumerEndpoint or any kind of event-based condition.
          • description
            • type: string
            • required: false
            • description: The parameter description.
          • validationRegexp
            • type: string
            • required: false
            • description: The regular expression used for validation of argument value.
          • defaultValue
            • type: string
            • required: false
            • description: The value used when the parameter isn't specified in the activation (invocation) request. The default value must be specified for optional parameters. It might be specified for mandatory parameters. When the validationRegexp is specified, the default value must match.
      • condition
        • description: The condition that defines when the trigger fires.
        • required: true
        • type: object
        • properties:
          • requests
            • description: The list of sources for direct trigger requests. For each such request, the trigger tries to fire.
            • required: one of
            • type: array of objects
            • item type properties:
              • source
                • description: The source of trigger requests: if it's present in the array it means the trigger fires on request from this resource.
                • required: true
                • type: enum
                • enum values:
                  • providerEndpoint: if the source with this type is presented in the requests section, triggering using HTTP request for providers is enabled (/mlApplicationInstanceView/<mlApplicationInstanceViewId>/action/trigger).
                  • consumerEndpoint: if the source with this type is presented in the requests section, triggering using HTTP request for consumers is enabled (/mlApplicationInstance/<mlApplicationInstanceId>/action/trigger).
          • events
            • description: Events for which the trigger fires.
            • required: one of
            • type: array (items are polymorphic objects)

            • common properties for items (parent:polymorphism):

              • source
                • description: The event source: if it's present in the array it means that the trigger fires based on incoming events coming from this event source (the first part of the discriminator for various events).
                • required: true
                • type: enum
                • enum values:
                  • mlApplicationInstance: The ML Applications instance is an event source. Supported types: onCreate, onVersionUpgrade
              • type
                • description: The event type (common property for all event sources: the second part of the discriminator for events).
                • required: true
                • type: enum
                • enum values: based on event source
                  • source: mlApplicationInstance

            • supported events (specific children:polymorphism):

              • source: mlApplicationInstance, type: onCreate
                • description: the trigger fires on the creation of the ML Applications instance.
                • type: object (child with compound delimiter: source, type).
              • source: mlApplicationInstance, type: onVersionUpgrade
                • description: the trigger fires on the version upgrade of the ML Application Instance.
                • type: object (child with compound delimiter: source, type)
                • properties:
                  • packageVersion
                    • description: The trigger fires only when the ML Applications instance is upgraded to ML Applications implementation version with this package version.
                    • required: true
                    • type: string
                  • mlApplicationInstanceViewTagName
                    • description: The trigger fires only when the upgraded ML Application Instance has a tag with this name (and defined value).
                    • required: false
                    • type: string
                  • mlApplicationInstanceViewTagValue
                    • description: The trigger fires only when upgraded ML Application Instance has a tag with this value (and defined name).
                    • required: false
                    • type: string
      • target
        • description: The trigger target (for example, JobRun).
        • type: object
        • properties:
          • type
            • description: the type of trigger target.
            • type: enum (values: DataScienceJobRun, DataSciencePipelineRun)
            • required: true
          • template
            • description: Create a payload for a resource used as a target. It can contain various placeholders that are dynamically resolved and replaced with actual values. Two types of placeholders exist: implicit variables and trigger parameters.
            • type: object (expected JSON payload, note: JSON is valid YAML)
            • required: true

Example Trigger YAML Definition

apiVersion: v1-beta
kind: ml-application-trigger
metadata:
  name: Training trigger
spec:
  parameters:
    docker_image_tag:
      description: "Tag of the docker image to be used by the DataScience Job"
      validationRegexp: ".+"
      defaultValue: "v1.0"
    scoring_threshold:
      validationRegexp: "[0-9]+"
      defaultValue: "10"
    an_optional_parameter_with_default:
      defaultValue: "v1.0"
    a_required_parameter:
      mandatory: true
      description: "This is a sample required parameter"
      validationRegexp: "abc.+"
    another_optional_parameter_with_default:
       mandatory: false
       defaultValue: "bar"
  condition:
    requests:
     :source: providerEndpoint # CP action for provider (provider set privilages)
     :source: consumerEndpoint # CP action for consumer (consumer set privilages)
    events:
     :source: mlApplicationInstance
        type: onCreate
     :source: mlApplicationInstance
        type: onVersionUpgrade
        packageVersion: 2.0
        mlApplicationInstanceViewTag: big-fish-customer    
   
  target:
    type: DataSciencePipelineRun
    template: {
      "projectId": "${app_impl.package_arguments.data_science_project_id}",
      "compartmentId": "${app.compartment_id}",
      "pipelineId": "${app_impl.application_components.oci_datascience_pipeline.ad_pipeline.id}",
      "stepOverrideDetails": [
        {
          "stepName": "ingestion",
          "stepConfigurationDetails": {
            "environmentVariables": {
              "MlApplicationInstance": "${app_instance.id}",
              "ML_APP_INST_OCID": "${app_instance.id}",
              "ENVIRONMENT_TYPE": "dev",
              "BIP_URL": "${app_instance.configuration.bip_url}",
              "BIP_USERNAME_SECRET_ID": "${app_instance.configuration.bip_username_secret_id}",
              "BIP_PASSWORD_SECRET_ID": "${app_instance.configuration.bip_password_secret_id}",
              "INGREDIENT_BUCKET": "${app_instance.instance_components.oci_objectstorage_bucket.data_storage_bucket.name}",
              "INGREDIENT_OBJECT": "${app_impl.package_arguments.ingredient_object_path_name}",
              "OS_NAMESPACE": "${app_impl.package_arguments.bucket_namespace}",
              "TARGET_INGESTION_PATH": "${app_impl.package_arguments.target_ingestion_dir}",
              "METRICS_NAMESPACE": "${app_impl.package_arguments.monitoring_namespace}",
              "METRICS_COMPARTMENT_ID": "${app.compartment_id}",
              "ML_APP_NAME": "${ml_app_name}",
              "ML_APP_IMPL_NAME":"${ml_app_impl_name}",
              "ML_APP_PACKAGE_VERSION":"1.11",
              "INGREDIENT_PATH":"${app_impl.package_arguments.ingredient_object_path_name}",
              "DIMENSION_CUSTOM": "CustomDimension1",
              "TENANT": "idsc-stripe"
            }
          }
        },
        {
          "stepName": "transformation",
          "stepConfigurationDetails": {
            "environmentVariables": {
              "MlApplicationInstance": "${app_instance.id}",
              "CUSTOMER_BUCKET": "${app_instance.instance_components.oci_objectstorage_bucket.data_storage_bucket.name}",
              "OS_NAMESPACE": "${app_impl.package_arguments.bucket_namespace}",
              "INPUT_FOLDER": "${app_impl.package_arguments.transformation_input_dir}",
              "OUTPUT_FOLDER": "${app_impl.package_arguments.transformation_output_dir}",
              "NUMBER_OF_WEEKS_TO_KEEP": "${app_impl.package_arguments.number_of_weeks_to_keep}"
            }
          }
        },
        {
          "stepName": "training",
          "stepConfigurationDetails": {
            "environmentVariables": {
              "MlApplicationInstance": "${app_instance.id}",
              "docker-image-tag": "${parameters.docker_image_tag}",
              "scoring-threshold": "${parameters.scoring_threshold}",
              "CUSTOMER_BUCKET": "${app_instance.instance_components.oci_objectstorage_bucket.data_storage_bucket.name}",
              "CONTAINER_ENTRYPOINT": "\"train\", \"oci://${app_instance.instance_components.oci_objectstorage_bucket.data_storage_bucket.name}/data/\", \"-a\", \"{\"refresh_date\":\"2030-01-01\", \"disable_requester_id\":\"1\"}\", \"-p\", \"http://pg\", \"-n\", \"ac_df_test_namespace\"",
              "MODEL_DEPLOYMENT_ID": "${app_instance.instance_components.oci_datascience_model_deployment.tf_model_deployment.id}"
            }
          }
        }
      ]
    }
Note

  • The current apiVersion (for trigger definition schema) is v1-beta.
  • kind must always be ml-application-trigger.
  • Either requests or events must be specified. A trigger definition that specifies neither requests nor events is invalid.
  • requests, whenever specified, must not be empty.
  • events, whenever specified, must not be empty.
  • Regarding event type ML Application Instance Version Upgrade (Source: mlApplicationInstance, type: onCreate):
    • Whenever mlApplicationInstanceViewTagName is specified, mlApplicationInstanceViewTagValue must also be specified.
    • Whenever mlApplicationInstanceViewTagValue is specified, mlApplicationInstanceViewTagName must also be specified.

    • Whenever specified, both mlApplicationInstanceViewTagName and mlApplicationInstanceViewTagValue must be specified.

  • The template for DataScienceJobRun target type must follow CreateJobRunDetails Schema.

  • The template for DataSciencePipelineRun target type must follow CreatePipelineRunDetails Schema.

  • The values used in the template payload can contain various placeholders as explained in Implicit Variables for ML Applications Packages.
  • The placeholders must use the format ${variable_name}, that is, the placeholder must begin with $ and must be followed by the variable name inside braces {}.
  • jobId in the DataScienceJobRun template must refer to, or resolve to, a DataScienceJob application component belonging to the ML Application Implementation.
  • pipelineId in the DataSciencePipelineRun template must refer to, or resolve to, a DataSciencePipeline application component belonging to the ML Applications Implementation.
  • design is defined in Defining Triggers.

Information on Trigger-Related Endpoints

Trigger-Related Endpoints
Endpoint Related Resource Required Permission Used By
/mlApplicationInstances/<mlApplicationInstanceId>/actions/trigger ML Application Instance DATA_SCIENCE_APPLICATION_INSTANCE_TRIGGER Consumers
/mlApplicationInstanceViews/<mlApplicationInstanceViewId>/actions/trigger ML Application Instance View DATA_SCIENCE_APPLICATION_INSTANCE_VIEW_TRIGGER Providers
/mlApplicationInstanceViews/<mlApplicationInstanceViewId>/actions/disableTrigger ML Application Instance View DATA_SCIENCE_APPLICATION_INSTANCE_VIEW_READDATA_SCIENCE_APPLICATION_INSTANCE_VIEW_UPDATE Providers
/mlApplicationInstanceViews/<mlApplicationInstanceViewId>/actions/enableTrigger ML Application Instance View DATA_SCIENCE_APPLICATION_INSTANCE_VIEW_READDATA_SCIENCE_APPLICATION_INSTANCE_VIEW_UPDATE Providers

Parameterized Triggers

Triggers can parameterize targets (Job or Pipeline runs) by referring to implicit variables. However, implicit variables are updated only when the instance is created, updated, or upgraded. When you need to pass a specific parameter with a value that's known only at the time of the trigger activation (invocation), you can use trigger parameters.

Trigger parameters can be optionally defined in the trigger YAML file. When the parameters are defined, you can include their names and values in the payload of the trigger activation (invocation) requests. All your references to the parameters in the target definition are replaced with actual values provided in the payload of the request.

To activate (invoke) a parameterized trigger, you need to send an HTTP POST to the trigger endpoint. For example:

Parameterized Trigger Activation:
oci raw-request --auth security_token --http-method POST \
    --target-uri https://<region>/20190101/mlApplicationInstanceViews/<ocidid>/actions/trigger \
  --request-body file://./triggerInvocationPayload.json
Parameterized Trigger Request:
{
    "triggerName": "training job",
    "parameters": [
        {
            "name": "scoring_threshold",
            "value": "99"
        },
        {
            "name": "a_required_parameter",
            "value": "must_value"
        }
    ]
}

Triggers and Resource Principals

Triggers that are defined by providers in ML Applications packages as YAML files can be exposed to both providers and consumers. Providers and consumers can activate (fire or trigger) the triggers and start a run of a pipeline or job.

The resource principal that's used to start the run differs for consumer and provider invocations.

When consumers activate triggers, the ML Application Instance resource principal is used (datasciencemlappinstanceint). On the other hand, when providers activate triggers, the ML Application Instance view resource principal is used (datasciencemlappinstanceviewint).

This implies that you need to define policies that let the instance or instance view resource principal create runs. Because runs depend on networking and logging, you must let the resource principals use networking and logging too. For details, see the Policy Setup section.