Config Validation Run

A ML Monitoring Application Run can be specifically defined as a Config Validation Run, when one wants to validate the application configuration provided as input to the application.

This ML Monitoring Job Run will be a dry run for validating the application configuration - Application configuration before running other ActionTypes RUN_BASELINE or RUN_PREDICTION

As part of this validation below things will be validated.

• Config Location Validation - Read Validation:

  • Config location validation verifies the existence and accessibility of configuration files or settings at specified locations.

  • This validation ensures that the application can read configuration data from the designated locations without errors or exceptions.

  • It may involve checking file paths, permissions, file formats, and contents to ensure that the configuration is valid and usable by the application.

  • In case config location not passed or not a valid HTTP url, ODSC Job Run will be marked Failed and error logs will visible in Job Run logs.

  • InvalidConfigException will be raised

• Application JSON Validation:

  • It validates that there should be one monitor_id present in the application configuration passed as input

  • It checks the semantics of the JSON

  • It validates the storage_details parameter required to run the application

  • In case of JSON validation failure, ODSC Job Run will be marked Failed and error logs will visible in Job Run logs.

  • ConfigValidationException will be raised, and a error dictionary would be logged

• Grammar Validation:

  • This validation ensures that statements, expressions, and constructs follow the correct syntax and semantics defined by the grammar.

  • It validates that the JSON format conforms to a predefined schema or structure.

  • In case of Grammar validation failure, ODSC Job Run will be marked Failed and error logs will visible in Job Run logs.

  • ConfigValidationException will be raised, and a error dictionary would be logged

• Component Validation:

  • This validates whether the names provided in the application configuration adheres to the names or identifiers assigned to components within a system or application.

  • This involves verifying the correctness and logic of conditional statements or expressions and metrics supported by application.

  • This validation ensures that conditions and logical operators used in expressions evaluate as expected and produce the desired outcomes.

  • It checks for errors such as syntax errors, incorrect operator precedence, and unintended logical behavior.

  • In case of Component validation failure, ODSC Job Run will be marked Failed and error logs will visible in Job Run logs.

  • ConfigValidationException will be raised, and a error dictionary would be logged

• IAM Validation:

  • IAM (Identity and Access Management) validation involves verifying the correctness and effectiveness of access control policies and permissions within a system.

  • This validation ensures that users, roles, and groups are assigned appropriate permissions and access levels based on their roles and responsibilities.

  • It helps prevent unauthorized access, data breaches, and security vulnerabilities.

  • As part of this step we will run the below validations:

    • Storage Details Validation - Read, List, Write Validation:

      "storage_details": {
          "storage_type": "OciObjectStorage",
          "params": {
            "namespace": "<NAMESPACE>",
            "bucket_name": "<BUCKET_NAME>",
            "object_prefix": "<OBJECT_PREFIX>"
          }
      }
      

      • For the storage type “OciObjectStorage” params namespace, bucket_name and object_prefix provided by the user in the storage_details block of application configuration, we will do the Data read, list and write operations in the object storage to ensures the correctness and effectiveness of access control policies and permissions within a system.

      • In case some policies went missing we will throw error specific to that failed operation like Read, List or write Read/List/Write permission for the file path is unauthorized

    • Baseline and Prediction Reader Validation - Read Validation:

      "baseline_reader": {
         "type": "CSVDaskDataReader",
         "params": {
            "file_path": "<OCI_STORAGE_FILE_PATH_FOR_BASELINE_INPUT_DATA>"
         }
      }
      
      "prediction_reader": {
         "type": "CSVDaskDataReader",
         "params": {
            "file_path": "<OCI_STORAGE_FILE_PATH_FOR_BASELINE_INPUT_DATA>"
         }
      }
      

      • For the params file_path provided in both case baseline_reader and prediction_reader provided by the user in the application configuration, we will do the Data read operations in the object storage to ensures the correctness and effectiveness of access control policies and permissions within a system.

      • In case some policies went missing we will throw error specific to that failed operation like Read Data reader [type] read permissions

    • Post Processor Validation - Write Validation:

      "post_processors": [
         {
            "type": "SaveMetricOutputAsJsonPostProcessor",
            "params": {
               "namespace": "<NAMESPACE>",
               "bucket_name": "<BUCKET_NAME>"
            }
         }
      ]
      

      • For the params namespace and bucket_name provided in case of SaveMetricOutputAsJsonPostProcessor by the user in the application configuration, we will do the Data write operations in the object storage to ensures the correctness and effectiveness of access control policies and permissions within a system.

      • In case some policies went missing we will throw error specific to that failed operation like Write SaveMetricOutputAsJsonPostProcessor write permissions

How to use it

Configuration required for running Config Validation Run

CONFIG_LOCATION: This is the HTTP location of oci storage application config, which is a mandatory parameter to kick start validation run. The application config must have a read access to the specified HTTP location for the OCI resources needed for the validation run.

RUNTIME_PARAMETER: Runtime parameter are the parameters that defines the state of a single monitor run. In order to specify, if a monitor run is validation run, user needs to pass ACTION_TYPE as RUN_CONFIG_VALIDATION as one of the arguments of runtime parameters as shown below.

"CONFIG_LOCATION": "<HTTP_LOCATION_OF_OCI_STORAGE_APPLICATION_CONFIG_FILE_PATH>"
"RUNTIME_PARAMETER": "{"ACTION_TYPE":"RUN_CONFIG_VALIDATION"}"

Output of Config Validation Run

When the action type is a config validation run the default application will return a list of errors specifying the component names and the corresponding issues, which can be seen on the Customer Job logs.

If the configuration is not correct then the corresponding Job run with ActionType RUN_CONFIG_VALIDATION will fail.