Low Code Setup with Config Reader

Along with the library provided APIs, ML Insight can be set up and customized by authoring and passing a JSON configuration.

This document shows how to use the Insights configuration reader to load the builder and compute the profile.

Begin by loading the required libraries and modules:

from mlm_insights.config_reader.insights_config_reader import InsightsConfigReader

Initialize the Insights builder by specifying the location of the monitor config JSON file (under config_location).

insights_builder = InsightsConfigReader(config_location="monitor_config.json").get_builder()

Run this builder object to get the run result, which yields the profile.

run_result = insights_builder.build().run()
profile = run_result.profile
print(profile.to_pandas())

How to setup configuration for low code ML Insights

Sample Config File:

ml-insights-config.json
{
  "input_schema": {
    "Age": {
      "data_type": "integer",
      "variable_type": "continuous",
      "column_type": "input"
    },
    "EnvironmentSatisfaction": {
      "data_type": "integer",
      "variable_type": "continuous",
      "column_type": "input"
    },
    "CreatedDate": {
      "data_type": "datetime",
      "variable_type": "datetime",
      "column_type": "input"
    },
    "ModeOfTransport": {
      "data_type": "string",
      "variable_type": "nominal",
      "column_type": "input"
    }
  },
  "engine_detail": {
    "engine_name": "native"
  },
  "reader": {
    "type": "JsonlNativeDataReader",
    "params": {
      "url_path": "oci://<bucket>/*.jsonl",
      "orient": "records",
      "lines": "True"
    }
  },
  "reference_profile_reader": {
    "type": "OCIObjectStorageProfileReader",
    "params": {
      "path": "oci://bucket@namespace/object_prefix/reference_profile.bin"
    }
  },
  "dataset_metrics": [
    {
      "type": "RowCount"
    },
    {
      "type": "CorrelationRatio"
    }
  ],
  "feature_metrics": {
    "Age": [
      {
        "type": "Min"
      },
      {
        "type": "Max"
      },
      {
        "type": "Count"
      }
    ],
    "EnvironmentSatisfaction": [
      {
        "type": "Mode"
      },
      {
        "type": "Count"
      }
    ],
    "CreatedDate": [
      {
        "type": "DateTimeMin"
      }
    ]
  },
  "transformers": [
    {
      "type": "ConditionalFeatureTransformer",
      "params": {
        "conditional_features": [
          {
            "feature_name": "Young",
            "data_type": "integer",
            "variable_type": "ordinal",
            "expression": "df.Age < 30"
          }
        ]
      }
    }
  ],
  "post_processors": [
    {
      "type": "ObjectStorageWriterPostProcessor",
      "params" : {
        "bucket_name": "test-bucket",
        "object_name": "config.json",
        "prefix": "mlinsights"
      }
    },
    {
      "type": "LocalWriterPostProcessor",
      "params" : {
        "file_name": "config.json" ,
        "file_location": "data"
      }
    },
    {
      "type": "OCIMonitoringPostProcessor",
      "params" : {
        "compartment_id": "compartment_id",
        "namespace": "namespace",
        "dimensions": {
          "monitor_id": "monitor_id"
        }
      }
    }
  ],
  "tags": {
    "tag": "value"
  },
  "test_config": {
    "feature_metric_tests": [
      {
        "feature_name": "Age",
        "tests": [
          {
            "test_name": "TestGreaterThan",
            "metric_key": "Min",
            "threshold_value": 5,
            "tags": {
              "key_1": "value_1"
            }
          },
          {
            "test_name": "TestIsComplete"
          }
        ]
      }
    ],
    "dataset_metric_tests": [
      {
        "test_name": "TestGreaterThan",
        "metric_key": "RowCount",
        "threshold_value": 40,
        "tags": {
          "subtype": "falls-xgb"
        }
      }
    ]
  }
}

Insight Components

Input Schema

This is a required component and must be defined in the config.

Input schema is the map of features and their data types, variable types, and column type.

Description

Key

Value

Example

feature_name

object of key value pair of data_type ,variable type and column_type

“Age”: { “data_type”: “integer”, “variable_type”: “continuous”, “column_type”: “input” }

  • Data Type (Required)

Data types can be provided for each feature of the input dataset which represent the type of the feature value.

Supported data_type - “integer”, “float”, “string”, “boolean”, “text”, “object”, “datetime”, “timestamp”

  • Variable Type (Required)

Variable types can be provided for each feature of the input dataset which represent the type of a statistical random variable.

Supported variable_type - “continuous”, “discrete”, “nominal”, “ordinal”, “binary”, “text”, “object”, “datetime”

  • Column Type (Optional - Default value “input”)

Insights supports performance metrics for regression and classification models. In addition to these, Insights also supports multivariate metrics like Feature Importance. These metrics require the prediction columns or target columns (ground truth) to be in the input dataset. To make it easier to configure the metrics, Insights allows users to configure the prediction or target columns using the feature schema.

Supported column_type - “input”, “prediction”, “target”, “prediction_score”

Example
{
    "input_schema": {
        "sepal length (cm)": {
          "data_type": "float",
          "variable_type": "continuous",
          "column_type": "input"
        },
        "sepal width (cm)": {
          "data_type": "float",
          "variable_type": "continuous"
          "column_type": "input"
        }
    }
}

Reader

This is a required component and must be defined in the config.

Reader allows for the ingestion of raw data into the framework.

Description

Field Name

Description

Example

type

type of data reader to be used

“type”: “JsonlDaskDataReader”

param

reader params (required)

data_source (optional)

“params”: { “url_path”: “oci://<bucket>/*.jsonl”,

“storage_options”: { “config”: “~/.oci/config” }, “orient”: “records”, “lines”: “True” }

“data_source”: {

“type”: “LocalDatePrefixDataSource”, “params”: {

“base_location”:”mlm_demo”, “file_type”: “csv”, “date_range”: {“start”: “2023-04-04”, “end”: “2023-04-05”}

}

}

Example using data source for determining the data location
{
    "reader": {
    "type": "CSVNativeDataReader",
        "params": {
          "data_source": {
            "type": "LocalDatePrefixDataSource",
            "params": {
              "base_location":"mlm_demo",
              "file_type": "csv",
              "date_range": {"start": "2023-04-04", "end": "2023-04-05"}
            }
          }
        }
    }
}
Example without using data_source
{
    "reader": {
        "type": "JsonlNativeDataReader",
        "params": {
          "url_path": "oci://<bucket>/*.jsonl",
          "storage_options": {
            "config": "~/.oci/config"
          },
          "orient": "records",
          "lines": "True"
        }
    }
}
Supported Readers
CSVDaskDataReader
JsonlDaskDataReader
CSVNativeDataReader
JsonlNativeDataReader
NestedJsonNativeDataReader
NestedJsonDaskDataReader
ADWDaskDataReader
ADWNativeDataReader

We can use reader params to define the location of the files to be read or can specify a data source in the reader.

Data Source

The Data Source component is responsible for interacting with a specific data source and returning a list of locations to be read.

Supported Data Sources
FileUrlDataSource
LocalDatePrefixDataSource
LocalFileDataSource
OCIDatePrefixDataSource
OCIObjectStorageDataSource

Feature Metrics

In this section, you need to add metrics that you need for each feature.

Description

Key

Value

feature_name

metric list

Supported Feature Metrics
# Data quality metrics
Count
DistinctCount
DuplicateCount
FrequencyDistribution
Max
Mean
Min
Mode
Percentiles
ProbabilityDistribution
Range
Skewness
StandardDeviation
Sum
IQR
Kurtosis
TopKFrequentElements
TypeMetric
Variance

# Data Integrity
IsConstantFeature
IsQuasiConstantFeature
Quartiles
IsNegative
IsNonZero
IsPositive

# Drift Metrics
KullbackLeibler
KolmogorovSmirnov
ChiSquare
JensenShannon
PopulationStabilityIndex

# Bias and Fairness Metrics
ClassImbalance

# Datetime Metrics
DateTimeMin
DateTimeMax
DateTimeDuration

Example
"feature_metric": {
    "sepal length (cm)" : [
    {"type": "Sum"},{"type": "Quartiles"}
    ],
    "sepal width (cm)": [
        {"type": "Min"},{"type": "DistinctCount"}
    ],
    "petal length (cm)": [
        {"type": "Count"},{"type": "Mean"}
    ],
    "petal width (cm)": [
        {"type": "IsQuasiConstantFeature"},{"type": "Kurtosis"}
    ]
}

Dataset Metrics

Description

List of metrics to be calculated on the data set.

Example
"data_set_metric": [
    {
    "type": "RowCount"
    }
]
Supported Data Set Metrics
# Data Quality Metrics
CramersVCorrelation
PearsonCorrelation
CorrelationRatio

# Regression Metrics
RowCount
MeanAbsoluteError
MeanSquaredError
R2Score
RootMeanSquaredError
MeanSquaredLogError
MeanAbsolutePercentageError
MaxError

# Classification metrics
AccuracyScore
PrecisionScore
RecallScore
FBetaScore
FalsePositiveRate
FalseNegativeRate
Specificity
ConfusionMatrix
LogLoss
ROCCurve
ROCAreaUnderCurve
PrecisionRecallCurve
PrecisionRecallAreaUnderCurve

# Conflict Metrics
ConflictPrediction
ConflictLabel

Post Processor:

Post processor components are responsible for running any action after the entire data set is processed and all the metrics are calculated.

Description

Field Name

Description

Example

type

type of post processor

“type”: “ObjectStorageWriterPostProcessor”

param

For LocalWriterPostProcessor - file_name, file_location

For ObjectStorageWriterPostProcessor -bucket_name, object_name, prefix

“params” : { “bucket_name”: “test-bucket”, “object_name”: “config.json”, “prefix”: “mlinsights” }

Example
  "post_processors": [
  {
    "type": "ObjectStorageWriterPostProcessor",
    "params" : {
      "bucket_name": "test-bucket",
      "object_name": "config.json",
      "prefix": "mlinsights"
    }
  },
  {
    "type": "LocalWriterPostProcessor",
    "params" : {
      "file_name": "config.json" ,
      "file_location": "data"
    }
  }
]
Supported Post Processors
LocalWriterPostProcessor
ObjectStorageWriterPostProcessor
OCIMonitoringPostProcessor

Transformer

The transformer component provides an easy way to do simple in-memory transformations on the input data.

The list of transformers to be used to add a conditional feature or transform the data before insights run.

Description

Field Name

Description

Example

type

type of transformer

“type”: “ConditionalFeatureTransformer”

param

conditional_features - List of conditional features

“params”: {
“conditional_features”: [
{

“feature_name”: “Young”, “data_type”: “integer”, “variable_type”: “ordinal”, “expression”: “df.Age < 30”

}

]

}

Conditional Features

Field Name

Value

Remarks

expression

Python expression, to be written using pandas series based functions. Only pandas series level functions are supported in a python expression and the symbol ‘df’.

the expression must return a valid output. For example: “expression”: “df.Age < 30”

feature_name

<any name that suits your feature>

data_type

The data type of the feature.

variable_type

The variable type of the feature.

Example
"transformers": [
{
  "type": "ConditionalFeatureTransformer",
  "params": {
    "conditional_features": [
      {
        "feature_name": "Young",
        "data_type": "integer",
        "variable_type": "continuous",
        "expression": "int(json_row['Age'] < 30)"
      }
    ]
  }
}
]

Engine

The underlying distributed framework used to run the computations. By default the execution engine is native and is based on native python and pandas.

Description

Field Name

Description

Example

engine_name

type of engine

“engine_name”: “native”

Example
"engine_detail": {
        "engine_name": "native"
}
Supported Engines
native
dask

Reference Profile Reader

Reference Profile Reader read Insights Reference/Baseline Profile from different data sources such as local file system and OCI Object Storage.

Description

Field Name

Description

Example

type

type of profile reader

“type”: “OCIObjectStorageProfileReader”

Example
  "reference_profile_reader": {
    "type": "OCIObjectStorageProfileReader",
    "params": {
      "path": "oci://bucket@namespace/.../reference_profile.bin"
    }
}
Supported Reference Profile Readers
OCIObjectStorageProfileReader
LocalProfileReader

Tags

Note, this is a library internal concept and should not be confused with OCI resource tags.

  • User provided key value pair

  • Applications consuming the library can provide tags to be associated with a profile. For eg: when running the baseline/evaluation run, we can store:

    <monitor_id: value>

Example
"tags": {
    "tenancy": "tenancy-xyz",
    "monitor_id": "ocid-********"
}

Tests/Test Suites

For detailed documentation, please refer to section: Test/Test Suites Component