mlm_insights.core.metrics.classification_metrics package

Submodules

mlm_insights.core.metrics.classification_metrics.accuracy_score module

class mlm_insights.core.metrics.classification_metrics.accuracy_score.AccuracyScore(config: ~typing.Dict[str, ~mlm_insights.constants.definitions.ConfigParameter] = <factory>, target_column: str = 'y_true', prediction_column: str = 'y_predict', actual_score: float = 0.0, total_count: int = 0)

Bases: DatasetMetricBase

This metric calculates the Accuracy score of a classification model. Accuracy is the ratio of the number of correct predictions to the total number of input samples.
This is a dataset metric which can process numerical as well as categorical data (target and prediction columns)
This is an exact metric.
This metric doesn’t handle NaN values. If there is a single NaN value in the target or prediction column, it will throw InvalidTargetPredictionException

Configuration

None

Returns

  • float: number of correctly classified samples

  • float: fraction of correctly classified samples

Examples

import numpy as np
import pandas as pd

from mlm_insights.builder.builder_component import MetricDetail, EngineDetail
from mlm_insights.builder.insights_builder import InsightsBuilder
from mlm_insights.constants.types import FeatureType, DataType, VariableType, ColumnType
from mlm_insights.core.metrics.classification_metrics.accuracy_score import AccuracyScore
from mlm_insights.core.metrics.metric_metadata import MetricMetadata


def main():
    input_schema = {
        'email_subject': FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.TEXT,
            column_type=ColumnType.INPUT),
        "spam_or_ham_prediction": FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.NOMINAL,
            column_type=ColumnType.PREDICTION),
        "spam_or_ham_target": FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.NOMINAL,
            column_type=ColumnType.TARGET)
    }

    y_true_binary = ['spam', 'spam', 'spam', 'ham', 'ham', 'ham', 'ham']
    y_pred_binary = ['spam', 'spam', 'ham', 'spam', 'spam', 'ham', 'ham']

    binary_class_data_frame = pd.DataFrame({'email_subject': np.random.rand(len(y_pred_binary)),
                                            'spam_or_ham_prediction': y_pred_binary,
                                            'spam_or_ham_target': y_true_binary
                                            })
    metric_details = MetricDetail(univariate_metric={},
                                  dataset_metrics=[MetricMetadata(klass=AccuracyScore)])

    runner = InsightsBuilder().                 with_input_schema(input_schema=input_schema).                 with_data_frame(data_frame=binary_class_data_frame).                 with_metrics(metrics=metric_details).                 with_engine(engine=EngineDetail(engine_name="native")).                 build()

    profile_json = runner.run().profile.to_json()
    dataset_metrics = profile_json['dataset_metrics']
    print(dataset_metrics["AccuracyScore"])


if __name__ == "__main__":
    main()

Returns the standard metric result as:
{
    'metric_name': 'AccuracyScore',
    'metric_description': 'Computes Accuracy classification score',
    'variable_count': 2,
    'variable_names': ['actual_score', 'normalized_accuracy_score'],
    'variable_types': [CONTINUOUS, CONTINUOUS],
    'variable_dtypes': [FLOAT, FLOAT],
    'variable_dimensions': [0, 0],
    'metric_data': [4.0, 0.5714],
    'metadata': {},
    'error': None
}
actual_score: float = 0.0
compute(dataset: DataFrame, **kwargs: Any) None

Computes accuracy score for the passed in dataset

Parameters

dataset : pd.DataFrame DataFrame object for either the entire dataset for a partition on which a Metric is being computed

classmethod create(config: Dict[str, ConfigParameter] | None = None, **kwargs: Any) AccuracyScore

Create an accuracy score metric using the configuration and kwargs

Parameters

config : Metric configuration kwargs: Key value pair for dynamic arguments. The current kwargs contains:

  • features_metadata: Contains input schema for each feature

get_result(**kwargs: Any) Dict[str, Any]

Returns the computed value of the metric

Returns

Dict[str, Any]: Dictionary with key as string and value as any metric property.

get_standard_metric_result(**kwargs: Any) StandardMetricResult

Returns Accuracy score Metric in Standard format.

Returns

StandardMetricResult: Accuracy score Metric in Standard format.

merge(other_metric: AccuracyScore, **kwargs: Any) AccuracyScore

Merge two Accuracy Score into one, without mutating the others.

Parameters

other_metricAccuracy Score metric

Other Accuracy Score that need be merged.

Returns

TypeMetric

A new instance of Accuracy Score

prediction_column: str = 'y_predict'
target_column: str = 'y_true'
total_count: int = 0

mlm_insights.core.metrics.classification_metrics.common module

mlm_insights.core.metrics.classification_metrics.common.compute_average_micro(multi_class_cm: MultiClassConfusionMatrix) float
mlm_insights.core.metrics.classification_metrics.common.compute_precision_score_macro(multi_class_cm: MultiClassConfusionMatrix) float
mlm_insights.core.metrics.classification_metrics.common.compute_precision_score_weighted(multi_class_cm: MultiClassConfusionMatrix) float
mlm_insights.core.metrics.classification_metrics.common.get_metric_result(ovr_list: List[float] | None = None, labels: List[str | int] | None = None, macro: float | None = None, micro: float | None = None, weighted: float | None = None, wrap_result: bool = True) Dict[str, Any]
This method returns metric result for Precision, Recall, Fbeta metrics. These metrics have the same shape , hence the method has been refactored to a common location

Parameters

  • ovr_list : Score for each label versus the rest of the labels

  • labels: List of labels present in the model dataset

  • macro: Macro averaging output where metrics is calculated for each label and then return their unweighted mean

  • micro: Micro averaging output where metrics is computed globally by counting the total true positives, false negatives and false positives

  • weighted: Weighted averaging output where metrics is computed for each label, and then return the average weights

mlm_insights.core.metrics.classification_metrics.confusion_matrix module

class mlm_insights.core.metrics.classification_metrics.confusion_matrix.ConfusionMatrix(config: ~typing.Dict[str, ~mlm_insights.constants.definitions.ConfigParameter] = <factory>)

Bases: DatasetMetricBase

This metric calculates the Confusion Matrix for a classification model. A confusion matrix is a table that is used to define the performance of a classification model.
This is a dataset metric which can process numerical as well as categorical data (target and prediction columns)
This is an exact metric.

The result returned by this metric is a Confusion matrix whose i-th row and j-th column entry indicates the number of samples with true label being i-th class and predicted label being j-th class

Configuration

None

Returns

matrix: List[List[Integer]]
  • confusion matrix for the classification model

labels: List[String]
  • Labels or classes within the classification model

Examples

import numpy as np
import pandas as pd

from mlm_insights.builder.builder_component import MetricDetail, EngineDetail
from mlm_insights.builder.insights_builder import InsightsBuilder
from mlm_insights.constants.types import FeatureType, DataType, VariableType, ColumnType
from mlm_insights.core.metrics.classification_metrics.confusion_matrix import ConfusionMatrix
from mlm_insights.core.metrics.classification_metrics.recall_score import RecallScore
from mlm_insights.core.metrics.metric_metadata import MetricMetadata


def main():
    input_schema = {
        'email_subject': FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.TEXT,
            column_type=ColumnType.INPUT),
        "spam_or_ham_prediction": FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.NOMINAL,
            column_type=ColumnType.PREDICTION),
        "spam_or_ham_target": FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.NOMINAL,
            column_type=ColumnType.TARGET)
    }

    y_true_binary = ['spam', 'spam', 'spam', 'ham', 'ham', 'ham', 'ham']
    y_pred_binary = ['spam', 'spam', 'ham', 'spam', 'spam', 'ham', 'ham']

    binary_class_data_frame = pd.DataFrame({'email_subject': np.random.rand(len(y_pred_binary)),
                                            'spam_or_ham_prediction': y_pred_binary,
                                            'spam_or_ham_target': y_true_binary
                                            })
    metric_details = MetricDetail(univariate_metric={},
                                  dataset_metrics=[MetricMetadata(klass=RecallScore),
                                                   MetricMetadata(klass=ConfusionMatrix)])

    runner = InsightsBuilder().                 with_input_schema(input_schema=input_schema).                 with_data_frame(data_frame=binary_class_data_frame).                 with_metrics(metrics=metric_details).                 with_engine(engine=EngineDetail(engine_name="native")).                 build()

    profile_json = runner.run().profile.to_json()
    dataset_metrics = profile_json['dataset_metrics']
    print(dataset_metrics["ConfusionMatrix"])


if __name__ == "__main__":
    main()

Returns the standard metric result as:
{
    'metric_name': 'ConfusionMatrix',
    'metric_description': 'Metric to compute confusion matrix to evaluate the accuracy of a classification.'
    'variable_count': 2,
    'variable_names': ['matrix', 'labels'],
    'variable_types': [DISCRETE, NOMINAL],
    'variable_dtypes': [INTEGER, STRING],
    'variable_dimensions': [2, 1],
    'metric_data': [[[2,2],[2,1]], ['ham', 'spam']],
    'metadata': {},
    'error': None
}
classmethod create(config: Dict[str, ConfigParameter] | None = None, **kwargs: Any) ConfusionMatrix

Factory Method to create an object. The configuration will be available in config.

Returns

MetricBase

An Instance of MetricBase.

get_required_shareable_dataset_components(**kwargs: Any) List[SDCMetaData]

Returns the Shareable Dataset Components that a Metric requires to compute its state and values Metrics which do not require SDC need not override this property

Returns

List of SDCMetaData. Each SDCMetaData must contain the klass attribute which points to the SDC class

get_result(**kwargs: Any) Dict[str, Any]

Returns the computed value of the metric

Returns

Dict[str, Any]: Dictionary with key as string and value as any metric property.

get_standard_metric_result(**kwargs: Any) StandardMetricResult

Returns Confusion Matrix Metric in Standard format.

Returns

StandardMetricResult: Confusion Matrix Metric in Standard format.

mlm_insights.core.metrics.classification_metrics.false_negative_rate module

class mlm_insights.core.metrics.classification_metrics.false_negative_rate.FalseNegativeRate(config: ~typing.Dict[str, ~mlm_insights.constants.definitions.ConfigParameter] = <factory>, labels: ~typing.List[str | int] | None = None)

Bases: DatasetMetricBase

This metric calculates the False Negative Rate of a classification model.
FalseNegativeRate is the ratio fn / (fn + tp) where fn is the number of false negatives and tp the number of true positives. The false negative rate is the proportion of positive transactions that were incorrectly scored as negative by the classifier.
The best value is 0 and the worst value is 1.
This is a dataset metric which can process numerical as well as categorical data (target and prediction columns)
This is an exact metric.

Configuration

labels: List[Union[str, int]], default=None
  • When no labels are provided to the metric, the metric returns empty result.

  • When labels are provided to the metric, the metric returns one-versus-rest FNR for each of the provided labels

Examples

import numpy as np
import pandas as pd

from mlm_insights.builder.builder_component import MetricDetail, EngineDetail
from mlm_insights.builder.insights_builder import InsightsBuilder
from mlm_insights.constants.types import FeatureType, DataType, VariableType, ColumnType
from mlm_insights.core.metrics.classification_metrics.false_negative_rate import FalseNegativeRate
from mlm_insights.core.metrics.metric_metadata import MetricMetadata


def main():
    input_schema = {
        'email_subject': FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.TEXT,
            column_type=ColumnType.INPUT),
        "spam_or_ham_prediction": FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.NOMINAL,
            column_type=ColumnType.PREDICTION),
        "spam_or_ham_target": FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.NOMINAL,
            column_type=ColumnType.TARGET)
    }

    y_true_binary = ['spam', 'spam', 'spam', 'ham', 'ham', 'ham', 'ham']
    y_pred_binary = ['spam', 'spam', 'ham', 'spam', 'spam', 'ham', 'ham']

    binary_class_data_frame = pd.DataFrame({'email_subject': np.random.rand(len(y_pred_binary)),
                                            'spam_or_ham_prediction': y_pred_binary,
                                            'spam_or_ham_target': y_true_binary
                                            })
    metric_details = MetricDetail(univariate_metric={},
                                  dataset_metrics=[MetricMetadata(klass=FalseNegativeRate)])

    runner = InsightsBuilder().                 with_input_schema(input_schema=input_schema).                 with_data_frame(data_frame=binary_class_data_frame).                 with_metrics(metrics=metric_details).                 with_engine(engine=EngineDetail(engine_name="native")).                 build()

    profile_json = runner.run().profile.to_json()
    dataset_metrics = profile_json['dataset_metrics']
    print(dataset_metrics["FalseNegativeRate"])


if __name__ == "__main__":
    main()


Returns the standard metric result as:
{
    'metric_name': 'FalseNegativeRate',
    'metric_description': 'Metric computes the False Negative Rate.'                          ' FalseNegativeRate is the ratio ``fn / (fn + tp)``'                          ' where ``fn`` is the number of false negatives and ``tp`` the number of true positives.'                          ' The false negative rate is the proportion of positive transactions that'                          ' were incorrectly scored as negative by the classifier.'
    'variable_count': 2,
    'variable_names': ['ovr', 'labels'],
    'variable_types': [CONTINUOUS, NOMINAL],
    'variable_dtypes': [FLOAT, STRING],
    'variable_dimensions': [1, 1],
    'metric_data': [[], []],
    'metadata': {},
    'error': None
}
classmethod create(config: Dict[str, ConfigParameter] | None = None, **kwargs: Any) FalseNegativeRate

Factory Method to create an object. The configuration will be available in config.

Returns

MetricBase

An Instance of MetricBase.

get_required_shareable_dataset_components(**kwargs: Any) List[SDCMetaData]

Returns the Shareable Dataset Components that a Metric requires to compute its state and values Metrics which do not require SDC need not override this property

Returns

List of SDCMetaData. Each SDCMetaData must contain the klass attribute which points to the SDC class

get_result(**kwargs: Any) Dict[str, Any]

Returns the computed value of the metric

Returns

Dict[str, Any]: Dictionary with key as string and value as any metric property.

get_standard_metric_result(**kwargs: Any) StandardMetricResult

Returns FalseNegativeRate Metric in Standard format.

Returns

StandardMetricResult: FalseNegativeRate Metric in Standard format.

labels: List[str | int] = None
merge(other_metric: FalseNegativeRate, **kwargs: Any) FalseNegativeRate

Merge two FalseNegativeRate metric into one, without mutating the others.

Parameters

other_metricFalseNegativeRate

Other FalseNegativeRate metric that need be merged.

Returns

TypeMetric

A new instance of FalseNegativeRate

mlm_insights.core.metrics.classification_metrics.false_positive_rate module

class mlm_insights.core.metrics.classification_metrics.false_positive_rate.FalsePositiveRate(config: ~typing.Dict[str, ~mlm_insights.constants.definitions.ConfigParameter] = <factory>, labels: ~typing.List[str | int] | None = None)

Bases: DatasetMetricBase

This metric calculates the False Positive Rate of a classification model (Type 1 Error).
False Positive Rate is the ratio fp / (fp + tn) where fp is the number of false positives and tn the number of true negatives. It indicates number of items wrongly classified as positive out of the total actual negatives
The best value is 0 and the worst value is 1.
This is a dataset metric which can process numerical as well as categorical data (target and prediction columns)
This is an exact metric.

Configuration

labels: List[Union[str, int]], default=None
  • When no labels are provided to the metric, the metric returns empty result

  • When labels are provided to the metric, the metric returns one-versus-rest FPR for each of the provided labels

Examples

import numpy as np
import pandas as pd

from mlm_insights.builder.builder_component import MetricDetail, EngineDetail
from mlm_insights.builder.insights_builder import InsightsBuilder
from mlm_insights.constants.types import FeatureType, DataType, VariableType, ColumnType
from mlm_insights.core.metrics.classification_metrics.false_positive_rate import FalsePositiveRate
from mlm_insights.core.metrics.metric_metadata import MetricMetadata


def main():
    input_schema = {
        'email_subject': FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.TEXT,
            column_type=ColumnType.INPUT),
        "spam_or_ham_prediction": FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.NOMINAL,
            column_type=ColumnType.PREDICTION),
        "spam_or_ham_target": FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.NOMINAL,
            column_type=ColumnType.TARGET)
    }

    y_true_binary = ['spam', 'spam', 'spam', 'ham', 'ham', 'ham', 'ham']
    y_pred_binary = ['spam', 'spam', 'ham', 'spam', 'spam', 'ham', 'ham']

    binary_class_data_frame = pd.DataFrame({'email_subject': np.random.rand(len(y_pred_binary)),
                                            'spam_or_ham_prediction': y_pred_binary,
                                            'spam_or_ham_target': y_true_binary
                                            })
    metric_details = MetricDetail(univariate_metric={},
                                  dataset_metrics=[MetricMetadata(klass=FalsePositiveRate)])

    runner = InsightsBuilder().                 with_input_schema(input_schema=input_schema).                 with_data_frame(data_frame=binary_class_data_frame).                 with_metrics(metrics=metric_details).                 with_engine(engine=EngineDetail(engine_name="native")).                 build()

    profile_json = runner.run().profile.to_json()
    dataset_metrics = profile_json['dataset_metrics']
    print(dataset_metrics["FalsePositiveRate"])


if __name__ == "__main__":
    main()


Returns the standard metric result as:
{
    'metric_name': 'FalsePositiveRate',
    'metric_description': 'Metric computes the False Positive Rate (Type 1 Error).'                      ' False Positive Rate is the ratio ``fp / (fp + tn)`` where'                      ' ``fp`` is the number of false positives and ``tn`` the number of true negatives.'                      ' It indicates number of items wrongly classified as positive out of the total actual negatives'
    'variable_count': 2,
    'variable_names': ['ovr', 'labels'],
    'variable_types': [CONTINUOUS, NOMINAL],
    'variable_dtypes': [FLOAT, STRING],
    'variable_dimensions': [1, 1],
    'metric_data': [[], []],
    'metadata': {},
    'error': None
}
classmethod create(config: Dict[str, ConfigParameter] | None = None, **kwargs: Any) FalsePositiveRate

Factory Method to create an object. The configuration will be available in config.

Returns

MetricBase

An Instance of MetricBase.

get_required_shareable_dataset_components(**kwargs: Any) List[SDCMetaData]

Returns the Shareable Dataset Components that a Metric requires to compute its state and values Metrics which do not require SDC need not override this property

Returns

List of SDCMetaData. Each SDCMetaData must contain the klass attribute which points to the SDC class

get_result(**kwargs: Any) Dict[str, Any]

Returns the computed value of the metric

Returns

Dict[str, Any]: Dictionary with key as string and value as any metric property.

get_standard_metric_result(**kwargs: Any) StandardMetricResult

Returns FalsePositiveRate Metric in Standard format.

Returns

StandardMetricResult: FalsePositiveRate Metric in Standard format.

labels: List[str | int] = None
merge(other_metric: FalsePositiveRate, **kwargs: Any) FalsePositiveRate

Merge two FalsePositiveRate metric into one, without mutating the others.

Parameters

other_metricFalsePositiveRate

Other FalsePositiveRate metric that need be merged.

Returns

TypeMetric

A new instance of FalsePositiveRate

mlm_insights.core.metrics.classification_metrics.fbeta_score module

class mlm_insights.core.metrics.classification_metrics.fbeta_score.FBetaScore(config: ~typing.Dict[str, ~mlm_insights.constants.definitions.ConfigParameter] = <factory>, labels: ~typing.List[str | int] | None = None, beta: float = 1)

Bases: DatasetMetricBase

This metric calculates the F-beta score of a classification model.
F-beta score is the weighted harmonic mean of precision and recall, reaching its optimal value at 1 and its worst value at 0.
This is a dataset metric which can process numerical as well as categorical data (target and prediction columns)
This is an exact metric.
The beta parameter determines the weight of recall in the combined score.
  • beta = 1, recall and the precision are equally important

  • beta < 1 lends more weight to precision

  • beta > 1 favors recall

Configuration

labels: List[Union[str, int]], default=None
  • When no labels are provided to the metric, the metric returns micro, macro and weighted average scores

  • When labels are provided to the metric, the metric returns one-versus-rest fbeta score for each of

the provided labels

beta: float, default=1
  • The default value of beta is set to 1, thus by default this metric computes F1 score

Examples

import numpy as np
import pandas as pd

from mlm_insights.builder.builder_component import MetricDetail, EngineDetail
from mlm_insights.builder.insights_builder import InsightsBuilder
from mlm_insights.constants.types import FeatureType, DataType, VariableType, ColumnType
from mlm_insights.core.metrics.classification_metrics.fbeta_score import FBetaScore
from mlm_insights.core.metrics.metric_metadata import MetricMetadata


def main():
    input_schema = {
        'email_subject': FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.TEXT,
            column_type=ColumnType.INPUT),
        "spam_or_ham_prediction": FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.NOMINAL,
            column_type=ColumnType.PREDICTION),
        "spam_or_ham_target": FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.NOMINAL,
            column_type=ColumnType.TARGET)
    }

    y_true_binary = ['spam', 'spam', 'spam', 'ham', 'ham', 'ham', 'ham']
    y_pred_binary = ['spam', 'spam', 'ham', 'spam', 'spam', 'ham', 'ham']

    binary_class_data_frame = pd.DataFrame({'email_subject': np.random.rand(len(y_pred_binary)),
                                            'spam_or_ham_prediction': y_pred_binary,
                                            'spam_or_ham_target': y_true_binary
                                            })
    metric_details = MetricDetail(univariate_metric={},
                                  dataset_metrics=[MetricMetadata(klass=FBetaScore)])

    runner = InsightsBuilder().                 with_input_schema(input_schema=input_schema).                 with_data_frame(data_frame=binary_class_data_frame).                 with_metrics(metrics=metric_details).                 with_engine(engine=EngineDetail(engine_name="native")).                 build()

    profile_json = runner.run().profile.to_json()
    dataset_metrics = profile_json['dataset_metrics']
    print(dataset_metrics["FBetaScore"])


if __name__ == "__main__":
    main()


Returns the standard metric result as:
{
    'metric_name': 'FBetaScore',
    'metric_description': 'Metric computes the F-beta score.'                          ' F-beta score is the weighted harmonic mean of precision and recall,'                          ' reaching its optimal value at 1 and its worst value at 0.'                          ' The beta parameter determines the weight of recall in the combined score.'
    'variable_count': 6,
    'variable_names': ['macro', 'micro', 'weighted', 'ovr', 'labels', 'beta_value'],
    'variable_types': [1, 1, 1, 1, 6, 1],
    'variable_dtypes': [2, 2, 2, 2, 6, 2],
    'variable_dtypes': [FLOAT, FLOAT, FLOAT, FLOAT, STRING, FLOAT],
    'variable_types': [CONTINUOUS, CONTINUOUS, CONTINUOUS, CONTINUOUS, TEXT, CONTINUOUS],
    'variable_dimensions': [0, 0, 0, 1, 1, 0],
    'metric_data': [4.0, 0.5714, 3.25, None, None, 1],
    'metadata': {},
    'error': None
}
beta: float = 1
classmethod create(config: Dict[str, ConfigParameter] | None = None, **kwargs: Any) FBetaScore

Factory Method to create an object. The configuration will be available in config.

Returns

MetricBase

An Instance of MetricBase.

get_required_shareable_dataset_components(**kwargs: Any) List[SDCMetaData]

Returns the Shareable Dataset Components that a Metric requires to compute its state and values Metrics which do not require SDC need not override this property

Returns

List of SDCMetaData. Each SDCMetaData must contain the klass attribute which points to the SDC class

get_result(**kwargs: Any) Dict[str, Any]

Returns the computed value of the metric

Returns

Dict[str, Any]: Dictionary with key as string and value as any metric property.

get_standard_metric_result(**kwargs: Any) StandardMetricResult

Returns FBetaScore Metric in Standard format.

Returns

StandardMetricResult: FBetaScore Metric in Standard format.

labels: List[str | int] = None
merge(other_metric: FBetaScore, **kwargs: Any) FBetaScore

Merge two FBetaScore metric into one, without mutating the others.

Parameters

other_metricFBetaScore

Other FBetaScore metric that need be merged.

Returns

TypeMetric

A new instance of FBetaScore

mlm_insights.core.metrics.classification_metrics.log_loss module

class mlm_insights.core.metrics.classification_metrics.log_loss.LogLoss(config: ~typing.Dict[str, ~mlm_insights.constants.definitions.ConfigParameter] = <factory>, target_column: str = 'y_true', prediction_score_column: str = 'y_score', actual_score: float = 0.0, total_count: int = 0, labels: ~typing.List[~typing.Any] | None = None)

Bases: DatasetMetricBase

This metric calculates the Log loss of a classification model.
This is the loss function used in (multinomial) logistic regression, defined as the negative log-likelihood of a logistic model that returns y_pred probabilities for its training data y_true.
The log loss is only defined for two or more labels. The log loss is non-negative and value closer to Zero is considered better.
This is a dataset metric which can process numerical as well as categorical data (target and prediction columns)
This is an exact metric.
Along with target and prediction columns, Log loss needs a prediction score column as well. Without the prediction score column defined, the metric will throw an Exception.
This metric doesn’t handle NaN values. If there is a single NaN value in the target or prediction column, it will throw InvalidTargetPredictionException

Configuration

labels: List[Union[str, int]], default=None
  • When no labels are provided to the metric, the metric assumes that data is of binary classification and 1 is considered as the positive label. If the target data is of multi-classification then loss and normalized loss is not calculated and None is returned.

  • When only 1 label is provided then the label is considered as positive label and data is assumed to be of binary classification.

  • When labels are provided to the metric, the labels are used to hot encode the prediction scores and the result is used to calculate the log loss. labels need to be provided in the same order as the values in prediction score column.

  • In case of any mismatch in labels and target data, loss and normalized loss is not calculated and None is returned.

Examples

from mlm_insights.core.metrics.classification_metrics.log_loss import LogLoss
classification_metrics = [
    MetricMetadata(klass=LogLoss)
]
metric_details = MetricDetail(univariate_metric={}, dataset_metrics=classification_metrics)

runner = InsightsBuilder().             with_input_schema(input_schema).             with_data_frame(data_frame=multi_class_data_frame).             with_metrics(metrics=metric_details).             with_engine(engine=EngineDetail(engine_name="native")).             build()

run_result = runner.run().profile.to_json()
dataset_metrics = profile_json['dataset_metrics']
dataset_metrics["LogLoss"]["value"]["loss"]

Returns the metric result as:
{
    'value': {
        'loss': log_loss,
        'normalized_loss' : normalized_log_loss,
    }
}
actual_score: float = 0.0
compute(dataset: DataFrame, **kwargs: Any) None

Computes log loss for the passed in dataset

Parameters

dataset : pd.DataFrame DataFrame object for either the entire dataset for a partition on which a Metric is being computed

classmethod create(config: Dict[str, ConfigParameter] | None = None, **kwargs: Any) LogLoss

Create a log loss metric using the configuration and kwargs

Parameters

config : Metric configuration kwargs: Key value pair for dynamic arguments. The current kwargs contains:

  • features_metadata: Contains input schema for each feature

get_result(**kwargs: Any) Dict[str, Any]

Returns the computed value of the metric

Returns

Dict[str, Any]: Dictionary with key as string and value as any metric property.

get_standard_metric_result(**kwargs: Any) StandardMetricResult

Returns log loss Score Metric in Standard format.

Returns

StandardMetricResult: log loss Score Metric in Standard format.

labels: List[Any] = None
merge(other_metric: LogLoss, **kwargs: Any) LogLoss

Merge two log loss Score into one, without mutating the others.

Parameters

other_metricLog Loss metric

Other Log Loss metric that need be merged.

Returns

TypeMetric

A new instance of Log Loss Metric

prediction_score_column: str = 'y_score'
target_column: str = 'y_true'
total_count: int = 0

mlm_insights.core.metrics.classification_metrics.precision_recall_auc module

class mlm_insights.core.metrics.classification_metrics.precision_recall_auc.PrecisionRecallAreaUnderCurve(config: ~typing.Dict[str, ~mlm_insights.constants.definitions.ConfigParameter] = <factory>, positive_label: str | int = 1)

Bases: DatasetMetricBase

This metric calculates the Area Under Curve for the Precision-Recall Curve.
A precision-recall curve is a graphical plot which illustrates the performance of a binary classifier system as its discrimination threshold is varied.
It is created by plotting precision (precision = tp / (tp + fp)) vs. recall (recall = tp / (tp + fn)), at various threshold settings. Once the precision-recall pairs are computed for different threshold values, the Area Under Curve is computed.
This is a dataset metric which can process numerical as well as categorical data (target and prediction columns)
This is an approximate metric.
This metric doesn’t handle NaN values. If there is a single NaN value in the target or prediction column, it will throw InvalidTargetPredictionException

Part of the implementation is taken from sklearn: https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/metrics/_ranking.py

Configuration

positive_label: int or str , default=1
  • The label of the positive class.

Examples

import pandas
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

from mlm_insights.builder.builder_component import MetricDetail, EngineDetail
from mlm_insights.builder.insights_builder import InsightsBuilder
from mlm_insights.constants.types import FeatureType, DataType, VariableType, ColumnType
from mlm_insights.core.metrics.classification_metrics.precision_recall_auc import PrecisionRecallAreaUnderCurve
from mlm_insights.core.metrics.metric_metadata import MetricMetadata


def main():
    input_schema = {
        "y_true": FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.NOMINAL,
            column_type=ColumnType.PREDICTION),
        "y_score": FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.NOMINAL,
            column_type=ColumnType.TARGET)
    }

    X, y = make_classification(n_samples=1000, n_classes=2, random_state=1)
    x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=2)
    model = LogisticRegression(solver='lbfgs')
    model.fit(x_train, y_train)
    y_score = model.predict_proba(x_test)[:, 1]
    df = pandas.DataFrame({'y_true': y_test, 'y_score': y_score})

    classification_metrics = [
        MetricMetadata(klass=PrecisionRecallAreaUnderCurve, config={"positive_label": 1})
    ]
    metric_details = MetricDetail(univariate_metric={}, dataset_metrics=classification_metrics)

    runner = InsightsBuilder().                 with_input_schema(input_schema).                 with_data_frame(data_frame=df).                 with_metrics(metrics=metric_details).                 with_engine(engine=EngineDetail(engine_name="native")).                 build()

    run_result = runner.run().profile.to_json()
    dataset_metrics = run_result['dataset_metrics']
    print(dataset_metrics["PrecisionRecallAreaUnderCurve"])


if __name__ == "__main__":
    main()


Returns the metric result as:
  return {
        "value": auc
    }
classmethod create(config: Dict[str, ConfigParameter] | None = None, **kwargs: Any) PrecisionRecallAreaUnderCurve

Factory Method to create an object. The configuration will be available in config.

Returns

MetricBase

An Instance of MetricBase.

get_required_shareable_dataset_components(**kwargs: Any) List[SDCMetaData]

Returns the Shareable Dataset Components that a Metric requires to compute its state and values Metrics which do not require SDC need not override this property

Returns

List of SDCMetaData. Each SDCMetaData must contain the klass attribute which points to the SDC class

get_result(**kwargs: Any) Dict[str, Any]

Returns the computed value of the metric

Returns

Dict[str, Any]: Dictionary with key as string and value as any metric property.

get_standard_metric_result(**kwargs: Any) StandardMetricResult

Returns PrecisionRecallAreaUnderCurve Metric in Standard format.

Returns

StandardMetricResult: PrecisionRecallAreaUnderCurve Metric in Standard format.

merge(other_metric: PrecisionRecallAreaUnderCurve, **kwargs: Any) PrecisionRecallAreaUnderCurve

Merge two PrecisionRecallAreaUnderCurve metric into one, without mutating the others.

Parameters

other_metricPrecisionRecallAreaUnderCurve

Other PrecisionScore metric that need be merged.

Returns

PrecisionRecallAreaUnderCurve

A new instance of PrecisionRecallAreaUnderCurve

positive_label: str | int = 1

mlm_insights.core.metrics.classification_metrics.precision_recall_curve module

class mlm_insights.core.metrics.classification_metrics.precision_recall_curve.PrecisionRecallCurve(config: ~typing.Dict[str, ~mlm_insights.constants.definitions.ConfigParameter] = <factory>, positive_label: str | int = 1)

Bases: DatasetMetricBase

This metric computes the Precision-Recall Curve, by computing precision-recall pairs for different probability thresholds.
A precision-recall curve is a graphical plot which illustrates the performance of a binary classifier system as its discrimination threshold is varied.
It is created by plotting precision (precision = tp / (tp + fp)) vs. recall (recall = tp / (tp + fn)), at various threshold settings.
This is a dataset metric which can process numerical as well as categorical data (target and prediction columns)
This is an approximate metric.
This metric doesn’t handle NaN values. If there is a single NaN value in the target or prediction column, it will throw InvalidTargetPredictionException

Configuration

positive_label: int or str , default=1
  • The label of the positive class.

Examples

import pandas
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

from mlm_insights.builder.builder_component import MetricDetail, EngineDetail
from mlm_insights.builder.insights_builder import InsightsBuilder
from mlm_insights.constants.types import FeatureType, DataType, VariableType, ColumnType
from mlm_insights.core.metrics.classification_metrics.precision_recall_curve import PrecisionRecallCurve
from mlm_insights.core.metrics.metric_metadata import MetricMetadata


def main():
    input_schema = {
        "y_true": FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.NOMINAL,
            column_type=ColumnType.PREDICTION),
        "y_score": FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.NOMINAL,
            column_type=ColumnType.TARGET)
    }

    X, y = make_classification(n_samples=1000, n_classes=2, random_state=1)
    x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=2)
    model = LogisticRegression(solver='lbfgs')
    model.fit(x_train, y_train)
    y_score = model.predict_proba(x_test)[:, 1]
    df = pandas.DataFrame({'y_true': y_test, 'y_score': y_score})

    classification_metrics = [
        MetricMetadata(klass=PrecisionRecallCurve, config={"positive_label": 1})
    ]
    metric_details = MetricDetail(univariate_metric={}, dataset_metrics=classification_metrics)

    runner = InsightsBuilder().                 with_input_schema(input_schema).                 with_data_frame(data_frame=df).                 with_metrics(metrics=metric_details).                 with_engine(engine=EngineDetail(engine_name="native")).                 build()

    run_result = runner.run().profile.to_json()
    dataset_metrics = run_result['dataset_metrics']
    print(dataset_metrics["PrecisionRecallCurve"])

if __name__ == "__main__":
    main()

Returns the metric result as:
{
    "value": {
        "precision": precision,
        "recall": recall,
        "thresholds": thresholds
    }
}
classmethod create(config: Dict[str, ConfigParameter] | None = None, **kwargs: Any) PrecisionRecallCurve

Factory Method to create an object. The configuration will be available in config.

Returns

MetricBase

An Instance of MetricBase.

get_required_shareable_dataset_components(**kwargs: Any) List[SDCMetaData]

Returns the Shareable Dataset Components that a Metric requires to compute its state and values Metrics which do not require SDC need not override this property

Returns

List of SDCMetaData. Each SDCMetaData must contain the klass attribute which points to the SDC class

get_result(**kwargs: Any) Dict[str, Any]

Returns the computed value of the metric

Returns

Dict[str, Any]: Dictionary with key as string and value as any metric property.

get_standard_metric_result(**kwargs: Any) StandardMetricResult

Returns PrecisionRecallCurve Metric in Standard format.

Returns

StandardMetricResult: PrecisionRecallCurve Metric in Standard format.

merge(other_metric: PrecisionRecallCurve, **kwargs: Any) PrecisionRecallCurve

Merge two PrecisionRecallCurve metric into one, without mutating the others.

Parameters

other_metricPrecisionRecallCurve

Other PrecisionScore metric that need be merged.

Returns

PrecisionRecallCurve

A new instance of PrecisionRecallCurve

positive_label: str | int = 1

mlm_insights.core.metrics.classification_metrics.precision_score module

class mlm_insights.core.metrics.classification_metrics.precision_score.PrecisionScore(config: ~typing.Dict[str, ~mlm_insights.constants.definitions.ConfigParameter] = <factory>, labels: ~typing.List[str | int] | None = None)

Bases: DatasetMetricBase

This metric calculates the Precision score of a classification model.
The precision is the ratio tp / (tp + fp) where tp is the number of true positives and fp the number of false positives. The precision is intuitively the ability of the classifier not to label as positive a sample that is negative.
The best value is 1 and the worst value is 0.
This is a dataset metric which can process numerical as well as categorical data (target and prediction columns)
This is an exact metric.

Configuration

labels: List[Union[str, int]], default=None
  • When no labels are provided to the metric, the metric returns micro, macro and weighted average scores

  • When labels are provided to the metric, the metric returns one-versus-rest precision score for each of the provided labels

Examples

import numpy as np
import pandas as pd

from mlm_insights.builder.builder_component import MetricDetail, EngineDetail
from mlm_insights.builder.insights_builder import InsightsBuilder
from mlm_insights.constants.types import FeatureType, DataType, VariableType, ColumnType
from mlm_insights.core.metrics.classification_metrics.precision_score import PrecisionScore
from mlm_insights.core.metrics.metric_metadata import MetricMetadata


def main():
    input_schema = {
        'email_subject': FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.TEXT,
            column_type=ColumnType.INPUT),
        "spam_or_ham_prediction": FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.NOMINAL,
            column_type=ColumnType.PREDICTION),
        "spam_or_ham_target": FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.NOMINAL,
            column_type=ColumnType.TARGET)
    }

    y_true_binary = ['spam', 'spam', 'spam', 'ham', 'ham', 'ham', 'ham']
    y_pred_binary = ['spam', 'spam', 'ham', 'spam', 'spam', 'ham', 'ham']

    binary_class_data_frame = pd.DataFrame({'email_subject': np.random.rand(len(y_pred_binary)),
                                            'spam_or_ham_prediction': y_pred_binary,
                                            'spam_or_ham_target': y_true_binary
                                            })
    metric_details = MetricDetail(univariate_metric={},
                                  dataset_metrics=[MetricMetadata(klass=PrecisionScore)])

    runner = InsightsBuilder().                 with_input_schema(input_schema=input_schema).                 with_data_frame(data_frame=binary_class_data_frame).                 with_metrics(metrics=metric_details).                 with_engine(engine=EngineDetail(engine_name="native")).                 build()

    profile_json = runner.run().profile.to_json()
    dataset_metrics = profile_json['dataset_metrics']
    print(dataset_metrics["PrecisionScore"])


if __name__ == "__main__":
    main()


Returns the standard metric result as:
{
    'metric_name': 'PrecisionScore',
    'metric_description': 'This metric computes the Precision.The precision is the ratio ``tp / (tp + fp)``'                          ' where ``tp`` is the number of true positives and ``fp`` the number of false positives.'                          ' The precision is intuitively the ability of the classifier not to label as positive'                          ' a sample that is negative.'                          'The best value is 1 and the worst value is 0.',
    'variable_count': 5,
    'variable_names': ['macro', 'micro', 'weighted', 'ovr', 'labels'],
    'variable_dtypes': [FLOAT, FLOAT, FLOAT, FLOAT, STRING],
    'variable_types': [CONTINUOUS, CONTINUOUS, CONTINUOUS, CONTINUOUS, NOMINAL],
    'variable_dimensions': [0, 0, 0, 1, 1],
    'metric_data': [4.0, 0.5714],
    'metadata': {},
    'error': None
}
classmethod create(config: Dict[str, ConfigParameter] | None = None, **kwargs: Any) PrecisionScore

Factory Method to create an object. The configuration will be available in config.

Returns

MetricBase

An Instance of MetricBase.

get_required_shareable_dataset_components(**kwargs: Any) List[SDCMetaData]

Returns the Shareable Dataset Components that a Metric requires to compute its state and values Metrics which do not require SDC need not override this property

Returns

List of SDCMetaData. Each SDCMetaData must contain the klass attribute which points to the SDC class

get_result(**kwargs: Any) Dict[str, Any]

Returns the computed value of the metric

Returns

Dict[str, Any]: Dictionary with key as string and value as any metric property.

get_standard_metric_result(**kwargs: Any) StandardMetricResult

Returns PrecisionScore Metric in Standard format.

Returns

StandardMetricResult: PrecisionScore Metric in Standard format.

labels: List[str | int] = None
merge(other_metric: PrecisionScore, **kwargs: Any) PrecisionScore

Merge two PrecisionScore metric into one, without mutating the others.

Parameters

other_metricPrecisionScore

Other PrecisionScore metric that need be merged.

Returns

TypeMetric

A new instance of PrecisionScore

mlm_insights.core.metrics.classification_metrics.recall_score module

class mlm_insights.core.metrics.classification_metrics.recall_score.RecallScore(config: ~typing.Dict[str, ~mlm_insights.constants.definitions.ConfigParameter] = <factory>, labels: ~typing.List[str | int] | None = None)

Bases: DatasetMetricBase

This metric computes the recall score of a classification model.
Recall is the ratio tp / (tp + fn) where tp is the number of true positives and fn the number of false negatives. The recall is intuitively the ability of the classifier to find all the positive samples.
The best value is 1 and the worst value is 0.
This is a dataset metric which can process numerical as well as categorical data (target and prediction columns)
This is an exact metric.

Configuration

labels: List[Union[str, int]], default=None
  • When no labels are provided to the metric, the metric returns micro, macro and weighted average scores

  • When labels are provided to the metric, the metric returns one-versus-rest recall score for each of the provided labels

Examples

import numpy as np
import pandas as pd

from mlm_insights.builder.builder_component import MetricDetail, EngineDetail
from mlm_insights.builder.insights_builder import InsightsBuilder
from mlm_insights.constants.types import FeatureType, DataType, VariableType, ColumnType
from mlm_insights.core.metrics.classification_metrics.recall_score import RecallScore
from mlm_insights.core.metrics.metric_metadata import MetricMetadata


def main():
    input_schema = {
        'email_subject': FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.TEXT,
            column_type=ColumnType.INPUT),
        'spam_or_ham_prediction': FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.NOMINAL,
            column_type=ColumnType.PREDICTION),
        'spam_or_ham_target': FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.NOMINAL,
            column_type=ColumnType.TARGET)
    }

    y_true_binary = ['spam', 'spam', 'spam', 'ham', 'ham', 'ham', 'ham']
    y_pred_binary = ['spam', 'spam', 'ham', 'spam', 'spam', 'ham', 'ham']

    binary_class_data_frame = pd.DataFrame({'email_subject': np.random.rand(len(y_pred_binary)),
                                            'spam_or_ham_prediction': y_pred_binary,
                                            'spam_or_ham_target': y_true_binary
                                            })
    metric_details = MetricDetail(univariate_metric={},
                                  dataset_metrics=[MetricMetadata(klass=RecallScore)])

    runner = InsightsBuilder().                 with_input_schema(input_schema=input_schema).                 with_data_frame(data_frame=binary_class_data_frame).                 with_metrics(metrics=metric_details).                 with_engine(engine=EngineDetail(engine_name="native")).                 build()

    profile_json = runner.run().profile.to_json()
    dataset_metrics = profile_json['dataset_metrics']
    print(dataset_metrics["RecallScore"])


if __name__ == "__main__":
    main()


Returns the standard metric result as:
{
    'metric_name': 'RecallScore',
    'metric_description': 'Metric computes the recall.Recall is the ratio tp / (tp + fn)'                          ' where tp is the number of true positives and fn the number of false negatives.'                          'The recall is intuitively the ability of the classifier to find all the positive samples',
    'variable_count': 5,
    'variable_names': ['macro', 'micro', 'weighted', 'ovr', 'labels'],
    'variable_dtypes': [FLOAT, FLOAT, FLOAT, FLOAT, STRING],
    'variable_types': [CONTINUOUS, CONTINUOUS, CONTINUOUS,
                        CONTINUOUS, NOMINAL],
    'variable_dimensions': [0, 0, 0, 1, 1],
    'metric_data': [4.0, 0.5714],
    'metadata': {},
    'error': None
}
classmethod create(config: Dict[str, ConfigParameter] | None = None, **kwargs: Any) RecallScore

Factory Method to create an object. The configuration will be available in config.

Returns

MetricBase

An Instance of MetricBase.

get_required_shareable_dataset_components(**kwargs: Any) List[SDCMetaData]

Returns the Shareable Dataset Components that a Metric requires to compute its state and values Metrics which do not require SDC need not override this property

Returns

List of SDCMetaData. Each SDCMetaData must contain the klass attribute which points to the SDC class

get_result(**kwargs: Any) Dict[str, Any]

Returns the computed value of the metric

Returns

Dict[str, Any]: Dictionary with key as string and value as any metric property.

get_standard_metric_result(**kwargs: Any) StandardMetricResult

Returns RecallScore Metric in Standard format.

Returns

StandardMetricResult: RecallScore Metric in Standard format.

labels: List[str | int] = None
merge(other_metric: RecallScore, **kwargs: Any) RecallScore

Merge two RecallScore metric into one, without mutating the others.

Parameters

other_metricRecallScore

Other RecallScore metric that need be merged.

Returns

TypeMetric

A new instance of RecallScore

mlm_insights.core.metrics.classification_metrics.roc module

class mlm_insights.core.metrics.classification_metrics.roc.ROCCurve(config: ~typing.Dict[str, ~mlm_insights.constants.definitions.ConfigParameter] = <factory>, positive_label: str | int = 1)

Bases: DatasetMetricBase

This metric computes the ROC Curve (receiver operating characteristic curve).
A receiver operating characteristic (ROC), or simply ROC curve, is a graphical plot which illustrates the performance of a binary classifier system as its discrimination threshold is varied.
It is created by plotting the fraction of true positives out of the positives (TPR = true positive rate) vs. the fraction of false positives out of the negatives (FPR = false positive rate), at various threshold settings. TPR is also known as sensitivity, and FPR is one minus the specificity or true negative rate.
This is a dataset metric which can process numerical as well as categorical data (target and prediction columns)
This is an approximate metric.
This metric doesn’t handle NaN values. If there is a single NaN value in the target or prediction column, it will throw InvalidTargetPredictionException

Configuration

positive_label: int or str , default=1
  • The label of the positive class.

Examples

import pandas
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

from mlm_insights.builder.builder_component import MetricDetail, EngineDetail
from mlm_insights.builder.insights_builder import InsightsBuilder
from mlm_insights.constants.types import FeatureType, DataType, VariableType, ColumnType
from mlm_insights.core.metrics.classification_metrics.roc import ROCCurve
from mlm_insights.core.metrics.metric_metadata import MetricMetadata


def main():
    input_schema = {
        "y_true": FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.NOMINAL,
            column_type=ColumnType.PREDICTION),
        "y_score": FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.NOMINAL,
            column_type=ColumnType.TARGET)
    }

    X, y = make_classification(n_samples=1000, n_classes=2, random_state=1)
    x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=2)
    model = LogisticRegression(solver='lbfgs')
    model.fit(x_train, y_train)
    y_score = model.predict_proba(x_test)[:, 1]
    df = pandas.DataFrame({'y_true': y_test, 'y_score': y_score})

    classification_metrics = [
        MetricMetadata(klass=ROCCurve)
    ]
    metric_details = MetricDetail(univariate_metric={}, dataset_metrics=classification_metrics)

    runner = InsightsBuilder().                 with_input_schema(input_schema).                 with_data_frame(data_frame=df).                 with_metrics(metrics=metric_details).                 with_engine(engine=EngineDetail(engine_name="native")).                 build()

    run_result = runner.run().profile.to_json()
    dataset_metrics = run_result['dataset_metrics']
    print(dataset_metrics["ROCCurve"])

if __name__ == "__main__":
    main()

Returns the metric result as:
{
    "value": {
        "fpr": fpr,
        "tpr": tpr,
        "thresholds": thresholds
    }
}
classmethod create(config: Dict[str, ConfigParameter] | None = None, **kwargs: Any) ROCCurve

Factory Method to create an object. The configuration will be available in config.

Returns

MetricBase

An Instance of MetricBase.

get_required_shareable_dataset_components(**kwargs: Any) List[SDCMetaData]

Returns the Shareable Dataset Components that a Metric requires to compute its state and values Metrics which do not require SDC need not override this property

Returns

List of SDCMetaData. Each SDCMetaData must contain the klass attribute which points to the SDC class

get_result(**kwargs: Any) Dict[str, Any]

Returns the computed value of the metric

Returns

Dict[str, Any]: Dictionary with key as string and value as any metric property.

get_standard_metric_result(**kwargs: Any) StandardMetricResult

Returns ROC Curve Metric in Standard format.

Returns

StandardMetricResult: ROC Curve Metric in Standard format.

merge(other_metric: ROCCurve, **kwargs: Any) ROCCurve

Merge two ROCCurve metric into one, without mutating the others.

Parameters

other_metricROCCurve

Other ROCCurve metric that need be merged.

Returns

TypeMetric

A new instance of ROCCurve

positive_label: str | int = 1

mlm_insights.core.metrics.classification_metrics.roc_auc module

class mlm_insights.core.metrics.classification_metrics.roc_auc.ROCAreaUnderCurve(config: ~typing.Dict[str, ~mlm_insights.constants.definitions.ConfigParameter] = <factory>, positive_label: str | int = 1)

Bases: DatasetMetricBase

This metric computes the ROC Area under Curve
A receiver operating characteristic (ROC), or simply ROC curve, is a graphical plot which illustrates the performance of a binary classifier system as its discrimination threshold is varied.
It is created by plotting the fraction of true positives out of the positives (TPR = true positive rate) vs. the fraction of false positives out of the negatives (FPR = false positive rate), at various threshold settings. TPR is also known as sensitivity, and FPR is one minus the specificity or true negative rate.
This is a dataset metric which can process numerical as well as categorical data (target and prediction columns)
This is an approximate metric.
This metric doesn’t handle NaN values. If there is a single NaN value in the target or prediction column, it will throw InvalidTargetPredictionException

Configuration

positive_label: int or str, default=None
  • The label of the positive class. When pos_label=1,

Examples

import pandas
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

from mlm_insights.builder.builder_component import MetricDetail, EngineDetail
from mlm_insights.builder.insights_builder import InsightsBuilder
from mlm_insights.constants.types import FeatureType, DataType, VariableType, ColumnType
from mlm_insights.core.metrics.classification_metrics.roc_auc import ROCAreaUnderCurve
from mlm_insights.core.metrics.metric_metadata import MetricMetadata


def main():
    input_schema = {
        "y_true": FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.NOMINAL,
            column_type=ColumnType.PREDICTION),
        "y_score": FeatureType(
            data_type=DataType.STRING,
            variable_type=VariableType.NOMINAL,
            column_type=ColumnType.TARGET)
    }

    X, y = make_classification(n_samples=1000, n_classes=2, random_state=1)
    x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=2)
    model = LogisticRegression(solver='lbfgs')
    model.fit(x_train, y_train)
    y_score = model.predict_proba(x_test)[:, 1]
    df = pandas.DataFrame({'y_true': y_test, 'y_score': y_score})

    classification_metrics = [
        MetricMetadata(klass=ROCAreaUnderCurve, config={"positive_label": 1})
    ]
    metric_details = MetricDetail(univariate_metric={}, dataset_metrics=classification_metrics)

    runner = InsightsBuilder().                 with_input_schema(input_schema).                 with_data_frame(data_frame=df).                 with_metrics(metrics=metric_details).                 with_engine(engine=EngineDetail(engine_name="native")).                 build()

    run_result = runner.run().profile.to_json()
    dataset_metrics = run_result['dataset_metrics']
    print(dataset_metrics["ROCAreaUnderCurve"])


if __name__ == "__main__":
    main()


Returns the metric result as:
{
    "value": auc
}
classmethod create(config: Dict[str, ConfigParameter] | None = None, **kwargs: Any) ROCAreaUnderCurve

Factory Method to create an object. The configuration will be available in config.

Returns

MetricBase

An Instance of MetricBase.

get_required_shareable_dataset_components(**kwargs: Any) List[SDCMetaData]

Returns the Shareable Dataset Components that a Metric requires to compute its state and values Metrics which do not require SDC need not override this property

Returns

List of SDCMetaData. Each SDCMetaData must contain the klass attribute which points to the SDC class

get_result(**kwargs: Any) Dict[str, Any]

Returns the computed value of the metric

Returns

Dict[str, Any]: Dictionary with key as string and value as any metric property.

get_standard_metric_result(**kwargs: Any) StandardMetricResult

Returns ROC Area Under Curve Metric in Standard format.

Returns

StandardMetricResult: ROC Area Under Curve Metric in Standard format.

merge(other_metric: ROCAreaUnderCurve, **kwargs: Any) ROCAreaUnderCurve

Merge two ROCAreaUnderCurve metric into one, without mutating the others.

Parameters

other_metricROCAreaUnderCurve

Other ROCAreaUnderCurve metric that need be merged.

Returns

TypeMetric

A new instance of ROCAreaUnderCurve

positive_label: str | int = 1

mlm_insights.core.metrics.classification_metrics.specificity module

class mlm_insights.core.metrics.classification_metrics.specificity.Specificity(config: ~typing.Dict[str, ~mlm_insights.constants.definitions.ConfigParameter] = <factory>, labels: ~typing.List[str | int] | None = None)

Bases: DatasetMetricBase

This metric computes the Specificity or True Negative Rate of a classification model.
Specificity is the ratio tn / (fp + tn) where tn is the number of true negatives and fp the number of false positives.
The specificity is intuitively the ability of the classifier to find all the negative samples.
The best value is 1 and the worst value is 0.
This is a dataset metric which can process numerical as well as categorical data (target and prediction columns)
This is an exact metric.

Configuration

labels: List[Union[str, int]], default=None
  • When no labels are provided to the metric, the metric returns empty result

  • When labels are provided to the metric, the metric returns one-versus-rest specificity for each of the provided labels

Examples

import numpy as np
import pandas as pd

from mlm_insights.builder.builder_component import MetricDetail, EngineDetail
from mlm_insights.builder.insights_builder import InsightsBuilder
from mlm_insights.constants.types import FeatureType, DataType, VariableType, ColumnType
from mlm_insights.core.metrics.classification_metrics.false_positive_rate import FalsePositiveRate
from mlm_insights.core.metrics.metric_metadata import MetricMetadata

def main():
    input_schema = {
            'email_subject': FeatureType(
                data_type=DataType.STRING,
                variable_type=VariableType.TEXT,
                column_type=ColumnType.INPUT),
            "spam_or_ham_prediction": FeatureType(
                data_type=DataType.STRING,
                variable_type=VariableType.NOMINAL,
                column_type=ColumnType.PREDICTION),
            "spam_or_ham_target": FeatureType(
                data_type=DataType.STRING,
                variable_type=VariableType.NOMINAL,
                column_type=ColumnType.TARGET)
        }
    y_true_binary = ['spam', 'spam', 'spam', 'ham', 'ham', 'ham', 'ham']
    y_pred_binary = ['spam', 'spam', 'ham', 'spam', 'spam', 'ham', 'ham']

    binary_class_data_frame = pd.DataFrame({'email_subject': np.random.rand(len(y_pred_binary)),
                                                'spam_or_ham_prediction': y_pred_binary,
                                                'spam_or_ham_target': y_true_binary
                                                })
    metric_details = MetricDetail(univariate_metric={},
                                      dataset_metrics=[MetricMetadata(klass=FalsePositiveRate)])

    runner = InsightsBuilder().                     with_input_schema(input_schema=input_schema).                     with_data_frame(data_frame=binary_class_data_frame).                     with_metrics(metrics=metric_details).                     with_engine(engine=EngineDetail(engine_name="native")).                     build()

    profile_json = runner.run().profile.to_json()
    dataset_metrics = profile_json['dataset_metrics']
    print(dataset_metrics["FalsePositiveRate"])

if __name__ == "__main__":
        main()


Returns the standard metric result as:
{
    'metric_name': 'Specificity',
    'metric_description': 'Metric computes the Specificity or True Negative Rate.'                      ' Specificity is the ratio ``tn / (fp + tn)`` where'                      ' ``tn`` is the number of true negatives and ``fp`` the number of false positives.'                      ' The specificity is intuitively the ability of the classifier to find all the negative samples.'
    'variable_count': 2,
    'variable_names': ['ovr', 'labels'],
    'variable_dtypes': [FLOAT, STRING],
    'variable_types': [CONTINUOUS, NOMINAL],
    'variable_dimensions': [1, 1],
    'metric_data': [[], []],
    'metadata': {},
    'error': None
}
classmethod create(config: Dict[str, ConfigParameter] | None = None, **kwargs: Any) Specificity

Factory Method to create an object. The configuration will be available in config.

Returns

MetricBase

An Instance of MetricBase.

get_required_shareable_dataset_components(**kwargs: Any) List[SDCMetaData]

Returns the Shareable Dataset Components that a Metric requires to compute its state and values Metrics which do not require SDC need not override this property

Returns

List of SDCMetaData. Each SDCMetaData must contain the klass attribute which points to the SDC class

get_result(**kwargs: Any) Dict[str, Any]

Returns the computed value of the metric

Returns

Dict[str, Any]: Dictionary with key as string and value as any metric property.

get_standard_metric_result(**kwargs: Any) StandardMetricResult

Returns Specificity Metric in Standard format.

Returns

StandardMetricResult: Specificity Metric in Standard format.

labels: List[str | int] = None
merge(other_metric: Specificity, **kwargs: Any) Specificity

Merge two Specificity metric into one, without mutating the others.

Parameters

other_metricSpecificity

Other Specificity metric that need be merged.

Returns

TypeMetric

A new instance of Specificity

Module contents