A confusion matrix is a summary of prediction results on a classification problem. The main feature of the confusion matrix is the number of correct and incorrect predictions which are summarized with count values and broken down by each class. The confusion matrix shows how your classification model is confused when it makes predictions. It provides an insight into the errors being made by a classifier and the types of errors that are being made.
Following are the terms used:
· Positive (P): Observation is positive (Bad /SAR /Suspicious).
· Negative (N): Observation is not positive (Good /Non-SAR /Non-Suspicious ).
· True Positive (TP): Observation is positive and is predicted to be positive.
· False Negative (FN): Observation is positive but is predicted negative.
· True Negative (TN): Observation is negative and is predicted to be negative.
· False Positive (FP): Observation is negative but is predicted positive.
· Classification Rate or Accuracy: The formula is as follows:
|
Accuracy = ( TP + TN ) / (TP + TN + FP + FN ) |
· Precision: This value gives the measure of how precise or accurate the model is out of those predicted positive and how many of them are positive.
|
Precision = TP / (TP + FP) |
· Recall: This value calculates how many of the actual positives of the model were captured by labeling it as a true positive.
|
Recall = TP / (TP + FN) |
· F1 Score: This value is the harmonic mean of precision and recall.
|
F1 Score = 2 * (Precision * Recall) / (Precision + Recall) |
To evaluate the confusion matrix plot for the KS cut-off, enter the commands as shown in the following paragraph:
|
%python
z.show( aif.show_model_performance( performance_metric = "Confusion Matrix", cut_off_method = "KS" ) )
aif.show_plots('png') |
The output is as shown in the following paragraph:
Figure 35: Kolmogorov-Smirnov (KS) Cut-off Plot Output
Plots are rendered as shown in the following paragraphs:
Figure 36: Confusion Matrix Plot @ Max ks cutoff
Figure 37: Confusion Matrix Plot @ Max ks cutoff (Continued…)
To evaluate the confusion matrix plot for the ROC curve, enter the commands as shown in the following paragraph:
|
%python
aif.show_model_performance( performance_metric = "ROC Curve" )
aif.show_plots() |
The plot is rendered as shown in the following paragraph:
Figure 38: ROC Curve Analysis
To evaluate the confusion matrix plot for Prediction Density, Precision-Recall, Kappa & F1, enter the commands as shown in the following paragraph:
|
%python
aif.show_model_performance( performance_metric = "Prediction Density" )
#aif.show_model_performance( performance_metric = 'Precision Recall Curve' ) #aif.show_model_performance( performance_metric = 'Kappa Curve' ) #aif.show_model_performance( performance_metric = 'F1 Curve' )
aif.show_plots() |
The plot is rendered as shown in the following paragraph:
Figure 39: Prediction Density, Precision-Recall, Kappa & F1 Plot Output
