4 Classification

Predict categorical targets using classification, a supervised machine learning techniquetechnique.

4.1 About Classification

Classification is a machine learning technique that assigns items to target categories or classes to predict outcomes.

The goal of classification is to accurately predict the target class for each case in the data. For example, a classification model can be used to identify loan applicants as low, medium, or high credit risks.

A classification task begins with a data set in which the class assignments are known. For example, a classification model that predicts credit risk can be developed based on observed data for many loan applicants over a period of time. In addition to the historical credit rating, the data might track employment history, home ownership or rental, years of residence, number and type of investments, and so on. Credit rating is the target, the other attributes are the predictors, and the data for each customer constitutes a case.

Classification are discrete and do not imply order. Continuous, floating-point values indicate a numerical, rather than a categorical, target. A predictive model with a numerical target uses a regression algorithm, not a classification algorithm.

The simplest type of classification problem is binary classification. In binary classification, the target attribute has only two possible values: for example, high credit rating or low credit rating. Multiclass targets have more than two values: for example, low, medium, high, or unknown credit rating.

In the model build (training) process, a classification algorithm finds relationships between the values of the predictors and the values of the target. Different classification algorithms use different techniques for finding relationships. These relationships are summarized in a model, which can then be applied to a different data set in which the class assignments are unknown.

Classification models are tested by comparing the predicted values to known target values in a set of test data. The historical data for a classification project is typically divided into two data sets: one for building the model; the other for testing the model.

Applying a classification model results in class assignments and probabilities for each case. For example, a model that classifies customers as low, medium, or high value also predicts the probability of each classification for each customer.

Classification has many applications in customer segmentation, business modeling, marketing, credit analysis, and biomedical and drug response modeling.

4.2 Testing a Classification Model

Apply the classification model to test data, compare predictions with actual outcomes, and evaluate accuracy using test metrics.

A classification model is tested by applying it to test data with known target values and comparing the predicted values with the known values.

The test data must be compatible with the data used to build the model and must be prepared in the same way that the build data was prepared. Typically the build data and test data come from the same historical data set. A percentage of the records is used to build the model; the remaining records are used to test the model.

Test metrics are used to assess how accurately the model predicts the known values. If the model performs well and meets the business requirements, it can then be applied to new data to predict the future.

4.2.1 Confusion Matrix

Display the number of correct and incorrect predictions compared to actual classifications using a confusion matrix.

A confusion matrix displays the number of correct and incorrect predictions made by the model compared with the actual classifications in the test data. The matrix is n-by-n, where n is the number of classes.

The following figure shows a confusion matrix for a binary classification model. The rows present the number of actual classifications in the test data. The columns present the number of predicted classifications made by the model.

Figure 4-1 Confusion Matrix for a Binary Classification Model

Description of Figure 4-1 follows
Description of "Figure 4-1 Confusion Matrix for a Binary Classification Model"

In this example, the model correctly predicted the positive class (also called true positive (TP)) for affinity_card 516 times and incorrectly predicted (also called false negative (FN)) it 25 times. The model correctly predicted the negative class (also called true negative (TN)) for affinity_card 725 times and incorrectly predicted (also called false positive (FP)) it 10 times. The following can be computed from this confusion matrix:

  • The model made 1241 correct predictions, that is, TP + TN, (516 + 725).

  • The model made 35 incorrect predictions, that is, FN + FP, (25 + 10).

  • There are 1276 total scored cases, (516 + 25 + 10 + 725).

  • The error rate is 35/1276 = 0.0274. (FN+FP/Total)

  • The overall accuracy rate is 1241/1276 = 0.9725 (TP+TN)/Total).

Precision and Recall

Consider the same example, the accuracy rate shows 0.97. However, there are cases where the model has incorrectly predicted. Precision (positive predicted value) is the ability of a classification model to return only relevant cases. Precision can be calculated as TP/TP+FP. Recall (sensitivity or true positive rate) is the ability of a classification model to return relevant cases. Recall can be calculated as TP/TP+FN. The precision in this example is 516/526 = 0.98. The recall in this example is 516/541 = 0.95. Ideally, the model is good when both precision and recall are 1. This can happen when the numerator and the denominator are equal. That means, for precision, FP is zero and for recall, FN is zero.

4.2.2 Lift

Lift measures the degree to which the predictions of a classification model are better than randomly-generated predictions.

Lift applies to binary classification only, and it requires the designation of a positive class. If the model itself does not have a binary target, you can compute lift by designating one class as positive and combining all the other classes together as one negative class.

Numerous statistics can be calculated to support the notion of lift. Basically, lift can be understood as a ratio of two percentages: the percentage of correct positive classifications made by the model to the percentage of actual positive classifications in the test data. For example, if 40% of the customers in a marketing survey have responded favorably (the positive classification) to a promotional campaign in the past and the model accurately predicts 75% of them, the lift is obtained by dividing .75 by .40. The resulting lift is 1.875.

Lift is computed against quantiles that each contain the same number of cases. The data is divided into quantiles after it is scored. It is ranked by probability of the positive class from highest to lowest, so that the highest concentration of positive predictions is in the top quantiles. A typical number of quantiles is 10.

Lift is commonly used to measure the performance of response models in marketing applications. The purpose of a response model is to identify segments of the population with potentially high concentrations of positive responders to a marketing campaign. Lift reveals how much of the population must be solicited to obtain the highest percentage of potential responders.

4.2.2.1 Lift Statistics

Oracle Machine Learning computes various lift statistics to evaluate the effectiveness of classification models in predicting positive outcomes.

Oracle Machine Learning computes the following lift statistics:

  • Probability threshold for a quantile n is the minimum probability for the positive target to be included in this quantile or any preceding quantiles (quantiles n-1, n-2,..., 1). If a cost matrix is used, a cost threshold is reported instead. The cost threshold is the maximum cost for the positive target to be included in this quantile or any of the preceding quantiles.

  • Cumulative gain is the ratio of the cumulative number of positive targets to the total number of positive targets.

  • Target density of a quantile is the number of true positive instances in that quantile divided by the total number of instances in the quantile.

  • Cumulative target density for quantile n is the target density computed over the first n quantiles.

  • Quantile lift is the ratio of the target density for the quantile to the target density over all the test data.

  • Cumulative percentage of records for a quantile is the percentage of all cases represented by the first n quantiles, starting at the end that is most confidently positive, up to and including the given quantile.

  • Cumulative number of targets for quantile n is the number of true positive instances in the first n quantiles.

  • Cumulative number of nontargets is the number of actually negative instances in the first n quantiles.

  • Cumulative lift for a quantile is the ratio of the cumulative target density to the target density over all the test data.

Related Topics

4.2.3 Receiver Operating Characteristic (ROC)

ROC is a metric for comparing predicted and actual target values in a classification model.

ROC, like Lift, applies to binary classification and requires the designation of a positive class.

You can use ROC to gain insight into the decision-making ability of the model. How likely is the model to accurately predict the negative or the positive class?

ROC measures the impact of changes in the probability threshold. The probability threshold is the decision point used by the model for classification. The default probability threshold for binary classification is 0.5. When the probability of a prediction is 50% or more, the model predicts that class. When the probability is less than 50%, the other class is predicted. (In multiclass classification, the predicted class is the one predicted with the highest probability.)

4.2.3.1 The ROC Curve

Plot ROC as a curve on an X-Y axis, showing true positive rate versus false positive rate.

ROC can be plotted as a curve on an X-Y axis. The false positive rate is placed on the X axis. The true positive rate is placed on the Y axis.

The top left corner is the optimal location on an ROC graph, indicating a high true positive rate and a low false positive rate.

4.2.3.2 Area Under the Curve

Measure model discrimination ability with the area under the ROC curve (AUC); higher AUC indicates better performance.

The area under the ROC curve (AUC) measures the discriminating ability of a binary classification model. The larger the AUC, the higher the likelihood that an actual positive case is assigned, and a higher probability of being positive than an actual negative case. The AUC measure is especially useful for data sets with unbalanced target distribution (one target class dominates the other).

4.2.3.3 ROC and Model Bias

Adjust probability thresholds to optimize accuracy or favor specific classes, using ROC to understand model bias.

The ROC curve for a model represents all the possible combinations of values in its confusion matrix. Changes in the probability threshold affect the predictions made by the model. For instance, if the threshold for predicting the positive class is changed from 0.5 to 0.6, then fewer positive predictions are made. This affects the distribution of values in the confusion matrix: the number of true and false positives and true and false negatives differ.

You can use ROC to find the probability thresholds that yield the highest overall accuracy or the highest per-class accuracy. For example, if it is important to you to accurately predict the positive class, but you don't care about prediction errors for the negative class, then you can lower the threshold for the positive class. This can bias the model in favor of the positive class.

A cost matrix is a convenient mechanism for changing the probability thresholds for model scoring.

Related Topics

4.2.3.4 ROC Statistics

Calculate ROC statistics to evaluate classification model performance, including true positives, false positives, and probability thresholds.

Oracle Machine Learning computes the following ROC statistics:

  • Probability threshold: The minimum predicted positive class probability resulting in a positive class prediction. Different threshold values result in different hit rates and different false alarm rates.

  • True negatives: Negative cases in the test data with predicted probabilities strictly less than the probability threshold (correctly predicted).

  • True positives: Positive cases in the test data with predicted probabilities greater than or equal to the probability threshold (correctly predicted).

  • False negatives: Positive cases in the test data with predicted probabilities strictly less than the probability threshold (incorrectly predicted).

  • False positives: Negative cases in the test data with predicted probabilities greater than or equal to the probability threshold (incorrectly predicted).

  • True positive fraction: Hit rate. (true positives/(true positives + false negatives))

  • False positive fraction: False alarm rate. (false positives/(false positives + true negatives))

4.3 Biasing a Classification Model

Adjust decision criteria using costs, prior probabilities, and class weights to influence model predictions.

Costs, prior probabilities, and class weights are methods for biasing classification models.

4.3.1 Costs

Influence model decisions by specifying a cost matrix to minimize costly misclassifications.

A cost matrix is a mechanism for influencing the decision making of a model. A cost matrix can cause the model to minimize costly misclassifications. It can also cause the model to maximize beneficial accurate classifications.

For example, if a model classifies a customer with poor credit as low risk, this error is costly. A cost matrix can bias the model to avoid this type of error. The cost matrix can also be used to bias the model in favor of the correct classification of customers who have the worst credit history.

ROC is a useful metric for evaluating how a model behaves with different probability thresholds. You can use ROC to help you find optimal costs for a given classifier given different usage scenarios. You can use this information to create cost matrices to influence the deployment of the model.

4.3.1.1 Costs Versus Accuracy

Compare cost and confusion matrices to evaluate model quality, considering both prediction accuracy and the relative importance of different predictions.

Like a confusion matrix, a cost matrix is an n-by-n matrix, where n is the number of classes. Both confusion matrices and cost matrices include each possible combination of actual and predicted results based on a given set of test data.

A confusion matrix is used to measure accuracy, the ratio of correct predictions to the total number of predictions. A cost matrix is used to specify the relative importance of accuracy for different predictions. In most business applications, it is important to consider costs in addition to accuracy when evaluating model quality.

Related Topics

4.3.1.2 Positive and Negative Classes

Identify and prioritize positive and negative classes in a confusion matrix, crucial for computing Lift and ROC metrics.

The positive class is the class that you care the most about. Designation of a positive class is required for computing Lift and ROC.

In the confusion matrix, in the following figure, the value 1 is designated as the positive class. This means that the creator of the model has determined that it is more important to accurately predict customers who increase spending with an affinity card (affinity_card=1) than to accurately predict non-responders (affinity_card=0). If you give affinity cards to some customers who are not likely to use them, there is little loss to the company since the cost of the cards is low. However, if you overlook the customers who are likely to respond, you miss the opportunity to increase your revenue.

Figure 4-2 Positive and Negative Predictions

Description of Figure 4-2 follows
Description of "Figure 4-2 Positive and Negative Predictions"

The true and false positive rates in this confusion matrix are:

  • False positive rate — 10/(10 + 725) =.01

  • True positive rate — 516/(516 + 25) =.95

4.3.1.3 Assigning Costs and Benefits

Use a cost matrix to influence model decisions by assigning costs to negative outcomes and benefits to positive outcomes.

In a cost matrix, positive numbers (costs) can be used to influence negative outcomes. Since negative costs are interpreted as benefits, negative numbers (benefits) can be used to influence positive outcomes.

Suppose you have calculated that it costs your business $1500 when you do not give an affinity card to a customer who can increase spending. Using the model with the confusion matrix shown in Figure 4-2, each false negative (misclassification of a responder) costs $1500. Misclassifying a non-responder is less expensive to your business. You estimate that each false positive (misclassification of a non-responder) only costs $300.

You want to keep these costs in mind when you design a promotion campaign. You estimate that it costs $10 to include a customer in the promotion. For this reason, you associate a benefit of $10 with each true negative prediction, because you can eliminate those customers from your promotion. Each customer that you eliminate represents a savings of $10. In your cost matrix, you specify this benefit as -10, a negative cost.

The following figure shows how you would represent these costs and benefits in a cost matrix:

Figure 4-3 Cost Matrix Representing Costs and Benefits

Description of Figure 4-3 follows
Description of "Figure 4-3 Cost Matrix Representing Costs and Benefits"

With Oracle Machine Learning you can specify costs to influence the scoring of any classification model. Decision Tree models can also use a cost matrix to influence the model build.

4.3.2 Priors and Class Weights

Offset differences in data distribution with prior probabilities and class weights to produce useful classification results.

With Bayesian models, you can specify Prior probabilities to offset differences in distribution between the build data and the real population (scoring data). With other forms of classification, you are able to specify Class Weights, which have the same biasing effect as priors.

In many problems, one target value dominates in frequency. For example, the positive responses for a telephone marketing campaign is 2% or less, and the occurrence of fraud in credit card transactions is less than 1%. A classification model built on historic data of this type cannot observe enough of the rare class to be able to distinguish the characteristics of the two classes; the result can be a model that when applied to new data predicts the frequent class for every case. While such a model can be highly accurate, it is not be very useful. This illustrates that it is not a good idea to rely solely on accuracy when judging the quality of a classification model.

To correct for unrealistic distributions in the training data, you can specify priors for the model build process. Other approaches to compensating for data distribution issues include stratified sampling and anomaly detection.

Related Topics

4.4 Classification Algorithms

Learn the different classification algorithms used in Oracle Machine Learning.

Oracle Machine Learning provides the following algorithms for classification:

  • Decision Tree

    Decision trees automatically generate rules, which are conditional statements that reveal the logic used to build the tree.

  • Explicit Semantic Analysis

    Explicit Semantic Analysis (ESA) is designed to make predictions for text data. This algorithm can address use cases with hundreds of thousands of classes.

  • Generalized Linear Model

    Generalized Linear Model (GLM) is a popular statistical technique for linear modeling. OML4SQL implements GLM for binary classification and for regression. GLM provides extensive coefficient statistics and model statistics, as well as row diagnostics. GLM also supports confidence bounds.

  • Naive Bayes

    Naive Bayes uses Bayes' Theorem, a formula that calculates a probability by counting the frequency of values and combinations of values in the historical data.

  • Random Forest

    Random Forest is a powerful and popular machine learning algorithm that brings significant performance and scalability benefits.

  • Support Vector Machine

    Support Vector Machine (SVM) is a powerful, state-of-the-art algorithm based on linear and nonlinear regression. OML4SQL implements SVM for binary and multiclass classification.

  • XGBoost

    XGBoost is machine learning algorithm for regression and classification that makes available the XGBoost open source package. Oracle Machine Learning for SQL XGBoost prepares training data, invokes XGBoost, builds and persists a model, and applies the model for prediction.

Note:

OML4SQL uses Naive Bayes as the default classification algorithm.