Building and Explaining a Text Classifier using AutoMLx

by the Oracle AutoMLx Team

AutoMLx Text Classification

Copyright © 2025, Oracle and/or its affiliates.

Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl/

Overview of this Notebook¶

In this notebook we will build a classifier using the Oracle AutoMLx tool for the public 20newsgroup dataset. The dataset is a binary classification dataset, and more details about the dataset can be found at http://qwone.com/~jason/20Newsgroups/. We explore the various options provided by the Oracle AutoMLx tool, allowing the user to exercise control over the AutoML training process. We then evaluate the different models trained by AutoML. Finally we provide an overview of the possibilities that Oracle AutoMLx offers for explaining the predictions of the tuned model.

Prerequisites¶

• Experience level: Novice (Python and Machine Learning)
• Professional experience: Some industry experience

Business Use¶

Data analytics and modeling problems using Machine Learning (ML) are becoming popular and often rely on data science expertise to build accurate ML models. Such modeling tasks differs according to type of problem to be solved, eg. Text classification primarily involve the following steps:

• Preprocess dataset (vectorize).
• Pick an appropriate model for the given dataset and prediction task at hand.
• Tune the chosen model’s hyperparameters for the given dataset.

All of these steps are significantly time consuming and heavily rely on data scientist expertise. Unfortunately, to make this problem harder, the best feature subset, model, and hyperparameter choice widely varies with the dataset and the prediction task. Hence, there is no one-size-fits-all solution to achieve reasonably good model performance. Using a simple Python API, AutoML can quickly jump-start the datascience process with an accurately-tuned model and appropriate features for a given prediction task.

Table of Contents¶

• Setup
• Load the 20newsgroup Income dataset
• AutoML
  • Set the engine
  • Create an Instance of Oracle AutoMLx
  • Train a Model using AutoMLx
  • Analyze the AutoMLX optimization process
    • Algorithm Selection
    • Adaptive Sampling
    • Feature Selection
    • Hyperparameter Tuning
  • Advanced AutoML Configuration
• Machine Learning Explainability (MLX)
  • Initialize an MLExplainer
  • Global Token Importance
  • Local Token Importance
• References

Setup¶

Basic setup for the Notebook.

Load the required modules.

Load the 20 News Group dataset¶

We start by reading in the dataset from sklearn. The dataset has already been pre-split into training and test sets. The training set will be used to create a Machine Learning model using AutoMLx, and the test set will be used to evaluate the model's performance on unseen data.

Lets look at a few of the values in the data. 20 NewsGroup is a classification dataset made of text samples. Each sample has an associated class (also called topic), which can be one of the followings:

We display some examples of data samples.

We further downsample the train set to have a reasonable training time for this demonstration.

Finally we generate a validation set and only use that for internal pipeline validation.

AutoML¶

Setting the execution engine¶

The AutoML package offers the function init, which allows to initialize the parallelization engine.

Create an instance of Oracle AutoMLx¶

The Oracle AutoMLx solution provides a pipeline that automatically finds a tuned model given a prediction task and a training dataset. In particular it allows to find a tuned model for any supervised prediction task, for example: classification or regression where the target can be binary, categorical or real-valued.

AutoML consists of five main steps:

• Preprocessing (Feature extraction and selection) : The pipeline extracts tabular features using TFIDF that it then selects between.
• Algorithm Selection : Identify the right classification algorithm for a given dataset, choosing from amongst:
  • AdaBoostClassifier
  • CatBoostClassifier
  • DecisionTreeClassifier
  • ExtraTreesClassifier
  • TorchMLPClassifier
  • KNeighborsClassifier
  • LGBMClassifier
  • LinearSVC
  • LogisticRegression
  • RandomForestClassifier
  • SVC
  • XGBClassifier
  • GaussianNB
  • TabNetClassifier
• Adaptive Sampling : Select a subset of the data samples for the model to be trained on.
• Feature Selection : Select a subset of the features (extracted with TFIDF), based on the previously selected model. In the rest of this notebook, we will implicitly refer to the extracted features as data features.
• Hyperparameter Tuning : Find the right model parameters that maximize score for the given dataset.

All these pieces are readily combined into a simple AutoMLx pipeline which automates the entire Machine Learning process with minimal user input/interaction.

Train a model using AutoMLx¶

The AutoMLx API is quite simple to work with. We create an instance of the pipeline. Next, the training data is passed to the fit() function which executes the four previously mentioned steps.

A model is then generated and can be used for prediction tasks. We use the roc_auc scoring metric to evaluate the performance of this model on unseen data (X_test).

Analyze the AutoMLx optimization process¶

During the AutoML process, a summary of the optimization process is logged. It consists of:

• Information about the training data
• Information about the AutoMLx Pipeline, such as:
  • selected features that AutoMLx found to be most predictive in the training data;
  • selected algorithm that was the best choice for this data;
  • hyperparameters for the selected algorithm.

AutoMLx provides a print_summary API to output all the different trials performed.

We also provide the capability to visualize the results of each stage of the AutoMLx pipeline.

Algorithm Selection¶

The plot below shows the scores predicted by Algorithm Selection for each algorithm. The horizontal line shows the average score across all algorithms. Algorithms below the line are colored turquoise, whereas those with a score higher than the mean are colored teal.

Feature Selection¶

After finding a sample subset, the next step is to find a relevant feature subset to maximize score for the chosen algorithm. The feature selection step identifies the smallest feature subset that does not compromise on the score of the chosen algorithm. The orange line shows the optimal number of features chosen by Feature Selection.

Hyperparameter Tuning¶

Hyperparameter Tuning is the last stage of the AutoMLx pipeline, and focuses on improving the chosen algorithm's score on the reduced dataset (after Adaptive Sampling and Feature Selection). We use a novel algorithm to search across many hyperparameters dimensions, and converge automatically when optimal hyperparameters are identified. Each trial in the graph below represents a particular hyperparameters configuration for the selected model.

Advanced AutoML Configuration¶

You can also configure the pipeline with suitable parameters according to your needs.

A few of the advanced settings can be passed directly to the pipeline's fit method, instead of its constructor.

Machine Learning Explainability¶

For a variety of decision-making tasks, getting only a prediction as model output is not sufficient. A user may wish to know why the model outputs that prediction, or which data features are relevant for that prediction. For that purpose the Oracle AutoMLx solution defines the MLExplainer object, which allows to compute a variety of model explanations

Initializing an MLExplainer¶

The MLExplainer object takes as argument the trained model, the training data and labels, as well as the task.

Model Explanations (Global Token Importance)¶

For text classification tasks, since we first extract tokens (features), the Oracle AutoMLx solution offers a single way to compute a notion of token importance: Global Token Importance. The notion of Global Token Importance intuitively measures how much a token impacts the model's predictions (relative to the provided train labels). Tokens are the most fine-grained building blocks of the NLP model, such as sentences, words, or characters.

Computing the importance¶

We use a permutation-based method to successively measure the importance of each token (feature). Such a method therefore runs in linear time with respect to the number of features (tokens) in the dataset.

The method explain_model() allows to compute such feature importances.

Visualization¶

There are two options to show the explanation's results:

• to_dataframe() will return a dataframe of the results.
• show_in_notebook() will show the results as a bar plot.

The features are returned in decreasing order of importance.