by the Oracle AutoMLx Team
\n", "\n", "***" ] }, { "cell_type": "markdown", "id": "57f069e3", "metadata": {}, "source": [ "Regression Demo Notebook.\n", "\n", "Copyright © 2025, Oracle and/or its affiliates.\n", "\n", "Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl/" ] }, { "cell_type": "markdown", "id": "a8ff7a02", "metadata": {}, "source": [ "## Overview of this Notebook\n", "\n", "In this notebook we will build a regressor using the Oracle AutoMLx tool for the public California Housing dataset to predict the value of house prices.\n", "We explore the various options provided by the Oracle AutoMLx tool, allowing the user to control the AutoMLx training process. We finally evaluate the different models trained by AutoMLx. Depending on the dataset size and the machine running it, it can take about tens of minutes. The dataset is sampled down for a snappier demo, with the option to run it with the full dataset. We finally provide an overview of the possibilities that Oracle AutoMLx provides for explaining the predictions of the tuned model.\n", "\n", "---\n", "## Prerequisites:\n", "\n", " - Experience level: Novice (Python and Machine Learning)\n", " - Professional experience: Some industry experience\n", "---\n", "\n", "## Business Use:\n", "\n", "Data analytics and modeling problems using Machine Learning (ML) are becoming popular and often rely on data science expertise to build accurate ML models. Such modeling tasks primarily involve the following steps:\n", "- Preprocess dataset (clean, impute, engineer features, normalize).\n", "- Pick an appropriate model for the given dataset and prediction task at hand.\n", "- Tune the chosen model’s hyperparameters for the given dataset.\n", "\n", "All of these steps are significantly time consuming and heavily rely on data scientist expertise. Unfortunately, to make this problem harder, the best feature subset, model, and hyperparameter choice widely varies with the dataset and the prediction task. Hence, there is no one-size-fits-all solution to achieve reasonably good model performance. Using a simple Python API, AutoML can quickly jump-start the datascience process with an accurately-tuned model and appropriate features for a given prediction task.\n", "\n", "## Table of Contents\n", "\n", "- Setup\n", "- Load California housing dataset\n", "- AutoML\n", " - Setting the execution engine\n", " - Create an Instance of AutoMLx\n", " - Train a Model using AutoMLx\n", " - Analyze the AutoMLx optimization process \n", " - Algorithm Selection\n", " - Adaptive Sampling\n", " - Feature Selection\n", " - Hyperparameter Tuning\n", " - Advanced AutoMLx Configuration \n", " - Use a custom validation set\n", "- Machine Learning Explainability (MLX)\n", " - Initialize an MLExplainer\n", " - Model Explanations (Global Feature Importance)\n", " - Feature Dependence Explanations (PDP + ICE)\n", " - Prediction Explanations (Local Feature Importance)\n", " - Aggregate Local Feature Importance\n", " - Interactive What-IF Explanations\n", " - Counterfactuals Explanations\n", " - Advanced Feature Importance Options\n", " - Configure prediction explanation\n", " - Explain the model or explain the world\n", " - Advanced Feature Dependence Options (ALE)\n", "- References" ] }, { "cell_type": "markdown", "id": "30938eba", "metadata": {}, "source": [ "\n", "## Setup\n", "\n", "Basic setup for the Notebook." ] }, { "cell_type": "code", "execution_count": 1, "id": "6dd055c5", "metadata": { "execution": { "iopub.execute_input": "2025-05-22T12:35:22.798339Z", "iopub.status.busy": "2025-05-22T12:35:22.798146Z", "iopub.status.idle": "2025-05-22T12:35:23.331028Z", "shell.execute_reply": "2025-05-22T12:35:23.330367Z" } }, "outputs": [], "source": [ "\n", "%matplotlib inline\n", "%load_ext autoreload\n", "%autoreload 2" ] }, { "cell_type": "markdown", "id": "49707089", "metadata": {}, "source": [ "Load the required modules." ] }, { "cell_type": "code", "execution_count": 2, "id": "9252070e", "metadata": { "execution": { "iopub.execute_input": "2025-05-22T12:35:23.333979Z", "iopub.status.busy": "2025-05-22T12:35:23.333299Z", "iopub.status.idle": "2025-05-22T12:35:25.862277Z", "shell.execute_reply": "2025-05-22T12:35:25.861563Z" } }, "outputs": [], "source": [ "import time\n", "import datetime\n", "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "import plotly.figure_factory as ff\n", "from sklearn.datasets import fetch_california_housing\n", "from sklearn.linear_model import LinearRegression\n", "from sklearn.metrics import mean_squared_error\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.linear_model import LinearRegression\n", "\n", "# Settings for plots\n", "plt.rcParams['figure.figsize'] = [10, 7]\n", "plt.rcParams['font.size'] = 15\n", "\n", "import automlx\n", "from automlx import init" ] }, { "cell_type": "markdown", "id": "4f322166", "metadata": {}, "source": [ "\n", "## Load the California housing dataset using sklearn.datasets\n", "\n", "Dataset details are available here: https://scikit-learn.org/stable/datasets/real_world.html#california-housing-dataset. The goal is to predict the median price of a house given some features." ] }, { "cell_type": "code", "execution_count": 3, "id": "f4b6c4a1", "metadata": { "execution": { "iopub.execute_input": "2025-05-22T12:35:25.865282Z", "iopub.status.busy": "2025-05-22T12:35:25.864423Z", "iopub.status.idle": "2025-05-22T12:35:25.914939Z", "shell.execute_reply": "2025-05-22T12:35:25.914392Z" } }, "outputs": [ { "data": { "text/plain": [ "(20640, 9)" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "X, y = fetch_california_housing(return_X_y=True)\n", "ds = fetch_california_housing(return_X_y=False)\n", "df = pd.concat([pd.DataFrame(X, columns=ds.feature_names),\n", " pd.DataFrame(y.ravel(), columns=['Median Price'])], axis=1)\n", "\n", "target_col='Median Price'\n", "df.shape" ] }, { "cell_type": "code", "execution_count": 4, "id": "ea312805", "metadata": { "execution": { "iopub.execute_input": "2025-05-22T12:35:25.916837Z", "iopub.status.busy": "2025-05-22T12:35:25.916402Z", "iopub.status.idle": "2025-05-22T12:35:25.949822Z", "shell.execute_reply": "2025-05-22T12:35:25.949225Z" } }, "outputs": [ { "data": { "text/html": [ "| \n", " | MedInc | \n", "HouseAge | \n", "AveRooms | \n", "AveBedrms | \n", "Population | \n", "AveOccup | \n", "Latitude | \n", "Longitude | \n", "Median Price | \n", "
|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "8.3252 | \n", "41.0 | \n", "6.984127 | \n", "1.023810 | \n", "322.0 | \n", "2.555556 | \n", "37.88 | \n", "-122.23 | \n", "4.526 | \n", "
| 1 | \n", "8.3014 | \n", "21.0 | \n", "6.238137 | \n", "0.971880 | \n", "2401.0 | \n", "2.109842 | \n", "37.86 | \n", "-122.22 | \n", "3.585 | \n", "
| 2 | \n", "7.2574 | \n", "52.0 | \n", "8.288136 | \n", "1.073446 | \n", "496.0 | \n", "2.802260 | \n", "37.85 | \n", "-122.24 | \n", "3.521 | \n", "
| 3 | \n", "5.6431 | \n", "52.0 | \n", "5.817352 | \n", "1.073059 | \n", "558.0 | \n", "2.547945 | \n", "37.85 | \n", "-122.25 | \n", "3.413 | \n", "
| 4 | \n", "3.8462 | \n", "52.0 | \n", "6.281853 | \n", "1.081081 | \n", "565.0 | \n", "2.181467 | \n", "37.85 | \n", "-122.25 | \n", "3.422 | \n", "
| \n", " |
|---|
| (14448, 8) | \n", "
| None | \n", "
| RepeatedKFoldSplit(Shuffle=True, Seed=7, number of splits=5, number of repeats=2) | \n", "
| neg_mean_squared_error | \n", "
| LGBMRegressor | \n", "
| {'num_leaves': 31, 'boosting_type': 'gbdt', 'subsample': 1, 'colsample_bytree': 0.7952797110155084, 'max_depth': 63, 'reg_alpha': 0, 'reg_lambda': 0, 'n_estimators': 377, 'learning_rate': 0.1, 'min_child_weight': 0.001} | \n", "
| 24.4.1 | \n", "
| 3.9.21 (main, Dec 11 2024, 16:24:11) \\n[GCC 11.2.0] | \n", "
| Step | \n", "# Samples | \n", "# Features | \n", "Algorithm | \n", "Hyperparameters | \n", "Score (neg_mean_squared_error) | \n", "Runtime (Seconds) | \n", "Memory Usage (GB) | \n", "Finished | \n", "
|---|---|---|---|---|---|---|---|---|
| Model Selection | \n", "11559 | \n", "8 | \n", "LGBMRegressor | \n", "{'num_leaves': 31, 'boosting_type': 'gbdt', 'subsample': 1, 'colsample_bytree': 1, 'max_depth': 63, 'reg_alpha': 0, 'reg_lambda': 0, 'n_estimators': 100, 'learning_rate': 0.1, 'min_child_weight': 0.001} | \n", "-0.2183 | \n", "5.3076 | \n", "0.3075 | \n", "Thu May 22 05:35:44 2025 | \n", "
| Model Selection | \n", "11559 | \n", "8 | \n", "XGBRegressor | \n", "{'n_estimators': 100, 'min_child_weight': 1, 'reg_alpha': 0, 'booster': 'gbtree', 'max_depth': 6, 'learning_rate': 0.1, 'reg_lambda': 1} | \n", "-0.2283 | \n", "16.4569 | \n", "0.3459 | \n", "Thu May 22 05:35:56 2025 | \n", "
| Model Selection | \n", "11559 | \n", "8 | \n", "RandomForestRegressor | \n", "{'n_estimators': 100, 'min_samples_split': 0.0003, 'min_samples_leaf': 0.00015, 'max_features': 0.777777778} | \n", "-0.2585 | \n", "34.3275 | \n", "0.3597 | \n", "Thu May 22 05:35:48 2025 | \n", "
| Model Selection | \n", "11559 | \n", "8 | \n", "ExtraTreesRegressor | \n", "{'n_estimators': 100, 'min_samples_split': 0.00125, 'min_samples_leaf': 0.000625, 'max_features': 0.777777778} | \n", "-0.2983 | \n", "6.5971 | \n", "0.2965 | \n", "Thu May 22 05:35:43 2025 | \n", "
| Model Selection | \n", "11559 | \n", "8 | \n", "DecisionTreeRegressor | \n", "{'min_samples_split': 0.004, 'min_samples_leaf': 0.002, 'max_features': 1.0} | \n", "-0.3877 | \n", "3.4246 | \n", "0.2808 | \n", "Thu May 22 05:35:42 2025 | \n", "
| Model Selection | \n", "11559 | \n", "8 | \n", "LinearRegression | \n", "{} | \n", "-0.5296 | \n", "0.7690 | \n", "0.3102 | \n", "Thu May 22 05:35:43 2025 | \n", "
| Model Selection | \n", "11559 | \n", "8 | \n", "AdaBoostRegressor | \n", "{'learning_rate': 0.667, 'n_estimators': 50} | \n", "-0.701 | \n", "13.9089 | \n", "0.2777 | \n", "Thu May 22 05:35:42 2025 | \n", "
| Model Selection | \n", "11559 | \n", "8 | \n", "LinearSVR | \n", "{'C': 1.0} | \n", "-1.6878 | \n", "6.5363 | \n", "0.3188 | \n", "Thu May 22 05:35:44 2025 | \n", "
| Model Selection | \n", "11558 | \n", "8 | \n", "TorchMLPRegressor | \n", "{'optimizer_class': 'Adam', 'shuffle_dataset_each_epoch': True, 'optimizer_params': {}, 'criterion_class': None, 'criterion_params': {}, 'scheduler_class': None, 'scheduler_params': {}, 'batch_size': 128, 'lr': 0.001, 'epochs': 18, 'input_transform': 'auto', 'tensorboard_dir': None, 'use_tqdm': None, 'prediction_batch_size': 128, 'prediction_input_transform': 'auto', 'shuffling_buffer_size': None, 'depth': 4, 'num_logits': 1000, 'div_factor': 2, 'activation': 'ReLU', 'dropout': 0.1} | \n", "-3.1322 | \n", "256.0426 | \n", "0.6121 | \n", "Thu May 22 05:36:14 2025 | \n", "
| Adaptive Sampling | \n", "11559 | \n", "8 | \n", "AdaptiveSamplingStage_LGBMRegressor | \n", "{'num_leaves': 31, 'boosting_type': 'gbdt', 'subsample': 1, 'colsample_bytree': 1, 'max_depth': 63, 'reg_alpha': 0, 'reg_lambda': 0, 'n_estimators': 100, 'learning_rate': 0.1, 'min_child_weight': 0.001} | \n", "-0.2183 | \n", "2.8107 | \n", "0.6115 | \n", "Thu May 22 05:36:17 2025 | \n", "
| ... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
| Model Tuning | \n", "11559 | \n", "6 | \n", "LGBMRegressor | \n", "{'num_leaves': 7, 'boosting_type': 'gbdt', 'subsample': 0.4, 'colsample_bytree': 0.4, 'max_depth': 2, 'reg_alpha': 1e-10, 'reg_lambda': 9.999999990000003e-07, 'n_estimators': 5, 'learning_rate': 0.1, 'min_child_weight': 0.001} | \n", "-1.1075 | \n", "0.7727 | \n", "0.6235 | \n", "Thu May 22 05:36:45 2025 | \n", "
| Model Tuning | \n", "11559 | \n", "6 | \n", "LGBMRegressor | \n", "{'num_leaves': 7, 'boosting_type': 'gbdt', 'subsample': 0.4, 'colsample_bytree': 0.4, 'max_depth': 2, 'reg_alpha': 9.999999990000003e-07, 'reg_lambda': 1e-10, 'n_estimators': 5, 'learning_rate': 0.1, 'min_child_weight': 0.001} | \n", "-1.1075 | \n", "0.7874 | \n", "0.6233 | \n", "Thu May 22 05:36:44 2025 | \n", "
| Model Tuning | \n", "11559 | \n", "6 | \n", "LGBMRegressor | \n", "{'num_leaves': 7, 'boosting_type': 'gbdt', 'subsample': 0.4, 'colsample_bytree': 0.4, 'max_depth': 2, 'reg_alpha': 1e-10, 'reg_lambda': 1.7784127779939314e-05, 'n_estimators': 5, 'learning_rate': 0.1, 'min_child_weight': 0.001} | \n", "-1.1075 | \n", "0.7908 | \n", "0.6235 | \n", "Thu May 22 05:36:46 2025 | \n", "
| Model Tuning | \n", "11559 | \n", "6 | \n", "LGBMRegressor | \n", "{'num_leaves': 7, 'boosting_type': 'gbdt', 'subsample': 0.4, 'colsample_bytree': 0.4, 'max_depth': 2, 'reg_alpha': 1e-10, 'reg_lambda': 1.8784127778939314e-05, 'n_estimators': 5, 'learning_rate': 0.1, 'min_child_weight': 0.001} | \n", "-1.1075 | \n", "0.8087 | \n", "0.6235 | \n", "Thu May 22 05:36:46 2025 | \n", "
| Model Tuning | \n", "11559 | \n", "6 | \n", "LGBMRegressor | \n", "{'num_leaves': 7, 'boosting_type': 'gbdt', 'subsample': 0.4, 'colsample_bytree': 0.4, 'max_depth': 2, 'reg_alpha': 1.7784127779939314e-05, 'reg_lambda': 1e-10, 'n_estimators': 5, 'learning_rate': 0.1, 'min_child_weight': 0.001} | \n", "-1.1075 | \n", "0.8432 | \n", "0.6233 | \n", "Thu May 22 05:36:44 2025 | \n", "
| Model Tuning | \n", "11559 | \n", "6 | \n", "LGBMRegressor | \n", "{'num_leaves': 7, 'boosting_type': 'gbdt', 'subsample': 0.4, 'colsample_bytree': 0.4, 'max_depth': 2, 'reg_alpha': 1.8784127778939314e-05, 'reg_lambda': 1e-10, 'n_estimators': 5, 'learning_rate': 0.1, 'min_child_weight': 0.001} | \n", "-1.1075 | \n", "0.7551 | \n", "0.6233 | \n", "Thu May 22 05:36:45 2025 | \n", "
| Model Tuning | \n", "11559 | \n", "6 | \n", "LGBMRegressor | \n", "{'num_leaves': 7, 'boosting_type': 'gbdt', 'subsample': 0.4, 'colsample_bytree': 0.4, 'max_depth': 2, 'reg_alpha': 1e-10, 'reg_lambda': 0.005623553830557401, 'n_estimators': 5, 'learning_rate': 0.1, 'min_child_weight': 0.001} | \n", "-1.1075 | \n", "0.6737 | \n", "0.6156 | \n", "Thu May 22 05:36:46 2025 | \n", "
| Model Tuning | \n", "11559 | \n", "6 | \n", "LGBMRegressor | \n", "{'num_leaves': 7, 'boosting_type': 'gbdt', 'subsample': 0.4, 'colsample_bytree': 0.4, 'max_depth': 2, 'reg_alpha': 1e-10, 'reg_lambda': 0.0056245538305564015, 'n_estimators': 5, 'learning_rate': 0.1, 'min_child_weight': 0.001} | \n", "-1.1075 | \n", "0.9709 | \n", "0.6156 | \n", "Thu May 22 05:36:46 2025 | \n", "
| Model Tuning | \n", "11559 | \n", "6 | \n", "LGBMRegressor | \n", "{'num_leaves': 7, 'boosting_type': 'gbdt', 'subsample': 0.4, 'colsample_bytree': 0.4, 'max_depth': 2, 'reg_alpha': 0.005623553830557401, 'reg_lambda': 1e-10, 'n_estimators': 5, 'learning_rate': 0.1, 'min_child_weight': 0.001} | \n", "-1.1075 | \n", "0.8067 | \n", "0.6233 | \n", "Thu May 22 05:36:45 2025 | \n", "
| Model Tuning | \n", "11559 | \n", "6 | \n", "LGBMRegressor | \n", "{'num_leaves': 7, 'boosting_type': 'gbdt', 'subsample': 0.4, 'colsample_bytree': 0.4, 'max_depth': 2, 'reg_alpha': 0.0056245538305564015, 'reg_lambda': 1e-10, 'n_estimators': 5, 'learning_rate': 0.1, 'min_child_weight': 0.001} | \n", "-1.1075 | \n", "0.7816 | \n", "0.6233 | \n", "Thu May 22 05:36:45 2025 | \n", "
| \n", " | feature | \n", "attribution | \n", "upper_bound | \n", "lower_bound | \n", "
|---|---|---|---|---|
| 0 | \n", "Latitude | \n", "1.262288 | \n", "1.322257 | \n", "1.202320 | \n", "
| 1 | \n", "Longitude | \n", "1.073291 | \n", "1.126496 | \n", "1.020086 | \n", "
| 2 | \n", "MedInc | \n", "0.369554 | \n", "0.377468 | \n", "0.361640 | \n", "
| 3 | \n", "AveOccup | \n", "0.154384 | \n", "0.164288 | \n", "0.144480 | \n", "
| 4 | \n", "AveRooms | \n", "0.116088 | \n", "0.122427 | \n", "0.109748 | \n", "
| 5 | \n", "HouseAge | \n", "0.063371 | \n", "0.065839 | \n", "0.060902 | \n", "
| \n | Prediction (True value: 3.6) | \n
|---|---|
| Original Sample | \n2.8988 | \n
| Modified Sample | \n2.8988 | \n
| \n | MedInc | \nHouseAge | \nAveRooms | \nAveBedrms | \nPopulation | \nAveOccup | \nLatitude | \nLongitude | \n
|---|---|---|---|---|---|---|---|---|
| Original Sample | \n2.0278 | \n31.0 | \n2.8469284994964754 | \n1.0916414904330312 | \n3107.0 | \n3.1289023162134946 | \n34.06 | \n-118.31 | \n
| Modified Sample | \n2.0278 | \n31.0 | \n2.8469284994964754 | \n1.0916414904330312 | \n3107.0 | \n3.1289023162134946 | \n34.06 | \n-118.31 | \n
| \n | MedInc | \nHouseAge | \nAveRooms | \nAveBedrms | \nPopulation | \nAveOccup | \nLatitude | \nLongitude | \n
|---|---|---|---|---|---|---|---|---|
| 0 | \n3.1906 | \n11.0 | \n5.877667 | \n1.099573 | \n2012.0 | \n2.862020 | \n33.22 | \n-116.90 | \n
| 1 | \n4.0727 | \n18.0 | \n3.957845 | \n1.079625 | \n2276.0 | \n2.665105 | \n33.90 | \n-118.36 | \n
| 2 | \n7.1754 | \n52.0 | \n7.429487 | \n1.051282 | \n779.0 | \n2.496795 | \n37.90 | \n-122.28 | \n
| 3 | \n4.9562 | \n33.0 | \n4.691196 | \n1.019711 | \n1922.0 | \n2.525624 | \n37.34 | \n-121.98 | \n
| 4 | \n7.7261 | \n32.0 | \n7.068311 | \n1.079696 | \n1304.0 | \n2.474383 | \n34.47 | \n-119.70 | \n
| 5 | \n3.3929 | \n39.0 | \n4.274011 | \n1.031073 | \n1227.0 | \n3.466102 | \n33.79 | \n-118.27 | \n
| 6 | \n1.3897 | \n38.0 | \n3.552239 | \n1.074627 | \n191.0 | \n2.850746 | \n36.61 | \n-121.85 | \n
| 7 | \n1.7778 | \n38.0 | \n3.379249 | \n1.010733 | \n2087.0 | \n3.733453 | \n33.95 | \n-118.21 | \n
| 8 | \n3.1528 | \n23.0 | \n5.652985 | \n1.086754 | \n2920.0 | \n2.723881 | \n34.88 | \n-116.99 | \n
| 9 | \n3.6369 | \n10.0 | \n5.646565 | \n1.045038 | \n4611.0 | \n3.519847 | \n35.30 | \n-119.02 | \n