Experiments (Preview)

This chapter provides information on creating, managing, and developing experiments in your workspace.

Experiments in Oracle AI Data Platform Workbench provide means for data scientists, machine-learning engineers, and managers to collaborate in the development of models. AI Data Platform Workbench enables you to:

  • Track experiments for building best models via performance analysis, collaboration, analysis of experimental conditions (hyper-parameters, input dataset, feature engineering etc.)
  • Ensure old experiments aren’t repeated with same experimental conditions
  • Rerun old experiments with different experimental conditions for likely performance gains
  • Reproducibility of earlier experiments
  • Facilitate retraining with newer datasets when performance degrades or retraining is warranted
  • Load a model from Model Registry into notebook and compare model performance versus a new model being developed

Note:

If you haven't used previously used Experiments or Models in your AI Data Platform Workbench, you need to either restart your associated compute cluster or create a new one to use with Experiments and Models.

Limitations

Experiments are currently not supported by ARM-based compute clusters. Ensure the attached compute cluster is either Intel- or AMD-based.

Create an Experiment (Preview)

You can create experiments directly in the Experiments page of Oracle AI Data Platform.

  1. On the home page, click Experiments.
  2. Click Create.
  3. Provide a name and description for your experiment.
  4. Optional: Provide additional metadata in the form of free-form or defined tags. Click Add to create additional tags.
  5. Click Create.

Edit an Experiment (Preview)

You modify details for experiments in your Oracle AI Data Platform Workbench workspace.

  1. On the home page, click Experiments.
  2. Next to the experiment you want to edit, click Edit.
  3. Modify details for your experiment.
  4. Click Save changes.

View Experiment Run Details (Preview)

You can see a history of past experiment runs, compare them, and see the details for specific runs for experiments in your Oracle AI Data Platform Workbench workspace.

  1. On the home page, click Experiments.
  2. Click the name of the experiment you want to view run details for.
  3. Use the drop-down lists and search bar to filter the displayed experiment runs.
  4. Click List or Compare to change the experiment runs are displayed.
    • List displays experiment runs based on your filters in ordered rows with metrics about the experiment run shown in each column. You can sort by metric by clicking that column header.
    • Compare displays bar graph comparisons of key metrics for every experiment run currently displayed by your filters.
  5. Click the name of an experiment run to view details for that experiment run.

Create an Experiment Run in a Notebook with Sample Code (Preview)

You can create runs for an experiment within a notebook by modifying sample code with existing experiment details.

  1. Navigate to your notebook where you want to create a run for an experiment.
  2. Click the Experiments tab.
  3. Click Sample code.
  4. In the sample code block, replace experiment name="Customer Churn Prediction" with experiment name="<your_experiment_name>". You can also copy this code and modify it with your experiment name:
    import mlflow 
experiment_name = "experiment name" #Replace this with your own experiment name 
mlflow.set_experiment(experiment_name) 

mlflow.autolog() 

with mlflow.start_run(): 
    # training code goes here 
    # OPTIONAL - Log additional items after training 
    mlflow.log_metric("custom_metric", 99.9)
  5. Autologs automatically record a default set of metrics, depending on the model selected. To manually specify your own metrics, you can modify this code to invoke mlflow.log_metric(“<metric_name>”,<metric_variable>):
    import mlflow 
import numpy as np 
from sklearn.tree import DecisionTreeRegressor 
from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score 

experiment_name = "Customer Churn Prediction" 
mlflow.set_experiment(experiment_name) 

mlflow.autolog() 

with mlflow.start_run(): 
    # training code goes here 
    model = DecisionTreeRegressor(random_state=42, max_depth=5) 
    model.fit(X_train, y_train) 

    preds = model.predict(X_test) 

    mse = mean_squared_error(y_test, preds) 
    rmse = float(np.sqrt(mse)) 

    mlflow.log_metric("test_rmse", rmse) 
    mlflow.log_metric("test_mae", float(mean_absolute_error(y_test, preds))) 
    mlflow.log_metric("test_r2", float(r2_score(y_test, preds))) 
    # OPTIONAL - Log additional items after training 
    mlflow.log_metric("custom_metric", 99.9)
  6. Run the code block from your notebook. The run is now registered to specified experiment.

    Note:

    Multiple runs for an experiment are automatically logged with different names. For parameter sweep scenarios, AI Data Platform Workbench automatically captures all the runs and specified metrics with different names to the specified experiment.

Register a Model from Experiment Run Details (Preview)

You can register a model in your Oracle AI Data Platform Workbench workspace from a specific experiment run by clicking through to the details of that experiment run.

  1. On the home page, click Experiments.
  2. Click the name of the experiment you want to view run details for.
  3. Click the name of the experiment run you want to register a model from.
  4. Click Register.
  5. Provide a name and description for your model.
  6. Choose the location in your master catalog to create the model.
  7. From the Models drop-down list, select the appropriate model.
  8. Optional: Provide additional metadata in the form of free-form or defined tags. Click Add to create additional tags.
  9. Optional: Provide additional information in the form of custom fields, like model type or use cases. Click Add to create additional custom fields.
  10. Click Register.

Delete Experiment (Preview)

You modify details for experiments in your Oracle AI Data Platform Workbench workspace.

Note:

You can't delete an experiment if there is a registered model based off a run of that experiment.
  1. On the home page, click Experiments.
  2. Next to the experiment you want to delete, click Delete.
  3. Click Delete.