Using Notebook Sessions to Build and Train Models

Authenticating to the OCI APIs from a Notebook Session

When you are working within a notebook session, you are operating as the Linux user datascience. This user does not have an OCI Identity and Access Management (IAM) identity, so it has no access to the OCI API. OCI resources include Data Science projects and models and the resources of other OCI services, such as Object Storage, Functions, Vault, Data Flow, and so on. To access these resources from the notebook environment, use one of the two authentication approaches:

(Recommended) Authenticating Using a Notebook Session's Resource Principal

A resource principal is a feature of IAM that enables resources to be authorized principal actors that can perform actions on service resources. Each resource has its own identity, and it authenticates using the certificates that are added to it. These certificates are automatically created, assigned to resources, and rotated, avoiding the need for you to store credentials in your notebook session.

The Data Science service enables you to authenticate using your notebook session's resource principal to access other OCI resources. Resource principals provides a more secure way to authenticate to resources compared to the OCI configuration and API key approach

Your tenancy administrator must write policies to grant permissions for your resource principal to access other OCI resources, see Configuring Your Tenancy for Data Science.

You can authenticate with resource principals in a notebook session using the following interfaces:

Oracle Accelerated Data Science SDK:

Run the following in a notebook cell:

import ads
ads.set_auth(auth='resource_principal')

For details, see the Accelerated Data Science documentation.

OCI Python SDK:

Run the following in a notebook cell.

import oci
from oci.data_science import DataScienceClient
rps = oci.auth.signers.get_resource_principals_signer()
dsc = DataScienceClient(config={}, signer=rps)

OCI CLI:

Use the `--auth=resource_principal` flag with commands.

Note

The resource principal token is cached for 15 minutes. If you change the policy or the dynamic group, you have to wait for 15 minutes to see the effect of your changes.

Important

If you don't explicitly use the resource principals when invoking an SDK or CLI, then the configuration file and API key approach is used

(Default) Authenticating Using OCI Configuration File and API Keys

You can operate as your own personal IAM user by setting up an OCI configuration file and API keys to access OCI resources. This is the default authentication approach

To authenticate using the configuration file and API key approach, you must upload an OCI configuration file into the notebook session's /home/datascience/.oci/ directory. For the relevant profile defined in the OCI configuration file, you also need to upload or create the required .pem files.

Alternatively, you can use the included individual getting-started.ipynb notebooks to interactively create configuration and key files, see Accessing the Conda Environment Notebook Examples.

You can use the api_keys.ipynb notebook to interactively create OCI configuration and API key files. To launch the api_keys.ipynb notebook, click Notebook Examples in the JupyterLab Launcher tab

Working with Existing Code Files

You can create new files or work with your own existing files.

Uploading Files

Files can be uploaded from your local machine by clicking Upload in the JupyterLab interface or by dragging and dropping files.

Creating a Key Pair in a Notebook Session to Use with a Third-Party Version Control Provider

Cloning a Git Repository Without an Existing Private Key

If you don't have a private key, you can create one in the notebook session by running the ssh-keygen command in the JupyterLab environment.

These instructions use a Git repository as an example though the steps are similar for other repositories. Flows between third-party version control providers and internal Git servers may differ.

Copy your public key into your version control provider. For example, on GitHub.com under your avatar menu, click settings then click SSH and GPC keys, and then click New SSH key.
Go back to your notebook environment.
Add your private key with:
```
ssh-agent bash -c ‘ssh-add <path_to_your_private_key>'
```
Now that you have added your private key, a new identity has been created.
Clone your repository on your notebook session environment. For example, you could clone a repository from GitHub.com.
```
git clone git@github.com:<your_account>/<your_repository>
```
If you are working with multiple version control hosts, we recommend that you edit your ~/.ssh/config file and add the information about your Git host and the location of the private key for that particular host:
```
Host <your_host.com> IdentityFile <path_to_your_private_key>
```

Using Additional Terminal Commands

Installing Additional Python Libraries

You can install a library that's not preinstalled in the provided image.

Access to the public internet is required to install additional libraries. Install a library by opening a notebook session and running this command:

%%bash
pip install <library-name>==<library-version>

Important

Data Science doesn't allow root privileges in notebook sessions. You can only install libraries using yum and pip as a normal user. Attempting to use sudu results in errors.

You can install any open source package available on a publicly-accessible Python Package Index (PyPI) repository. You can also install private or custom libraries from your own internal repositories.

Note

The VCN or subnet that you used to create the notebook session must have network access to the source locations for the packages you want to download and install, see Manually Configuring Your Tenancy for Data Science.

Using the Provided Environment Variables in Notebook Sessions

When you start up a notebook session, the service creates useful environment variables that you can use in your code:

NB_SESSION_COMPARTMENT_OCID: The compartment OCID of the current notebook session.
NB_SESSION_OCID: The OCID of the current notebook session.
PROJECT_OCID: The OCID of the project associated with the current notebook session.
USER_OCID: Your user OCID.
PROJECT_COMPARTMENT_OCID: The compartment OCID of the project associated with the current notebook session.

To access these environment variables in your notebook session, use the Python os library. For example:

import os 
project_ocid = os.environ[‘PROJECT_OCID’]
print(project_ocid)

Note

The NB_SESSION_COMPARTMENT_OCID and PROJECT_COMPARTMENT_OCID values do not update in a running notebook session if the resources has moved compartments after the notebook session was created.

Using the Oracle Accelerated Data Science SDK

The Oracle Accelerated Data Science (ADS) SDK is a Python library that is included as part of the OCI Data Science service notebook session resource. ADS offers a friendly user interface that covers many of the steps involved in the lifecycle of machine learning models, from connecting to different data sources to using AutoML for model training to model evaluation and explanation. ADS also provides a simple interface to access the OCI Data Science service model catalog and other OCI services including object storage.

Note

For complete documentation on how to use the Accelerated Data Science SDK, see Accelerated Data Science Library and Accessing the Conda Environment Notebook Examples.

Connecting to Your Data

You can connect to your data in these ways:

Connecting to Data in Oracle Cloud Infrastructure Object Storage

Connecting to Data on the Autonomous Data Warehouse

You can connect to the Autonomous Data Warehouse (ADW) from your notebook session. The autonomous_database.ipynb example notebook interactively illustrates this type of connection.

Note

The VCN and subnet configuration that you selected when creating your notebook session should permit access to your ADW database. Contact your IT administrator to confirm that access with the networking configuration you selected is permitted.

To connect to ADW and pull data into a dataframe in your notebook session:

Go to the OCI Console, and access your ADW instance page.
Click DB Connection to download the wallet file.
Create your own password for the download action.
Create a folder for your wallet files on the notebook environment. We recommend that you create that folder in /home/datascience.
Upload your wallet files into your <path_to_wallet_folder> folder in JupyterLab by clicking Upload.
Open the sqlnet.ora file from the wallet files, then configure the METHOD_DATA to be:
```
METHOD_DATA = (DIRECTORY="<path_to_wallet_folder>")
```
Create a new notebook file, and set the following environment variables in a notebook cell:
```
%env TNS_ADMIN=<path_to_wallet_folder>
```
You can find SID names from the tnsname.ora file from the wallet file, and then set ADW_SID as an environment variable (the SID is an identifier that identifies the unique name of the Oracle Database):
```
%env ADW_SID=<your_SID_name>
```
Retrieve your ADW user name and password from your database administrator, and then store your credentials as environment variables in your notebook session:
```
%env ADW_USER=<your_username>
%env ADW_PASSWORD=<your_password>
```

In a separate notebook cell, run this command to test the connection to the database:

!sqlplus $ADW_USER/$ADW_PASSWORD@$ADW_SID

If it's successful, you should see messages similar to:

SQL*PLUS Release 19.0.0.0.0 - Production on Tue Dec 17 16:14:32 2019
 
Copyright (c) 1982, 2019, Oracle. All rights reserved.
 
Last Successful login time: Mon De 16 2019 14:19:21 -08:00
 
Connected to:
Oracle Database 18c Enterprise Edition Release 18.0.0.0.0 - Production
Version 18.4.0.0.0

SQL>

Define a URI as your connection source:

uri=f'oracle+cx_oracle://{os.environ["ADW_USER"]}:
{os.environ["ADW_PASSWORD"]}@{os.environ["ADW_SID"]}'

In a different cell, import sqlalchemy and pandas, and then create an engine to connect to your database. Define your SQL query and pass it to the read_sql() method that is part of the Pandas Python library:
```
from sqlalchemy import create_engine 
import pandas as pd 
engine = create_engine(uri) 
ds_01 = pd.read_sql('SELECT * from <table_name>', con=engine)
```
The returned object is a Pandas dataframe.

You can access notebook examples within notebook sessions that show you the different steps involved in connecting and querying data from ADW and other data sources.

Connecting to Data on OCI Streaming

Connecting to Data Using Oracle Vault

Oracle Cloud Infrastructure Documentation