Deploying and Configuring the Training Service

Deploying the Training Service

You can deploy the training service either on Oracle Cloud Infrastructure or on a non-OCI environment according to your requirements.

For OCI deployments:

Complete all common installation tasks described in "Overview of Installation Tasks".
When uploading files to Object Storage, also upload the following:
- OCI configuration file
- Private key file
- Preprocessing script (Optional)
Note:

A sample for each of these files is available as part of the artifacts provided with the deployment package.
When setting up data science jobs, create the following five jobs:
- Data Fetch Job
  
  Set JOB_RUN_ENTRYPOINT to entry_file.py
  
  Job artifact: data_fetch_from_os.zip
- Preprocess Job
  
  Set JOB_RUN_ENTRYPOINT to preprocess_entry.py
  
  Job artifact: preprocess.zip
- Train Job
  
  Set JOB_RUN_ENTRYPOINT to train_entry.py
  
  Job artifact: train.zip
- Artifact Job
  
  Set JOB_RUN_ENTRYPOINT to artifact_entry.py
  
  Job artifact: artifact.zip
- Deploy Job
  
  Set JOB_RUN_ENTRYPOINT to deployment_entry.py
  
  Job artifact: deploy.zip
Note:

Oracle provides all of these zip files as part of the artifacts given with the deployment package.

For non-OCI deployments: You only need to install the Helm charts for the training service as described below.

To install the training services using Helm charts:

Download the training utility service Helm charts from the deployment package.

For more information, see "Downloading Packages for the Cloud Native Helm Charts and Docker Files" and "Setting Up Prerequisite Software and Tools".
Create a new key and generate an SSL certificate to run the data service over HTTPS.

(Optional) Install the Ingress NGINX controller:

Create a Kubernetes TLS secret using the certificate you generated and your private key:

kubectl create secret tls secretName --cert=certificateName.crt --key=privateKeyName.key

Add the ingress-nginx Helm repository:
```
helm repo add ingress-nginx
```
Update the Helm repository:
```
helm repo update
```

Install the Ingress Controller:

helm install ingress-nginx ingress-nginx/ingress-nginx --namespace nginxNamespace --create-namespace

Attach the service to the NGINX Controller:

helm install ingress-nginx ingress-nginx/ingress-nginx --namespace nginxNamespace \ 
  --set controller.service.enableHttp=false \
  --set controller.service.enableHttps=true \ 
  --set controller.service.ports.https=443 \ 
  --set controller.service.nodePorts.https=31231 nodePort \ 
  --set controller.config.ssl-redirect=true \ 
  --set controller.config.force-ssl-redirect=true \ 
  --set controller.ingressClassResource.name=ingressClassName \ 
  --set controller.ingressClass=ingressClassName \ 
  --create-namespace

Update the values.yaml file for your deployment:
1. Set mount paths:
  - mountPath.modelFilesHostPath: Specify the path where model files are stored. For a single worker node, specify a path on that node. For multiple worker nodes, specify the NFS or OCI File Storage export path. Set this value even if the models are stored in OCI.
  - mountPath.dataFilesHostPath: Specify the path where data files are stored when storage_type is pvc. For a single worker node, specify a path on that node, such as /mnt/data_service. For multiple worker nodes, specify the NFS or OCI File Storage export path.
  - mountPath.logFilesHostPath: Specify the path where log files are stored. For multiple worker nodes, specify the NFS or OCI File Storage export path. If this value is not set, log files are stored in the directory specified by mountPath.modelFilesHostPath.
  - Set folder permissions:
```
groupadd -g 10001 oracle
useradd -mr -u 10001 -g oracle oracle
chown 10001:10001 -R mountPath
```
2. Set storage configuration:
  - modelFilesStorage.storageType: Set to hostPath for files on a single worker node, nfs for an NFS export shared across worker nodes, or ociFss for an OCI File Storage export shared across worker nodes. The default is nfs.
  - modelFilesStorage.nfs.server: For NFS storage, specify the DNS name or IP address of the NFS server. The server must export the path specified in mountPath.modelFilesHostPath and be reachable from the worker nodes.
  - modelFilesStorage.nfs.existingClaim: When upgrading a deployment that uses an existing model-files PVC, specify its current claim name.
  - logFilesStorage.existingClaim and dataFilesStorage.existingClaim: When upgrading a deployment that uses existing log-files or data-files PVCs, specify their current claim names.
  - modelFilesStorage.ociFss.fileSystemOcid: For OCI File Storage, specify the OCID of the existing file system.
  - modelFilesStorage.ociFss.mountTargetIp: For OCI File Storage, specify the mount target IP address.
3. If you are not using NGINX Ingress, configure HTTPS:
  - https.enabled: Set to true.
  - https.source: Set to hostPath when Kubernetes Secrets are used for certificate ingestion, or to chartFiles to read the certificate and key from chart files.
  - https.orchestratorServicePort: Specify the port on which the service is exposed.
  - https.path: If https.source is hostPath, specify the host system path containing the certificate and key.
  - https.certFile and https.keyFile: If https.source is hostPath, specify the certificate and key file names in that path.
  - https.certPath and https.keyPath: If https.source is chartFiles, specify the certificate and key files in the helm/train-utility/files/certs/ directory.
4. Add imageRepository and image tags for trainUtilityOrchestrator.image and trainUtilityProcessor.image.
5. Under the trainUtilityPredictor.configurableFiles entry, specify the paths within the mount paths to place the following files:
  - preprocessScript: Specify the path to the Python preprocessing script. Place it in helm/train-utility/files/scripts/.
  - datascienceConfig: Specify the path to the configuration file containing the Data Science job details. Place it in helm/train-utility/files/config/.
  - labelFile: Specify the path to the label file used to train a DNN, DNN_MH, or XGB model. Place it in helm/train-utility/files/config/.
  - logConfigFile: Specify the path to the logging configuration file if you need to change the log level or format. Place it in helm/train-utility/files/config/.
  Specify these paths relative to the mount path. If a configurable file is larger than 1 MB, place it under mountPath.modelFilesHostPath and specify its absolute path inside the container.
  
  Note:
  
  Configurable sample files are available for each of the files mentioned above in the artifacts provided with the package: sample_preprocess.py (for preprocessScript), oci_config (for ociConfigFile), config.json (for datascienceConfig), sample_label.csv (for labelFile), and logging_config.py (for logConfigFile). You can configure these files as per your requirements.
6. If your deployment uses OCI services, configure OCI access:
  - ociConfig.enabled: Set to true if the training service uses OCI Data Flow, Data Science, or Object Storage.
  - ociConfig.configPath and ociConfig.privateKeyPath: Specify the relative paths of the OCI configuration file and private key, such as paths under files/secrets/oci/.
7. If you are using NGINX, set the NGINX Ingress values:
  - host: Set to the host name where the service is deployed.
  - tlsSecretName: Set to the name of the Kubernetes TLS secret created in step 3.
  - ingressClassName: Set to your ingress class name.
8. Set identityURI, issuer, and audience to your IDCS configuration. Do one of the following:
  - To use IDCS, set trainUtilityOrchestrator.idcs.enabled to true.
  - To disable IDCS, set trainUtilityOrchestrator.idcs.enabled to false.

Install the Helm charts:

helm upgrade --install training-utility-services helm/train-utility/ --namespace=training-utility-service

Configuring and Using the Training Service

You interact with the training service using REST APIs, which enable integration with external systems.

For training service, you use the following REST service:

Train the Model (/utility/train)

Use this API to train a model based on the data acquired by the data service and the configured parameters.

For more information, see REST API Reference for AI Services.

Training a Multi-Head Offer Recommendation Model

Use a multi-head offer recommendation model when you want to generate multiple related recommendations for the same customer. For example, the model can recommend a charge or product offer and a discount offer.

To train a multi-head offer recommendation model:

Configure the data service to collect the required data. See "Configuring Multiple Data Sources".
Collect and persist the required data fragments.

Ensure that the data fragments required for training are available in the configured storage location.
Create a label file.

The label file must contain two or more columns that use the LABEL_ prefix. Each label column becomes an output head in the model. For example, LABEL_1 can contain charge or product offer labels and LABEL_2 can contain discount offer labels.
Use a preprocessing script to prepare the input data, select features, encode categorical values, or apply business rules before training.
Configure the storage location for the model, supporting artifacts, and metrics.
Configure OCI Data Science or MLflow, depending on your deployment environment.
Configure the training service helm chart.

Specify the label file, preprocessing script, model storage, and logging configuration. For OCI deployments, also configure the required OCI Data Science settings.
Submit a training request that specifies the dnn_mh algorithm.

The training service validates the label columns and input data before training starts.
Verify that the training service created the model and supporting artifacts.

The training service trains a multi-head model, evaluates each output head, and stores the model in ONNX format. Supporting artifacts can include label encoders, feature-column metadata, and metrics files.
Verify that the model is available in the configured model-management system.

The training service publishes the model and metrics to OCI Data Science or MLflow, depending on the deployment configuration.

Note:

Do not configure the single-head DNN algorithm and the multi-head DNN algorithm in the same training request.

Training an Anomaly Detection Model

Use an anomaly detection model to identify unusual behavior in time-series operational metrics. The supplied sample use case uses an XGBoost model.

To train an anomaly detection model:

Configure the data service to collect the required Prometheus metrics. See "Collecting Metrics for Anomaly Detection".
Collect and persist the historical metric data.

Ensure that the metric data required for training is available in the configured storage location.
Create a label file.

The label file must contain a label for each record in the collected metric data. The supplied sample use case uses 0 for normal metrics and 1 for potential outage detection.

You can add labels, such as 2, 3, and 4, when your use case identifies different anomaly types. Define the meaning of each label in the label file.
Use a preprocessing script to prepare the input data, select the required metric features, and associate the metric data with the labels before training.

Ensure that the preprocessing script prepares the features in the format expected by the model. The same features must be available when you request a prediction.
Configure the storage location for the trained model, metadata, supporting artifacts, and training metrics.
Configure OCI Data Science or MLflow, depending on your deployment environment.
Configure the training service helm chart.

Specify the label file, preprocessing script, model storage location, and logging configuration. For OCI deployments, also configure the required OCI Data Science settings.
Submit a training request that specifies the algorithm configured for the model.

The training service validates the input data and configured parameters before training starts.
Verify that the training service created the model and supporting artifacts.

The training service trains the model and stores the metadata required by the prediction service. The metadata identifies the features that the prediction service uses to prepare the metric data.
Verify that the model is available in the configured model-management system.

The training service publishes the model and training metrics to OCI Data Science or MLflow, depending on the deployment configuration.

5 Deploying and Configuring the Training Service

Deploying the Training Service

Configuring and Using the Training Service

Training a Multi-Head Offer Recommendation Model

Training an Anomaly Detection Model