Deploying and Configuring the Data Service

Deploying the Data Service

You can deploy the data service in an OCI environment or in a non-OCI environment.

Before deploying and configuring the data service, ensure that you have all the prerequisites in place.
- For OCI deployments, you do this by performing the instructions in "Overview of Installation Tasks".
- For non-OCI deployments, set up your environment according to your product instructions.
Then, for both OCI and non-OCI environments, deploy the data service using Helm charts as described below.

To deploy the data service using Helm charts:

Download the data service Helm charts from the provided deployment package.

For more information, see "Downloading Packages for the Cloud Native Helm Charts and Docker Files" and "Setting Up Prerequisite Software and Tools".
Create a new key and generate an SSL certificate to run the data service over HTTPS.

(Optional) Install the NGINX Ingress Controller:

Create a Kubernetes TLS secret using the certificate you generated and your private key:

kubectl create secret tls secretName --cert=certificateName.crt --key=privateKeyName.key

Add the ingress-nginx Helm repository:
```
helm repo add ingress-nginx
```
Update the Helm repository:
```
helm repo update
```

Install the Ingress Controller:

helm install ingress-nginx ingress-nginx/ingress-nginx --namespace nginxNamespace --create-namespace

Use the following command to attach your service to the NGINX controller:

helm install ingress-nginx ingress-nginx/ingress-nginx --namespace nginxNamespace \
   --set controller.service.enableHttp=false \ 
   --set controller.service.enableHttps=true \ 
   --set controller.service.ports.https=443 \
   --set controller.service.nodePorts.https=31231 nodePort \
   --set controller.config.ssl-redirect=true \
   --set controller.config.force-ssl-redirect=true \
   --set controller.ingressClassResource.name=ingressClassName \
   --set controller.ingressClass=ingressClassName \
   --create-namespace

Update the values.yaml file for your deployment:
1. Set mount paths:
  - mountPath.dataFilesHostPath: Specify the path where data files are stored. This field is required and is identified as the default log-file location. For a single worker node, specify a path on that node, such as /mnt/data_service. For multiple worker nodes, specify the NFS or OCI File Storage export path. When tech_choice is spark and storage_type is pvc, data files are stored under this directory.
  - mountPath.logFilesHostPath: Specify a different path if log files are to be stored separately. For multiple worker nodes, specify a path in the shared export.
  - Set folder permissions:
```
groupadd -g 10001 oracle 
useradd -mr -u 10001 -g oracle oracle 
chown 10001:10001 -R mountPath
```
2. Set storage configuration:
  - modelFilesStorage.storageType: Set to hostPath for files on a single worker node, nfs for an NFS export shared across worker nodes, or ociFss for an OCI File Storage export shared across worker nodes. The default is nfs.
  - modelFilesStorage.nfs.server: For NFS storage, specify the DNS name or IP address of the NFS server. Ensure that the export is reachable from the worker nodes.
  - modelFilesStorage.nfs.existingClaim: When upgrading a deployment that uses an existing PVC, specify its current claim name.
  - logFileStorage.existingClaim: When upgrading a deployment that uses a separate existing log-file PVC, specify its current claim name.
  - modelFilesStorage.ociFss.fileSystemOcid: For OCI File Storage, specify the file system OCID.
  - modelFilesStorage.ociFss.mountTargetIp: For OCI File Storage, specify the mount target IP address.
3. If you are not using NGINX Ingress, configure HTTPS:
  - https.enabled: Set to true.
  - https.source: Set to hostPath when Kubernetes secrets are used for certificate ingestion, or to chartFiles to read the certificate and key from chart files.
  - https.orchestratorServicePort: Specify the port on which the service is exposed.
  - https.path: If https.source is hostPath, specify the host path containing the certificate and key.
  - https.certFile and https.keyFile: If https.source is hostPath, specify the certificate and key file names.
  - https.certPath and https.keyPath: If https.source is chartFiles, specify the certificate and key files in the helm/data-fetch/files/certs/ directory.
4. Add imageRepository and image tags for dataFetchOrchestrator and dataFetchProcessor.
5. Under the dataFetchProcessor.configurableFile entry, specify the following files:
  - filterFile: Specify the path to the Python script used to filter data for a /fetch request. Place it in helm/data-fetch/files/scripts/ and specify its file name. The script can import additional files as required.
  - dataSourceFile: Specify the path to the configuration file containing data-source and OCI connection information. Place it in helm/data-fetch/files/config/.
  - logConfigFile: Specify the path to the configuration file used to change the log level or format. Place it in helm/data-fetch/files/config/
  Specify these paths relative to the chart’s mount path. If a configurable file is larger than 1 MB, place it under mountPath.dataFilesHostPath and specify its absolute path inside the container.
  
  Note:
  
  Configurable sample files are available for each of the files mentioned above in the artifacts provided with the package: sample_filter_data.py (for filterFile), dnn_preprocess.py (for additional files), config.json (for dataSourceFile), oci_config (for ociConfigFile), and logging_config.py (for logConfigFile).
6. If your deployment uses OCI services, configure OCI access:
  - ociConfig.enabled: Set to true if the data service uses OCI Data Flow, Data Science, or Object Storage.
  - ociConfig.configPath and ociConfig.privateKeyPath: Specify the relative paths of the OCI configuration file and private key, such as paths under files/secrets/oci/.
7. If you are using NGINX, set the NGINX Ingress values:
  - nginx.enabled: Set to true.
  - host: Set to the host name where the service is deployed.
  - tlsSecretName: Set to the name of the Kubernetes TLS secret created in step 3.
  - ingressClassName: Set to your ingress class name.
8. Set identityURI, issuer, and audience to your IDCS configuration. Do one of the following:
  - To use IDCS, set dataFetchOrchestrator.idcs.enabled to true.
  - To disable IDCS, set dataFetchOrchestrator.idcs.enabled to false.

Install the Data Service Helm charts:

helm upgrade --install data-fetch-services helm/data-fetch/ --namespace=data-fetch-service

You can schedule the data fetching process at regular intervals. To do this, configure a cron job to run a suitable shell script.

A sample script, sample_bash.sh, is provided with the package. This script sets delta_data to false for the first run and to true for all subsequent runs, so that after the first fetch, only changed data is fetched.

Before using the script, set it to be executable.

Configuring and Using the Data Service

You interact with the data service using REST APIs and can connect other systems through API calls.

For the data service, use the following endpoints to provide data to downstream machine learning and analytical tasks:

Collect Data from Source (/data/collect)

Collects data from specified sources, such as Billing and Revenue Management databases or your custom databases.
Fetch Data from Cache (/data/fetch)

Retrieves collected data based on request parameters and delivers it to required clients.
Cache Persisted Data (/data/cache)

Caches fetched data for fast, repeated access.

These APIs are designed to collect, cache, and make data available for downstream machine learning and analytical tasks.

For more information, see REST API Reference for AI Services.

Configuring Multiple Data Sources

You can configure the data service to collect data from multiple sources in one request. For example, you can connect to BRM, Siebel, and Prometheus as data sources and use the collected data together to generate a next best offer recommendation.

To configure multiple data sources:

Identify the required data and respective data sources to configure.

You can configure BRM to provide account, usage, product, service, launch, and related billing data. Similarly, you can configure Siebel to provide customer, purchase, campaign, offer, and label data.
Configure BRM and Siebel as separate data sources in the data service configuration file.
Specify the source type, query configuration, and connection details:
- In the input JSON request, add the data sources to the data_source_type list.
- In the input JSON request, add the query configuration for each data source to the query_builder list.
- In the configuration file, add the connection details for each data source to the data_source_details list.
Ensure that the entries in the data_source_type, query_builder, and data_source_details lists use the same order. For example, the connection details in the first data_source_details entry apply to the first source type and first query-builder entry.
Use the collect API to collect and prepare the data.

The data service processes the data sources and persists refined data fragments for downstream processing.
Use the cache API to cache the data required by the prediction service.

The training service uses the persisted data fragments to train models.
Use the Fetch API to retrieve the data required for a customer.

The prediction service uses the retrieved data to generate predictions.

Note:

If a data source is unavailable, the data service returns an error. Verify the source connection details, credentials, query configuration, and source order before retrying the request.

Collecting Metrics for Anomaly Detection

You can configure the data service to collect historical metrics from Prometheus for training a time-series anomaly detection model. The data service persists the collected metric data for use by the training service.

To collect metrics from Prometheus:

Use Prometheus to identify the available metric names, and select the metrics to collect in the query configuration.
Configure Prometheus as a data source in the data service configuration file.
In the query_builder entry for the Prometheus data source in the Data Service configuration file, specify the query in the following format:

filters|start|end|step|metrics

where:
- filters is optional. Specify filters as comma-separated key-value pairs: key1=value1,key2=value2.
- start and end are mandatory. Specify the collection start and end times in ISO 8601 format.
- step is mandatory. It specifies the interval between collected data points.
- metrics is optional. Specify metric names as a comma-separated list: metric_name1,metric_name2.
Use filters to limit collection to a particular resource, such as a pod or virtual machine. If you specify only start|end|step, the query collects all available metrics from Prometheus without filtering.
Use the collect API to collect the metrics.

The data service retrieves the metrics from Prometheus and persists the collected data for downstream processing.
Use the persisted metric data to train the anomaly detection model.

If labels can be generated automatically, you can use the collected data directly for training. Otherwise, manually label the collected data before training. The Training Service uses the metric data and labels to train the model.

4 Deploying and Configuring the Data Service

Deploying the Data Service

Configuring and Using the Data Service

Configuring Multiple Data Sources

Collecting Metrics for Anomaly Detection