Detect Anomalies Using Asynchronous Detection

In this tutorial, you detect anomalies in detection datasets using the Anomaly Detection service asynchronous detection feature in both univariate and Multivariate detection datasets.

Overview

The terms detecting anomalies and inferencing are used interchangeably in this tutorial to mean the same thing, detecting anomalies in time series data.

Typical use cases and scenarios suited for asynchronous inferencing are:

Detect anomalies in large datasets

The maximum number of data points supported by the detectAnomalies REST API (Synchronous API) is 30K. This might impose restrictions in anomaly detection scenarios where a large number of data points, typically in the millions, need to be detection. Use asynchronous detection to analyze and detect anomalies in large datasets such as 10 million data points.
Automate detection workflows

In IoT use cases, time series data is collected from large number of sensors and devices, and stored in a persistent data store such as a database or a file system. Often, this raw data has to be preprocessed (enriched) using PaaS services such as Data Flow, prior to performing inferencing. You can easily integrate asynchronous detection API calls within data processing pipelines to automate detection workflows.
Postprocess anomalous events

In certain anomaly detection scenarios, the detection data (detected anomalies) might need to be transformed or enriched before it can be consumed by downstream applications. With asynchronous detection, detected anomalies are saved in an Object Storage location (bucket). You can use PaaS services, such as Data Flow, to analyze, process, and enrich the anomalous events. Furthermore, the anomalies can also be easily consumed and rendered on visualization graphs in Oracle Analytics Cloud to monitor target systems and take corrective actions.

At a high level, the process for detecting anomalies using asynchronous inferencing is:

Training a model with a training dataset.
Detect anomalies with an detection dataset. Use the asynchronous detection API or the OCI Console to create an asynchronous job.

Key tasks:

Upload detection dataset to an Object Storage bucket.
Create an asynchronous job.
Review the anomaly detection results.

For more information, see the following resources:

1. Prepare

Prepare your environment for authenticating and using the Anomaly Detection service.

A paid Oracle Cloud Infrastructure (OCI) account, or a new account with Oracle Cloud Promotions, see Request and Manage Free Oracle Cloud Promotions.
Administrator privilege for the OCI account.
At least one user in your tenancy that wants to use Anomaly Detection service. This user must be created in IAM.
By default, only users in the Administrators group have access to all Anomaly Detection resources. If you aren't an administrator user, you must request an administrator to create OCI and Anomaly Detection policies and assign them to your group.
You must have a compartment to use to provision the required resources for use in this tutorial.
Ensure that you have created and trained an Anomaly Detection model.

2. Upload the detection Dataset File

The detection dataset file must be uploaded to an Object Storage bucket so that the Anomaly Detection service can access it when creating and running jobs.

(2.1) Uploading the detection Dataset

Before you begin:

Download the detection dataset for Use Case No. 4 (Monitor Blood Glucose Levels) from the Univariate Tutorial and save it to a local directory. This use case pertains to detecting abnormal blood glucose levels (anomalies) in a patient's blood data. In case you trained the Anomaly Detection model with a different dataset, ensure you are using the corresponding detection dataset.

Open the navigation menu and click Storage.
Under Storage, click Buckets.

Next, store the detection dataset in an Object Store bucket.
Click Create Bucket to create a new bucket for storing training data files, and enter the following values:
You can use the default name or specify a name for the bucket. Leave the other fields as is, and then click Create.

The page for the new bucket appears after it's created.
Under Objects, click Upload to upload the detection dataset.
(Optional) Specify a name for the file to upload to the bucket.
Select the detection file from a local directory and click Upload.
Click Close after the file completes uploading.

The uploaded file is listed under Objects.

3. Create an Asynchronous Job

An asynchronous job is required to detect anomalies for large datasets.

(3.1) Creating an Asynchronous Job

Open the navigation menu and click Analytics & AI. Under AI Services, click Anomaly Detection.
Select the project you want to use.
Click Jobs.
Click Create job.

Specify these values for the job and leave all other fields as is:


Field Name	Action
Model	Select the anomaly detection model for the job to use.
Input request	Select Object store.
Input bucket	Select the bucket that contains the detection dataset file.
detection data	Select the detection dataset file.
Output bucket	Select the bucket where you want the service to save anomaly detection results.

Click Create job.
The asynchronous job Status is initially Accepted until the job starts running, then it's In Progress, and finally when the job finishes the status changes to Succeeded. The detection dataset is relatively small so the job should finish within a minute or so.