Deploy ONNX Format Models

Open Neural Network Exchange or ONNX is an open standard format of machine learning models. By using the Oracle Machine Learning Services REST API, you can deploy and score your ONNX format models. The REST API is exposed as part of the Oracle Machine Learning Services on Oracle Autonomous Database cloud service.

Note:

Oracle Machine Learning Services supports ONNX Runtime 1.11.1.

Supported Functions

Storing, deleting, and listing deployed models
Retrieving metadata and content of models
Organizing models under namespace
Creating, deleting, and listing model endpoints
Getting Swagger document for the model endpoint
Getting model details
Getting endpoint details
Scoring with model endpoints

Supported Machine Learning Models

The Oracle Machine Learning Services REST API supports ONNX format model deployment through REST endpoints for:

Classification models (both non-image models and image models)
Clustering models (non-image models)
Feature Extraction models (image models)
Regression models (non-image models)

Prerequisites

Before deploying your ONNX model, ensure the following:

Create the ONNX model zip file containing the modelName.onnx file, metadata.json file, and label.txt (optional) file.
Ensure that the metadata.json file contains the following as listed in the table Contents and Description of metadata.json file.

Note:

For more information about ONNX format model specifications, see Specifications for ONNX Format Models

Table - Contents and Description of metadata.json file

Field Name	Data Type	Description	Mandatory/Optional	Example
`function`	String	The mining function of the model. It accepts the value `"classification"` , `"regression"` or `"clustering"` Case-insensitive	Mandatory	`"classification"`
`clusteringDistanceOutput`	String	The name of the Clustering model output storing distances. For `topN` results, the outputs will be sorted by ascending order. The output is expected to be: A tensor value of type float/double. Case-sensitive	You must specify either: `clusteringDistanceOutput` or `clusteringProbOutput` Note: You cannot specify both.	`"scores"`
`clusteringProbOutput`	String	The name of the Clustering model output storing probabilities. For `topN` results, the outputs will be sorted by descending order. The output is expected to be: A tensor value of type float/double. Case-sensitive	You must specify either: `clusteringDistanceOutput` or `clusteringProbOutput` Note: You cannot specify both.	`"probabilities"`
`regressionOutput`	String	The name of the regression model output that stores the regression results. The output is expected to be a tensor with value type `float/double`. Case-sensitive	Optional - If the regression model has only one tensor output Mandatory - If the regression model has multiple outputs	`"variable"`
`classificationLabelOutput`	String	The name of the Classification model output storing label results. The output is expected to be a tensor. Case-sensitive	Ignored - If the classification model has only one output, and it is a tensor storing probabilities Optional - If the classification model has exactly two outputs, one tensor storing label info and one sequence storing probabilities For other cases for Classification models: Optional - If `label.txt` is present. Mandatory - If `label.txt` is not provided.	`"label"`
`classificationProbOutput`	String	The name of the Classification model output storing probabilities. The output is expected to be either: A tensor value of type `float/double`, or A sequence of maps whose `<key, value>` pairs with value of type `float/double` storing `<label, probability>` information. Case-sensitive	Optional - If the Classification model has only one output and it is a tensor storing probabilities Optional - If the Classification model has exactly two outputs, one tensor storing label info and one sequence storing probabilities Mandatory - For other cases for Classification models	`"dense/Sigmoid:0"`
`featureExtractionOutput`	String	The name of the ONNX image model output that stores the features. The output is expected to be a tensor with value type `float/double`. Case-sensitive.	Optional - If the feature extraction model has only one tensor output Mandatory - If the feature extraction model has multiple outputs	`"1446"`
`inputTensorDimNames` For ONNX image models only	String	Specify what each dimension means in the input tensor. The dimension name can be `"batch", "height", "width", "channel"`. Case-sensitive. Use lower case only.	Mandatory - For image models Specify the value along with `height, width, channel` Else, an exception is raised Ignored - For non-image models	`["batch", "height", "width", "channel"]`
`height` For ONNX image models only	Integer	Specify the image height for preprocessing.	Mandatory - or image models Specified the value along with `inputTensorDimNames, width, channel` Else, an exception is raised Ignored - For non-image models	`224`
`width` For ONNX image models only	Integer	Specify the image width for preprocessing.	Mandatory - For image models Specify the value along with `inputTensorDimNames, height, channel` Else, an exception is raised Ignore - For non-image models	`224`
`channel` For ONNX image models only	Integer	Specify the image channel number for preprocessing.	Mandatory - For image models Specify the value along with `inputTensorDimNames, height, width`. Else, an exception is raised Ignored - For non-image models	`3`
`mean` For ONNX image models only	Float[]	Specify the mean value for R, G, B (red, green, blue) channel for normalization.	Optional - For image models. If not specified, then the default value is `[0.0, 0.0, 0.0]` in the order RGB (red, green, blue) Ignored - For non-image models	`[112.1, 105.2, 89.4]`
`scale` For ONNX image models only	Float[]	Specify the scale value for R, G, B (red, green, blue) channel for normalization.	Optional - For image models. If not specified, then the default value is `[1.0, 1.0, 1.0]` in the order RGB (red, green, blue) Ignored - For non-image models	`[0.8, 1.0, 0.9]`
`inputChannel` For ONNX image models only	String	Specify the channel order for preprocessing. Currently, only three channel images are supported. Value can be either "RGB" or "BGR". Case-insensitive.	Optional - For image models. If not specified, then the default value is `"RGB"` Ignored - For non-image models	`"BGR"`

To deploy your ONNX model:

Create the ONNX.zip file by using the following command:
```
zip -r modelName.zip modelName.onnx, metadata.json and label.txt
```
The zip file must contain the following files:
- modelName.onnx file (Mandatory). This is the ONNX model file.
  
  Note:
  You can create your own ONNX models, for example modelName.onnx, in your favorite framework which has integrated ONNX support. Alternatively, you can also convert your models to the equivalent ONNX format with ONNX model converters.
- metadata.json file (Mandatory). This file contains the following information function, regressionOutput, classificationLabelOutput, classificationProbOutput, inputTensorDimNames, height, width, channel, mean, scale, inputChannel, clusteringDistanceOutput, clusteringProbOutput
  
  Example of a metadata.json file:
```
{
  "function": "classification",
  "classificationProbOutput": "dense/Sigmoid:0",
  "inputTensorDimNames": ["batch", "height", "width", "channel"],
  "height": 224,
  "width": 224,
  "channel": 3,
  "mean": [123.68, 116.779, 103.939],
  "scale": [1.0,1.0,1.0],
  "inputChannel": "BGR"
}
```
- label.txt file (Optional). This file stores labels for Classification or Clustering models. The labels should be listed in the same order as the model was trained. This file is not required if the scoring outputs contain label information. It those cases, it will be ignored.
  
  Example of a label.txt file:
```
scorpion-winter
cinturato-winter
ice-zero-fr
winter-sottozero
```
Obtain a token by following the instructions provided in Authenticate. This is the bearer token - a security token in the HTTP authentication scheme. You must send this token in the Authorization header when making requests.

Note:
The database accounts that can obtain tokens must have OML_DEVELOPER role.

Scoring an ONNX Regression Model

Step 3 - Step 6 provides the steps to score an ONNX regression model.

Store the ONNX model in the model repository in the database by sending a POST request to the Model Repository Service.
Example of a POST request to store an ONNX model:
```
curl -X POST --header "Authorization: Bearer $token" 
<oml-cloud-service-location-url>/omlmod/v1/models \
-H 'content-type: multipart/form-data; boundary=Boundary' \
-F modelData=@sk_rg_onnx.zip \
-F modelName=onnxRegressionModel \
-F modelType=ONNX \
-F version=1.0 \
-F 'description=onnx sk regression model version 1.' \
-F shared=true
```
Note:
When you store the model in the repository, a unique ID is generated. This is the modelId that you use when creating the model endpoint.
This is an example of a regression model in ONNX format. In this example:
- $token represents an environmental variable that is assigned to the token obtained through the Authorization API.
- sk_rg_onnx.zip is the ONNX zip file.
- onnxRegressionModel is the model name.
Create the model endpoint by sending a POST request to the Deployment Service. This request takes the modelId and URI as inputs.

Note:
Only the model owner can deploy the model. The model owner is the user who stores the model. A new endpoint is created for the deployed model.
Example of POST request to deploy an ONNX model:
```
curl -X POST --header "Authorization: Bearer $token" 
<oml-cloud-service-location-url> /omlmod/v1/deployment \
-H 'Content-Type: application/json' \
-d '{\
  "modelId": "29e99b4b-cfc2-4189-952a-bd8e50fb8046",\
  "uri": "onnxrg"\
}' 
```
In this example:
- $token represents an environmental variable that is assigned to the token obtained through the Authorization API.
- 29e99b4b-cfc2-4189-952a-bd8e50fb8046 is the modelId. The modelId is generated when you store the model in the repository.
- onnxrg is the URI.
Get the Open API document for this model endpoint by sending a GET request to /deployment/{uri}/api.

Example of getting the Open API document of a deployed ONNX regression model
```
curl -X GET --header "Authorization: Bearer $token" 
<oml-cloud-service-location-url>/omlmod/v1/deployment/onnxrg/api
```
In this example:
- $token represents an environmental variable that is assigned to the token obtained through the Authorization API.
- onnxrg is the URI.

Score with the model by sending a POST request to deployment/{uri}/score. The GET response to {uri}/api provides detailed information about it.

Note:

Prediction details are not supported for ONNX model scoring.

Example of Scoring a Mini-batch with two Input Records

curl -X POST --header "Authorization: Bearer $token" 
<oml-cloud-service-location-url> /omlmod/v1/deployment/onnxrg/score \
-H 'Content-Type: application/json' \
-d '{\
  "inputRecords": [\
    {\
      "input": [\
        [\
          -0.07816532,\
          0.05068012,\
          0.07786339,\
          0.05285819,\
          0.07823631,\
          0.0644473,\
          0.02655027,\
          -0.00259226,\
          0.04067226,\
          -0.00936191\
        ]\
      ]\
    },\
    {\
      "input": [\
        [\
          0.0090156,\
          0.05068012,\
          -0.03961813,\
          0.0287581,\
          0.03833367,\
          0.0735286,\
          -0.07285395,\
          0.1081111,\
          0.01556684,\
          -0.04664087\
        ]\
      ]\
    }\
  ]\
}

This is a deployed ONNX regression model with URI onnxrg. In this example:

$token represents an environmental variable that is assigned to the token obtained through the Authorization API.
POST request is sent to /deployment/onnxrg/score.

This completes the task of scoring an ONNX regression model.

Scoring an ONNX Image Model

Step 7 - Step 10 provides an example of deploying an ONNX image model. An image model is one that consumes image data.

You can score an ONNX image model in two ways:

By uploading a single image file as input. In this case, the equivalent tensor format of the uploaded image is generated with proper preprocessing for the model.
By uploading a collection of scoring input records (similar to ONNX non-image models). Here, mini batch scoring with ONNX image models is done with <model_attribute_name, value/array> pairs. Here, value/array can be either an encoded image in base64 string or a multidimensional array storing processed image pixel values. The supported ONNX image model must have either 1 or -1 tensor batch size.

To deploy an ONNX image model:

After creating the ONNX model.zip file, store the model in the model repository by using the POST request.
Example of Storing an ONNX Image Model
```
curl -X POST --header "Authorization: Bearer $token" 
<oml-cloud-service-location-url>/omlmod/v1/models \
-H 'content-type: multipart/form-data; boundary=Boundary' \
-F modelData=@img_onnx.zip \
-F modelName=onnxImageModel \
-F modelType=ONNX_IMG \
-F version=1.0 \
-F 'description=onnx image model version 1.' \
-F shared=true
```
This example shows a POST request to store an ONNX image model in the Model Repository. In this example:
- $token represents an environmental variable that is assigned to the token obtained through the Authorization API.
- img_onnx.zip is the ONNX Model zip file.
- onnxImageModel is the model name.
Create the model endpoint by sending a POST request to the Deployment Service. This method takes the modelId and URI as inputs.
Example of Creating a Model Endpoint
```
curl -X POST --header "Authorization: Bearer $token" 
<oml-cloud-service-location-url> /omlmod/v1/deployment \
-H 'Content-Type: application/json' \
-d '{\
  "modelId": "5ee516bb-e35e-49f4-b282-bb2d38bedc32",\
  "uri": "onnximg"\
}'
```
This example shows a POST to create an endpoint. In this example:
- $token represents an environmental variable that is assigned to the token obtained through the Authorization API.
- 5ee516bb-e35e-49f4-b282-bb2d38bedc32 is the modelId. The modelID is generated when you store the model in the repository.
- onnximg is the URI.
Note:
Only the model owner can deploy the model. The model owner is the user who stores the model.
Get the Open API document for this ONNX model by sending a GET request to /deployment/{uri}/api.
Example of Getting the Open API document of a Deployed ONNX Image Model
```
curl -X GET --header "Authorization: Bearer $token" 
<oml-cloud-service-location-url>/omlmod/v1/deployment/onnximg/api
```
This example shows a GET request to return the Open API document of the deployed ONNX image model. In this example:
- $token represents an environmental variable that is assigned to the token obtained through the Authorization API.
- onnximg is the URI.
Score the model by sending a POST request to deployment/{uri}/score.
Example of a POST request to score the ONNX image model
```
curl -X POST --header "Authorization: Bearer $token" 
<oml-cloud-service-location-url>/omlmod/v1/deployment/onnximg/score \
-F imageData=@test.png
```
In this example:
- $token represents an environmental variable that is assigned to the token obtained through the Authorization API.
- onnximg is the URI
- test.png is the image file
This completes the task of scoring an ONNX image model.