Creating Oracle Object Storage Sources

This topic explains how to create an Oracle Object Storage Source using the Oracle Unity API.

In this topic:

Introduction

You can ingest data from your Oracle Object Storage (OOS) into Oracle Unity by creating an OOS source. This tutorial will walk through how to retrieve your OOS details needed to create an OOS source using the Oracle Unity APIs. Using the API endpoints is currently the only method to create an Oracle Object Storage source.

Retrieving Oracle Object Storage Details

This section will walk through getting the details from your OOS to create your source. To create an OOS source, you'll need:

The Source's Unique Instance ID
The Oracle Object Storage Instance URL Path
The Storage Endpoint
The Storage Key
The Storage Secret

Source Unique Instance ID

When creating an Oracle Object Storage, create a unique identifier for the source. Your Source ID is displayed in the interface.

When using the APIs to create a source, ensure the uniqueInstanceId you specify is unique.

Store Instance URL Path

An Oracle Object Storage Instance URL Path is comprised of three unique components:

s3a://<oci_bucket_name>/<folder_structure>/{*<Source_Object>*}

<oci_bucket_name>

You can find your bucket name in your Oracle Object Storage under Object Storage > Buckets.
<folder_structure>

The folder path where your source file is located. For example "unity_data".
{*<Source_Object>*}

The name of your source file to ingest. For example, "Account_List_1.csv".

Oracle Object Storage Endpoint

An Oracle Object Storage endpoint is comprised of two unique components:

<namespace>.compat.objectstorage.<oci_region>.oraclecloud.com

<namespace>

Your Oracle Object Storage Source file's namespace is found within in your bucket details.
<oci_region>

The active region of your Oracle Object Storage. You can find your region under Infrastructure Regions (for example, "ap-hyderabad-1", "eu-frankfurt-1", "ca-toronto-1", and so on).

Oracle Object Storage Key

Your Oracle Object Storage Key is located in Oracle Object Storage under Identity & Security > Users > Customer Secret Keys.

If you don't already have a key, create a new user account.

Oracle Object Storage Secret

Your Oracle Object Storage Secret was displayed when you generated a Secret Key.

If you haven't already generated a secret key, follow the instructions below.

To generate a Secret Key:

In your Oracle Object Storage navigate to Identity & Security > Users.
Select the user.
Under Resources click Customer Secret Keys.
Enter a name for the key and click Generate Secret Key.
Copy the secret key and save it for your records.

Creating Oracle Object Storage Sources

This section will walk through how to structure the request to create the source.

Send a POST request to create a new source.

Request URL

POST /api/cx-unity-api/source_mcps_instances

Request body

{
   "tenantId":"<tenant_id>",
   "name":"<new_source_name>",
   "description":"<source_description>",
   "active":true,
   "componentDefinitionId":"FTPCustom",
   "uniqueInstanceId":"<source_id>",
   "isAutomaticFmt":true,
   "isStandardFmt":false,
   "parameters":{
      "targetSystem":"OOS",
      "delimiter":",",
      "charset":"UTF-8",
      "fileCompressionFormat":"gzip",
      "rfc4180Csv":true,
      "csv":false,
      "type":"auto",
      "folderName":"s3a://<oci_bucket_name>/<folder_structure>/{*<Source_Object>*}",
      "objectStoreEndpoint":"<namespace>.compat.objectstorage.<oci_region>.oraclecloud.com",
      "objectStoreKey":"<object_store_key>",
      "objectStoreSecret":"<object_store_secret>"
   }
}

The following table provides more information about the required parameters.

Name	Type	Description	Possible values
`tenantId`	string	Enter your MCPS tenant ID.
`name`	string	Enter a name for the source.
`description`	string	Enter a meaningful description for the source.
`active`	boolean	Whether the source instance is active. Specify false to disable this source instance, preventing jobs from referencing it.	`true` `false`
`componentDefinitionId`	string	Enter the source definition this source instance uses (a source definition is a template for a source instance.) For Oracle Object Storage Sources, must be `FTPCustom`.	`FTPCustom` (Oracle Object Storages)
`uniqueInstanceId`	string	Enter a unique ID for the source. Refer to Source Unique Instance ID for more information on how to retrieve this in Oracle Unity after the source is created.
`isAutomaticFmt`	boolean	Whether the source will have an automatic file name pattern. You can learn more about each file name pattern type in the Unity Help Center. A source's file pattern type cannot be changed after the source is created. To enable automatic file name patterns: `isAutomaticFmt`: true `isStandardFmt`: false To enable custom regular expression file patterns: `isAutomaticFmt`: false `isStandardFmt`: false	`true` `false`
`isStandardFmt`	boolean	Whether the source will have a traditional file name pattern where you must specify the file prefix, file suffix, and file name. A source's file pattern type cannot be changed after the source is created. You can learn more about each file name pattern type in the Unity Help Center. To enable traditional file name patterns: `isAutomaticFmt`: false `isStandardFmt`: true	`true` `false`

Nested schema: parameters

Name	Type	Description	Possible values
`targetSystem`	string	Target system for the source, must be "`OOS`".
`folderName`	string	Location of the Oracle Object Storage. Refer to Store Instance URL Path for more information on how to retrieve this.
`objectStoreEndpoint`	string	The URL endpoint location of your Oracle Object Storage. Refer to Oracle Object Storage Endpoint for more information on how to retrieve this.
`objectStoreKey`	string	The object store key of your Oracle Object Storage instance. Refer to Oracle Object Storage Key for more information on how to retrieve this.
`objectStoreSecret`	string	The store secret of your Oracle Object Storage instance. Refer to Oracle Object Storage Secret for more information on how to retrieve this.
`referenceName`	string	Must be the same value as your `uniqueInstanceId`. Refer to Source Unique Instance ID for more information on how to retrieve this.
`type`	string	Must be `regex`.
`fileRegex`	string	The name of your file stored in the Oracle Object Storage, enclosed by a file regular expression with periods and brackets. The syntax is: `.(<Source_Object>).` For example if the name of my file is "Account_List_1.csv", the `fileRegex` would resemble: `"fileRegex": ".(Account_List_1.csv)."` This value should must be the same as the specified `path` 's source object. Refer to Store Instance URL Path for more information on how to structure this.
`delimiter`	string	The field delimiter is the character in the source files that divides the fields. If you select the RFC 4180 parser, the comma (,) delimiter must be selected. If you select the Open CSV parser, you can select one of the available characters (comma (,), semi-colon (;), pipe (\|), tab( ), or enter a custom delimiter. When entering a custom delimiter, keep the following in mind: You cannot use the backward slash or double quote characters. You must use a single character. You cannot enter multiple characters for the delimiter.	(,) (;) (\|) tab( ) custom delimiter
`charset`	string	Select the format of the character set for the source file. The following character sets are supported: UTF-8, ISO-8859-1, Windows-1251, and Windows-1252.	`UTF-8` `ISO-8859-1` `Windows-1251` `Windows-1252`
`csv`	boolean	Applicable to SFTP sources only. Whether Oracle Unity will parse the csv files using the Opencsv parser. If the files in the SFTP site are "true" CSV files (optionally quoted) set to `true`. If the files are delimited files where quotes are parsed as input content, set to `false`. See Opencsv file standards for more information.	`true` `false`
`rfc4180Csv`	boolean	Applicable to FTP source instances only. Whether Oracle Unity will parse the csv files using the RFC4180 parser. The RFC4180 parser can ingest multi-line CSV data. See RFC 4180 file standards for more information.	`true` `false`

Example

POST https://{accountUrl}/api/cx-unity-api/source_mcps_instances

Sample request body

{
  "tenantId": 100,
  "name": "OCI_Source",
  "description": "OCI Source for data ingestion",
  "isAutomaticFmt": false,
  "isStandardFmt": false,
  "active": true,
  "componentDefinitionId": "FTPCustom",
  "uniqueInstanceId": "source_1",
  "isAutomaticFmt": true,
  "isStandardFmt": false,
  "parameters": {
    "targetSystem": "OOS",
    "folderName": "s3a://oci/data/{*Account_List*}",
    "objectStoreEndpoint": "nc1j2zwn3oid.compat.objectstorage.ap-hyderabad-1.oraclecloud.com",
    "objectStoreKey": "a2d2c12345d456b585df08886474ae4fb09f0d5",
    "objectStoreSecret": "NC123UdvF/EYi7ZifoV1o5geHQQuVWpRJdT1OVNaOzg=",
    "referenceName": "source_1",
    "type": "regex",
    "fileRegex": ".*(Account_List).*",
    "delimiter": ",",
    "charset": "UTF-8",
    "csv": "true"
  }
}

Sample response

A successful response will echo the details of the source specified in the request, with additional properties such as the created and modified date and timestamps.

{
  "tenantId": 100,
  "name": "OCI_Source",
  "versionTS": 1621341251094,
  "description": "OCI Source for data ingestion",
  "isAutomaticFmt": false,
  "isStandardFmt": false,
  "active": true,
  "lastModifiedBy": "df5ce199266b4bdf86d9f33d10cf850a",
  "createdBy": "df5ce199266b4bdf86d9f33d10cf850a",
  "createdTS": 1621341251094,
  "componentDefinitionId": "FTPCustom",
  "uniqueInstanceId": "source_1",
  "parameters": {
    "targetSystem": "OOS",
    "folderName": "s3a://oci/data/{*Account_List*}",
    "objectStoreEndpoint": "nc1j2zwn3oid.compat.objectstorage.ap-hyderabad-1.oraclecloud.com",
    "objectStoreKey": "a2d2c12345d456b585df08886474ae4fb09f0d5",
    "objectStoreSecret": "NC123UdvF/EYi7ZifoV1o5geHQQuVWpRJdT1OVNaOzg=",
    "referenceName": "source_1",
    "type": "regex",
    "fileRegex": ".*(Account_List).*",
    "delimiter": ",",
    "charset": "UTF-8",
    "csv": "true"
  }
}

Next you can create an ingest job to use your new source.

Learn more

Sources API

Creating Oracle Object Storage Destinations

Creating sources in the Help Center