Oracle Storage Cloud Service

30.1 Introduction

Oracle Storage Cloud Service provides a reliable, secure, and scalable object storage solution for storing unstructured data that can be accessed anytime anywhere. It serves as a gateway for data consumption to many OPC services such as BDP, BICS and so on. These cloud services directly picks up files from Oracle Storage Cloud Service and therefore, integration with Oracle Storage Cloud Service becomes very essential and useful to manage end-to-end integration flows using Oracle Data Integrator.

Oracle Data Integrator (ODI) seamlessly integrates with Oracle Storage Cloud Service. With this integration, you can now connect to Oracle Storage Cloud Service from ODI for uploading or downloading files/objects onto/from local directory or HDFS present in Oracle Storage Cloud Service.

30.1.1 Concepts

Oracle Storage Cloud Service is an Infrastructure as a Service (IaaS) product, which provides an enterprise-grade, large-scale, object storage solution for files and unstructured data.

Oracle Storage Cloud Service comprises of the following concepts:

Oracle Storage Cloud Service Hierarchy

Oracle Storage Cloud Service stores data as objects within a flat hierarchy of containers. You can create an object within a container most commonly by uploading a file or from ephemeral unstructured data. A single object can hold up to 5 GB of data, but multiple objects can be linked together to hold more than 5 GB of contiguous data.

A container is a user-created resource, which can hold an unlimited number of objects, unless you specify a quota for the container. Note that containers cannot be nested. You can define custom metadata for both objects and containers.

Storage Types

This integration provides support for both Oracle Standard Storage and Archive Storage.

Standard Storage - It is useful for storing one or more files that are accessed frequently.
Archive Storage — It is ideal for storing data that are not frequently accessed such as email archives, data backups, and digital video and so on.

30.2 Installation and Configuration

Make sure you have read the information in this section before you start working with the Oracle Storage Cloud Service technology:

30.2.1 System Requirements & Certifications

Before performing any installation, you should read the system requirements and certification documentation to ensure that your environment meets the minimum installation requirements for the products you are installing.

The list of supported platforms and versions is available on Oracle Technical Network (OTN): http://www.oracle.com/technetwork/middleware/data-integrator/documentation/index.html

30.2.2 Technology Specific Requirements

The technology specific requirement for using Oracle Storage Cloud Service in ODI are:

You should have a dedicated pre-built technology named “Oracle Storage Cloud Service” defined similar to Oracle Object Storage.
As an ODI user, you should be able to create a Data Server from this technology and corresponding Physical and Logical schemas for the created Data Server. These physical and logical schemas are used by ODI Tools supported for Oracle Storage Cloud Service integration, used for uploading and downloading files/objects.

Supported datatypes for this technology are:

Numeric
String
Date

Datatype Mappings to and from Oracle will be defined for these data types and is similar to File Data Server Mappings.

30.3 Setting up the Topology

Setting up the topology consists of:

30.3.1 Creating an Oracle Storage Cloud Service Data Server

Create a data server for the Oracle Storage Cloud Service technology using the standard procedure, as described in Creating a Data Server of Administering Oracle Data Integrator guide. This section details only the fields required or specific for defining Oracle Storage Cloud Service data server:

In the Definition tab:
1. Data Server
  - Name – Name of the data server that will appear in Oracle Data Integrator.
2. Connection
  - Service URL – Oracle Cloud Storage Service URL. For Example: https://<identity-domain>.storage.oraclecloud.com
  - Service Name – It denotes the name of the service for the created service URL. For Example - Storage
  - User Name - Name of the user logging into the Oracle Storage Cloud Service
    
    Note:
    User Names should start with upper case and should not be real server names.
  - Password – Password of the logged in user
  - Identity Domain - It denotes the domain specific to the created storage instance. For Example – https://<identity-domain>.storage.oraclecloud.com

30.3.2 Creating an Oracle Storage Cloud Service Physical Schema

Create an Oracle Storage Cloud Service physical schema using the standard procedure, as described in Creating a Physical Schema in Administering Oracle Data Integrator guide.

Oracle Storage Cloud Service specific parameters are:

Name— Name of the physical schema created.
Container Name — It specifies the container to which you wish to associate the created physical schema. Select the required container from the Container Name drop-down list.
Directory (Work Schema) — This is the temporary folder on the local system used for getting files from Oracle Storage Cloud Service. If the directory does not exist, it is created. Specify the required location in the local system.

Create a logical schema for this physical schema using the standard procedure, as described in Creating a Logical Schema in Administering Oracle Data Integrator and associate it with a relevant context.

We use the created logical schema for getting Oracle Storage instance details. These details are used for connecting to Oracle Storage Cloud Service technology.

.

30.4 Working with Oracle Storage Cloud Service Tools

You can upload and download files to or from Oracle Storage Cloud Service through the following tools:

Note:

Apart from ODI Studio, you can also work with ODI Storage Cloud Service Tools from command line.

30.4.1 Uploading Files/Objects to Oracle Storage Cloud Service

ODI Storage Cloud Service Upload tool is used to upload single, multiple files, or an entire directory from HDFS or a local file system on to Oracle Storage Cloud Service.

To upload file(s) or directories to Oracle Storage Cloud Service,

Create a new Project.

For more details on how to create a project, see Creating an Integration Project of Developing Integration Projects with Oracle Data Integrator.
Below the created Project folder, create a Package.

For more details on how to create a package, see Creating and Using Packages of Developing Integration Projects with Oracle Data Integrator.

Select OdiStorageCSUpload tool available in the Toolbox. Add it to the created package.

Note:

All the parameters of the tool are displayed under General Tab.

Configure the required parameters:

Table 30-1 ODI Storage Cloud Service Upload Tool Parameters

Parameter	Description
`Target Logical schema`	Target Logical schema name of Oracle Storage Cloud Service instance. Container information is obtained from Logical schema through configured physical schema.
`Source Logical schema`	Name of the Source Logical schema configured for File or HDFS Data Server for upload of Local or HDFS Files to Oracle Storage Cloud Service. Directory structure is obtained from Logical schema through configured physical architecture.
`File Names filter`	Field to specify one or more files or directories to be uploaded to Oracle Storage Cloud Service recursively. It also supports the list of files separated by \| as a delimiter. The pattern followed is: .txt - should upload all the files ending with .txt test - uploads all the files and directories that matches with prefix “test” test - uploads all the files and directories having substring “test” test.xml \| test1.xml \| test2.xml - Uploads all the files specified test* \| test1* - Uploads all the files matching pattern test* and test1* test.xml - Only one file is uploaded
`Overwrite`	This parameter indicates if upload operation should overwrite an existing file or not. Default value for this parameter is No.
`Retry on error`	It represents the number of times the retry attempt should occur when a failure or error happens during upload.
`Retry interval seconds`	Retry interval indicates after how many seconds a retry attempt should happen.
`Encrypt Key`	This is the user provided key used for encrypting objects while uploading files or directories to Oracle Storage Cloud Service. Note: This parameter cannot be null, if you want to encrypt objects while upload.

For more details on the above parameters, refer to OdiStorageCSUpload Tool in Oracle Data Integrator Tools Reference guide.

Save and execute the package.

The required files from the source directory are uploaded to the target container of the Oracle Storage Cloud Service.
Upon successful upload, you can find a complete log of this upload operation at the Details tab. To get to the details tab, from the Operator tab, expand the associated session for the upload tool and open the Session task window to find the Details tab with the required log information.

The details include:
- Source directory is : <source directory path>
- Target container is : <Storage container name>
- Filter used is : <input filter>
- Number of file uploaded:<Total number of files that were uploaded>
- Uploaded files are: <File1, File2>
- Number of files failed:<Total number of files that were not uploaded>
- Failed files are:<File1, File2>

30.4.2 Downloading File/Objects from Oracle Storage Cloud Service

ODI Storage Cloud Service Download tool is used to download single, multiple files, or an entire directory to HDFS or a local file system from Oracle Storage Cloud Service. For HDFS files, the files from Oracle Storage Cloud Service are first copied to the local directory (as you specified in Directory (Work Schema) for Oracle Storage Cloud Service Physical Schema) and then from local directory, files are downloaded to HDFS.

To download file(s) or directories from Oracle Storage Cloud Service,

Create a new Project

For more details on how to create a project, see Creating an Integration Project of Developing Integration Projects with Oracle Data Integrator.
Below the created Project folder, create a Package

For more details on how to create a package, see Creating and Using Packages of Developing Integration Projects with Oracle Data Integrator.
Select OdiStorageCSDownload tool available in the Toolbox. Add it to the created Package.

Note:
All the parameters of the tool are displayed under General Tab.

Configure the required parameters:

Table 30-2 ODI Storge CS Download Tool Parameters

Parameter	Description
`Source Logical schema`	Source Logical Schema name configured for Oracle Storage Cloud Service instance. Container information is obtained from Logical schema through configured physical schema.
`Target Logical schema`	Logical Schema name configured for File or HDFS Data Server for download of Local or HDFS Files from Oracle Storage Cloud Service. Directory structure is obtained from Logical schema through configured physical architecture.
`File Names filter`	Field to specify one or more files or directories to be downloaded from Oracle Storage Cloud Service recursively. It also supports delimiter \| for separated files list. The pattern followed is: .txt - should download all files ending with .txt test - Downloads all the files and directories that matches with prefix “test” test - Downloads all the files and directories having substring “test” test.xml \| test1.xml \| test2.xml - Downloads all the files specified test* \| test1* - Downloads all the files matching pattern test* and test1* test.xml - Only one file is downloaded
`Overwrite`	This parameter indicates, if download operation should overwrite an existing file or not. The default value for this parameter is No.
`Retry on error`	It represents the number of times the retry attempt should occur when a failure or error happens during download.
`Retry interval seconds`	Retry interval indicates after how many seconds a retry attempt should happen.
`Decrypt Key`	This is the user provided key used for decrypting objects while downloading from Oracle Storage Cloud Service. This key should be same as the encrypt key provided during the upload of the same file (that you had uploaded earlier) to Oracle Storage Cloud Service. If you provide the wrong key then the download operation fails. Note: This parameter cannot be null, if you want to decrypt objects while download.

For more details on the above parameters, refer to OdiStorageCSDownload Tool Oracle Data Integrator Tools Reference.

Save and execute the package.

The required files from Oracle Storage Cloud Service are downloaded to the directory specified in the Target logical schema.
Upon successful download, you can find a detailed log of this download operation at the Details tab. To get to the details tab, from the Operator tab, expand the associated session for the download tool and open the Session task window to find the Details tab with the required log information.

The details include:
- Source container is : <Storage container name>
- Target directory is : <target directory path>
- Filter used is : <input filter>
- Number of file downloaded:<Total number of files that were downloaded>
- Downloaded files are:<File1, File2>
- Number of files failed:<Total number of files that were not downloaded>
- Failed files are:< File1, File2>