Before You Begin

In this 15-minute tutorial, you learn how to use Oracle Big Data Manager to register Object Storage Classic as a storage provider, ingest external data into HDFS, and back up data using Oracle Object Store Classic.

Background

Oracle Big Data Manager provides a browser-based GUI to simplify data transfer tasks. You can use the Data explorer feature in Oracle Big Data Manager to upload local files to HDFS, and transfer files between a variety of storage providers.

What Do You Need?

• Access to either an instance of Oracle Big Data Cloud Service or to an Oracle Big Data Appliance, and the required administrator login credentials.
• Access to Oracle Big Data Manager, on either an instance of Oracle Big Data Cloud Service or on an Oracle Big Data Appliance, and the required administrator privileges. A port must be opened to permit access to Oracle Big Data Manager, as described in Enabling Oracle Big Data Manager.

Register Object Storage Classic as a Storage Provider

In this section, you register your Object Storage Classic as a storage provider with Oracle Big Data Manager.

1. Sign in to Oracle Big Data Manager with your administrator login credentials. See Access Oracle Big Data Manager.
2. On the Oracle Big Data Manager page, click the Administration tab, and then click the Storage providers tab on the left side of the page. The Storage providers page is displayed. The available registered storage providers are displayed in a list. The hdfs storage provider is registered by default when you create your Big Data Cloud Service instance. In this case, the installer also registered hive as an additional storage provider. There are a variety of other storage providers that may be registered, including Amazon S3, Github, Oracle Database, and MySQL database.

   

3. To register Object Storage Classic as a storage provider with Oracle Big Data Manager, click Register a new storage. The Register storage providers wizard is displayed. It has four pages: General, Storage Details, Access, and Confirmation.
4. In the General wizard page, enter oci-osc in the Name field and enter Object Storage Classic in the Description field. Select Oracle Cloud Infrastructure Object Storage Classic from the Storage type drop-down list, and then click Next.
5. In the Storage Details wizard page, enter the Storage URL and Tenant values for your Object Storage Classic. Next, enter your username and password, and then click Test access to storage. If the storage details that you provided are correct, the Successful, storage details are correct message is displayed. A Preview of storage content section is displayed on the page. Click Next.



6. In the Access wizard page, select the users that can see this storage in Oracle Big Data Manager. Whether or not the selected users can actually access data on a given storage depends on the settings of the service that provides the storage. Select the bigdatamgr user from the available users list on the left side of the page, and then click Add selected. The selected user is moved to the selected users list on the right side of the page. Click Next.

   

7. In the Confirmation wizard page, review the settings. If you need to make a correction, click the back arrow . If you are satisfied with the settings, click Register. The Storage providers page is re-displayed and the newly registered Object Storage Classic storage is displayed in the list of available storage providers.

   

Ingest Data Using Big Data Manager

In this section, you upload the local taxidropoff.csv data file to HDFS.

1. Right-click the taxidropoff.csv file, select Save link as from the context menu, and then save it to your local machine. This dataset was created from several datasets on the NYC Taxi & Limousine Commission website.
2. On the Oracle Big Data Manager page, click the Data tab.
3. In the Data explorer section, select HDFS storage (hdfs) from the Storage drop-down list in the left portion of the Data explorer section. Click Open home directory on the toolbar to change the current directory to /user/bigdatamgr. Click New Directory on the toolbar and create a new directory named tax-limo, and then click Create.

   

4. Navigate to the /user/bigdatamgr/limo-taxi directory, and then click File upload on the toolbar. In the Files upload dialog box, click the Select files drop-down list, and then select Files upload. In the Open dialog box, navigate to your local directory that contains the taxidropoff.csv file, and then select the file. The taxidropoff.csv file is displayed in the Name column. Click Upload.

   

5. When the file is uploaded successfully to the /user/bigdatamgr/limo-taxi HDFS directory, the Upload has finished message is displayed in the Details section of the dialog box. Click Close to close the dialog box. The file is displayed in the /user/bigdatamgr/limo-taxi directory.

   

Back up Data Using Object Storage Classic

In this section, you back up the taxidropoff.csv data file in HDFS to Oracle Object Storage Classic.

1. In the Data explorer section, ensure that the HDFS storage (hdfs) storage is selected in the left-hand side storage provider panel and that the Oracle Object Storage Classic (oci-osc) storage is selected in the right-hand side storage provider panel.
2. Scroll down in the Oracle Object Storage Classic (oci-osc) storage provider panel and locate the /csv container. Double-click the container to open it. Click and drag the taxidropoff.csv file from the HDFS storage (hdfs) storage provider panel onto the csv container in the Oracle Object Storage Classic (oci-osc) storage provider panel.

   

3. The New copy data job dialog box is displayed. Accept the default selections, and then click Create. When the copy data job is completed successfully, click Close to close the dialog box.

   

   A back-up copy of the taxidropoff.csv file is displayed in the /csv container.

   

4. When you are finished with this session, sign out of your Oracle Big Data Manager and Oracle Big Data Cloud service.

Want to Learn More?

Oracle Big Data Manager Tutorials

Overview of Oracle Big Data Manager

Oracle Big Data Cloud Service