Get Started with Oracle Big Data Service

Oracle Big Data Service provides enterprise-grade Hadoop as a service, with end-to-end security, high performance, and ease of management and upgradeability.

About Oracle Big Data Service

Oracle Big Data Service is an Oracle Cloud Infrastructure service designed for a diverse set of big data use cases and workloads. From short-lived clusters used to tackle specific tasks to long-lived clusters that manage large data lakes, Big Data Service scales to meet an organization’s requirements at a low cost and with the highest levels of security.

Big Data Service includes:

  • A full Hadoop stack, with a complete installation of the Cloudera Distribution Including Apache Hadoop (CDH). CDH includes Cloudera Manager, Apache Hadoop, Apache Flume, Apache Hive, Apache Spark, Apache Hue, Apache Kafka, Apache Solr, Apache Sentry, and other services for working with and securing big data.
  • All of the tools and utilities from Cloudera Enterprise Data Hub.
  • Oracle Cloud Infrastructure features and resources, including identity management, networking, compute, storage, and monitoring.
  • A REST API for creating and managing clusters.
  • bda-oss-admin command line interface for managing storage providers.
  • odcp command line interface for copying and moving data.
  • The ability to create clusters of any size, based on native Oracle Cloud Infrastructure shapes. For example, you can create small, short-lived clusters in flexible virtual environments, very large, long-running clusters on dedicated hardware, or any combination between.
  • Optional secure, high availablity (HA) clusters.
  • Oracle Cloud SQL integration, for analyzing data across Apache Hadoop, Apache Kafka, NoSQL, and object stores using Oracle SQL query language.
Note

Oracle Big Data Manager, a browser-based tool for managing and analyzing data, isn't included in this release of Big Data Service. It is expected to be included in future releases.

Take a Getting Started Workshop to Learn Big Data Service

If you're new to Oracle Big Data Service and want to get up and running quickly, try one of the Getting Started with Oracle Big Data Service workshops. (There's one for a highly-available (HA) cluster and one for a non-HA cluster.) A series of step-by-step labs guide you through the process of setting up a simple environment and creating a small cluster. You'll also explore basic standard operations such as:
  • Add Cloud SQL to the cluster
  • Make cluster nodes accessible from the public internet
  • Use Cloudera Manager and Hue to access the cluster
  • Create a Hadoop administrator user

Big Data Service Regions, Limits, Quotas, Events, and Work Requests

Details about Oracle Big Data Service regions, limits, quotas, events, and work requests are listed below.

Region Availability

For the latest information on the regions where Oracle Big Data Service, Oracle Cloud SQL, and related services are available, see Data Regions for Platform and Infrastructure Services.

Service Limits

Big Data Service has the following default limits for paid accounts in all regions:

Resource Monthly Universal Credits Pay-as-You-Go
VM.Standard2.1 12 instances (12 OCPUs) 8 instances (8 OCPUs)
VM.Standard2.2 12 instances (24 OCPUs) 8 instances (16 OCPUs)
VM.Standard2.4 12 instances (48 OCPUs) 8 instances (32 OCPUs)
VM.Standard2.8 8 instances (64 OCPUs) Contact us
VM. Standard2.16 8 instances (128 OCPUs) Contact us
VM.Standard2.24 8 instances (192 OCPUs) Contact us

VM.DenseIO2.8

VM.DenseIO2.16

VM.DenseIO2.24

BM.HPC2.36

BM.DenseIO2.52

BM.Standard2.52

Contact us Contact us

Big Data Service also has the following limits for trial accounts. To obtain a free trial account, go to Oracle Free Tier.

Resource Trial Accounts
VM.Standard2.1 3 instances (3 OCPUs)
VM.Standard2.4 2 instances (8 OCPUs)

For more information about service limits, see Service Limits in the Oracle Cloud Infrastructure documentation.

To submit a request to increase your service limits, see Requesting a Service Limit Increase in the Oracle Cloud Infrastructure documentation.

Service Quotas

Big Data Service administrators can set quota policies to enforce restrictions on users by limiting the resources that they can create.

For information about how Oracle Cloud Infrastructure handles quotas, see About Compartment Quotas.

Use the following information to create quotas:

Service name: big-data

Quotas:
Quota Name Scope Description
vm-standard-2-1-ocpu-count Regional Number of VM.Standard2.1 OCPUs
vm-standard-2-2-ocpu-count Regional Number of VM.Standard2.2 OCPUs
vm-standard-2-4-ocpu-count Regional Number of VM.Standard2.4 OCPUs
vm-standard-2-8-ocpu-count Regional Number of VM.Standard2.8 OCPUs
vm-standard-2-16-ocpu-count Regional Number of VM.Standard2.16 OCPUs
vm-standard-2-24-ocpu-count Regional Number of VM.Standard2.24 OCPUs
vm-dense-io-2-8-ocpu-count Regional Number of VM.DenseIO2.8 OCPUs
vm-dense-io-2-16-ocpu-count Regional Number of VM.DenseIO2.16 OCPUs
vm-dense-io-2-24-ocpu-count Regional Number of VM.DenseIO2.24 OCPUs
bm-hpc2-36-ocpu-count Regional Number of BM.HPC2.36 OCPUs
bm-dense-io-2-52-ocpu-count Regional Number of BM.DenseIO2.52 OCPUs
bm-standard-2-52-ocpu-count Regional Number of BM.Standard2.52 OCPUs

Big Data Service quota policy examples:

  • Limit the number of VM.Standard2.4 OCPUs that users can allocate to services they create in the mycompartment compartment to 40.

    Set big-data quota vm-standard-2-4-ocpu-count to 40in Compartment mycompartment

  • Limit the number of BM.DenseIO2.52 OCPUs that users can allocate to services they create in the testcompartment compartment to 20.

    Set big-data quota bm-dense-io-2-52-ocpu-count to 20 in Compartment testcompartment

  • Don't allow users to create any VM.Standard2.4 OCPUs in the examplecompart compartment.

    Zero big-data quota vm-standard-2-4-ocpu-count in Compartment examplecompart

Service Events

Certain actions performed on Big Data Service clusters emit events.

You can define rules that trigger a specific action when an event occurs. For example, you might define a rule that sends a notification to administrators when someone deletes a resource. See Overview of Events and Get Started with Events.

The following table lists Big Data Service event types.

Event Type Event Type
Create Instance Begin com.oraclecloud.bds.cp.createinstance.begin
Create Instance End com.oraclecloud.bds.cp.createinstance.end
Terminate Instance Begin com.oraclecloud.bds.cp.terminateinstance.begin
Terminate Instance End com.oraclecloud.bds.cp.terminateinstance.end
Add Node Begin com.oraclecloud.bds.cp.addnode.begin
Add Node End com.oraclecloud.bds.cp.addnode.end
Add Block Storage Begin com.oraclecloud.bds.cp.addblockstorage.begin
Add Block Storage End com.oraclecloud.bds.cp.addblockstorage.end
Add Cloud SQL Begin com.oraclecloud.bds.cp.addcloudsql.begin
Add Cloud SQL End com.oraclecloud.bds.cp.addcloudsql.end
Remove Cloud SQL Begin com.oraclecloud.bds.cp.removecloudsql.begin
Remove Cloud SQL End com.oraclecloud.bds.cp.removecloudsql.end

Asynchronous Work Requests

Service Name Operation entityType actionType
Big Data Service

CreateBdsInstance

UpdateBdsInstance

DeleteBdsInstance

AddBlockStorage

AddWorkerNodes

AddCloudSql

RemoveCloudSql

ChangeBdsInstanceCompartment

UpdateBdsInstance

bds-instance

ACCEPTED

IN_PROGRESS

FAILED

SUCCEEDED

CANCELING

CANCELED

Open Big Data Service in the Oracle Cloud Console

Use the Big Data Service pages in the Oracle Cloud Console to view and manage the service and to create and manage clusters.
Note

You can't create clusters until you've set up your Oracle Infrastructure environment to support Big Data Service. See Set Up Oracle Cloud Infrastructure for Oracle Big Data Service.

Note

The Big Data Service pages give you access to your clusters, but you'll use other Oracle Cloud Console pages to manage other aspects of your service, for example the Networking pages for your network and the Identity pages for identity and access management.

To open Big Data Service in the Oracle Cloud Console:
  1. Open the Oracle Cloud Console. See Signing In to the Console in the Oracle Cloud Infrastructure documentation.
  2. In the Oracle Cloud Console, open the navigation menu navigation menu. Under AI and Big Data, select Big Data.