Creating and Modifying Clusters

Planning a Cluster

Before you create a cluster, understand your options.

Understand Instance Types and Shapes

Big Data Service cluster nodes run in Oracle Cloud Infrastructure compute instances (servers).

When you create a cluster, you choose an instance type, which determines whether the instance will run directly on the "bare metal" of the hardware or in a virtualized environment. You also choose a shape, which configures the resources assigned to the instance.

About Instance Types

There are two types of Oracle Cloud Infrastructure compute instances:

  • Bare Metal: A bare metal compute instance uses a dedicated physical server for the node, for highest performance and strongest isolation.

  • Virtual Machine (VM): Through virtualization, a VM compute instance can host multiple, isolated nodes that run on a single physical bare metal machine. VM instances are less expensive than bare metal instances and are useful for building less demanding clusters that don't require the performance and resources (CPU, memory, network bandwidth, storage) of an entire physical machine for each node.

VM instances run on the same hardware as bare metal instances, with the same firmware, software stack, and networking infrastructure.

For more information about compute instances, see Overview of Compute Service.

About Shapes

The shape determines the number of CPUs, amount of memory, and other resources allocated to the compute instance hosting the cluster node. See Supported Node Shapes in the Oracle Cloud Infrastructure documentation for the available shapes.

The shapes of the Big Data Service master nodes and the worker nodes don't have to match. But the shapes of all of the master nodes must match each other and the shapes of all the worker nodes must match each other.

Plan Your Cluster Layout, Shape, and Storage

Before you start the process to create a cluster, you must plan the layout of the cluster, the node shapes, and storage.

Cluster Layout

Nodes and services are organized differently on clusters, based on whether the cluster is highly available (HA) and secure, or not.

About Using HA Clusters

Use HA clusters for production environments. They're required for resiliency and to minimize downtime.

In this release, a cluster must be both HA and secure, or neither.

Types of Nodes

The types of nodes are as follows:

  • Master or utility nodes include the services required for the operation and management of the cluster. These nodes don’t store or process data.
  • Worker nodes store and process data. The loss of a worker node doesn't affect the operation of the cluster, although it can affect performance.
  • Compute only worker nodes process data. The loss of a compute only worker node doesn't affect the operation of the cluster, although it can affect performance.
    Note

    Compute only worker nodes aren’t supported for CDH clusters.
  • Edge nodes are extended nodes to the cluster that have only clients installed. You can install additional packages and run additional applications in this node instead of worker/compute/master nodes to avoid classpath conflicts and resources issues with cluster services.

High Availability (HA) Cluster Layout

A high availability cluster has two master nodes, two utility nodes, three or more worker nodes, and zero or more compute only worker nodes.

Type of node Services on ODH Services on CDH
First master node
  • Ambari Metrics Monitor
  • HDFS Client
  • HDFS JournalNode
  • HDFS NameNode
  • HDFS ZKFailoverController
  • Hive Client
  • Kerberos Client
  • MapReduce2 Client
  • Spark3 Client
  • Spark3 History Server
  • YARN Client
  • YARN ResourceManager
  • ZooKeeper Server
  • HDFS Failover Controller
  • HDFS JournalNode
  • HDFS NameNode
  • Hive Client
  • Key Trustee KMS Key Management Server Proxy
  • Key Trustee Server Active Database
  • Key Trustee Server Active Key Trustee Server
  • Spark Client
  • Spark History Server
  • YARN (MR2 Included) JobHistory Server
  • YARN (MR2 Included) ResourceManager
  • ZooKeeper Server
Second master node
  • Ambari Metrics Monitor
  • HDFS Client
  • HDFS JournalNode
  • HDFS NameNode
  • HDFS ZKFailoverController
  • Kerberos Client
  • MapReduce2 Client
  • MapReduce2 History Server
  • Spark3 Client
  • Tez Client
  • YARN Client
  • YARN Registry DNS
  • YARN ResourceManager
  • YARN Timeline Service V1.5
  • ZooKeeper Server
  • HDFS Balancer
  • HDFS Failover Controller
  • HDFS HttpFS
  • HDFS JournalNode
  • HDFS NameNode
  • Hive Client
  • Hue Load Balancer
  • Hue Server
  • Hue Kerberos Ticket Renewer
  • Key Trustee KMS Key Management Server Proxy
  • Key Trustee Server Passive Database
  • Key Trustee Server Passive Key Trustee Server
  • YARN (MR2 Included) ResourceManager
  • ZooKeeper Server
First utility node
  • Ambari Metrics Monitor
  • Ambari Server
  • HDFS Client
  • HDFS JournalNode
  • Hive Metastore
  • HiveServer2
  • Kerberos Client
  • MapReduce2 Client
  • Oozie Server
  • Spark3 Client
  • Tez Client
  • YARN Client
  • ZooKeeper Client
  • ZooKeeper Server
  • HDFS Client
  • HDFS JournalNode
  • Hive Client
  • Cloudera Management Service Alert Publisher
  • Cloudera Management Service Event Server
  • Cloudera Management Service Host Monitor
  • Cloudera Management Service Navigator Audit Server
  • Cloudera Management Service Navigator Metadata Server
  • Cloudera Management Service Reports Manager
  • Cloudera Management Service Monitor
  • Sentry Server
  • Spark Client
  • YARN (MR2 Included) Client
  • ZooKeeper Server
Second utility node
  • Ambari Metrics Collector
  • Ambari Metrics Monitor
  • HDFS Client
  • Hive Client
  • Kerberos Client
  • MapReduce2 Client
  • Spark3 Client
  • YARN Client
  • HDFS Client
  • Hive Client
  • Hive Metastore Server
  • HiveServer2
  • Hive WebHCat Server
  • Hue Load Balancer
  • Hue Server
  • Hue Kerberos Ticket Renewer
  • Oozie Server
  • Sentry Server
  • Spark Client
  • YARN (MR2 Included) Client
Worker nodes (3 minimum)
  • Ambari Metrics Monitor
  • HDFS DataNode
  • HDFS Client
  • Hive Client
  • Kerberos Client
  • MapReduce2 Client
  • Oozie Client
  • Spark3 Client
  • Spark3 Thrift Server
  • Tez Client
  • YARN Client
  • YARN NodeManager
  • ZooKeeper Client
  • HDFS DataNode
  • Hive Client
  • Spark Client
  • YARN (MR2 Included) NodeManager
Compute only worker nodes
  • Ambari Metrics Monitor
  • HDFS Client
  • Hive Client
  • Kerberos Client
  • MapReduce2 Client
  • Oozie Client
  • Spark3 Client
  • Tez Client
  • YARN Client
  • YARN NodeManager
  • ZooKeeper Client
NA
Edge nodes
  • Ambari Metrics Monitor
  • HDFS Client
  • Hive Client
  • Kerberos Client
  • MapReduce2 Client
  • Oozie Client
  • Spark3 Client
  • Tez Client
  • YARN Client
  • ZooKeeper Client
NA

Minimal (non-HA) Cluster Layout

A non-high availability cluster has one master node, one utility node, three or more worker nodes, and zero or more compute only worker nodes.

Type of node Services on ODH Services on CDH
Master node
  • Ambari Metrics Monitor
  • HDFS Client
  • HDFS NameNode
  • Hive Client
  • MapReduce2 Client
  • Spark3 Client
  • Spark3 History Server
  • YARN Client
  • YARN Registry DNS
  • YARN ResourceManager
  • ZooKeeper Server
  • HDFS Balancer
  • HDFS NameNode
  • Hive Client
  • Spark Client
  • Spark History Server
  • YARN (MR2 Included) JobHistory Server
  • YARN (MR2 Included) ResourceManager
  • ZooKeeper Server
Utility node
  • Ambari Metrics Collector
  • Ambari Metrics Monitor
  • Ambari Server
  • HDFS Client
  • HDFS Secondary NameNode
  • Hive Metastore
  • HiveServer2
  • MapReduce2 Client
  • MapReduce2 History Server
  • Oozie Server
  • Spark3 Client
  • Tez Client
  • YARN Client
  • YARN Timeline Service V1.5
  • ZooKeeper Client
  • ZooKeeper Server
  • HDFS HttpFS
  • HDFS SecondaryNameNode
  • Hive Client
  • Hive Metastore Server
  • HiveServer2
  • Hive WebHCat Server
  • Hue Load Balancer
  • Hue Server
  • Cloudera Management Service Alert Publisher
  • Cloudera Management Service Event Server
  • Cloudera Management Service Host Monitor
  • Cloudera Management Service Navigator Audit Server
  • Cloudera Management Service Navigator Metadata Server
  • Cloudera Management Service Reports Manager
  • Cloudera Management Service Monitor
  • Oozie Server
  • Spark Client
  • YARN (MR2 Included) Client
  • ZooKeeper Server
Worker nodes
  • Ambari Metrics Monitor
  • HDFS DataNode
  • HDFS Client
  • Hive Client
  • MapReduce2 Client
  • Oozie Client
  • Spark3 Client
  • Spark3 Thrift Server
  • Tez Client
  • YARN Client
  • YARN NodeManager
  • ZooKeeper Client
  • ZooKeeper Server
  • HDFS DataNode
  • Hive Client
  • Spark Client
  • YARN (MR2 Included) NodeManager
  • ZooKeeper Server
Compute only worker nodes
  • Ambari Metrics Monitor
  • HDFS Client
  • Hive Client
  • MapReduce2 Client
  • Oozie Client
  • Spark3 Client
  • Tez Client
  • YARN Client
  • YARN NodeManager
  • ZooKeeper Client
NA
Edge nodes
  • HDFS Client
  • Hive Client
  • MapReduce2 Client
  • Oozie Client
  • Spark3 Client
  • Tez Client
  • YARN Client
  • ZooKeeper Client
NA
Supported Node Shapes

The shape describes the compute resources allocated to the node.

The shapes used for master/utility nodes and worker nodes can be different. But all master/utility nodes must be of the same shape and all worker nodes must be of the same shape.

The following table shows what shapes can be used for the different node types. For a list of the resources provided by each shape, see Compute Shapes.

Node Type Available Shapes Required Number of Virtual Network Interface Cards (VNICs)
Master or utility

VM.Standard2.4

VM. Standard2.8

VM. Standard2.16

VM.Standard2.24

VM.Standard.E4.Flex *

VM.Standard3.Flex*

VM.Optimized3.Flex*

VM.DenseIO.E4.Flex*

VM.DenseIO2.8

VM.DenseIO2.16

VM.DenseIO2.24

BM.Standard2.52

BM.DenseIO2.52

BM.HPC2.36

BM.Standard3.64*

BM.Optimized3.36*

BM.DenseIO.E4.128*

BM.Standard.E4.128*

3 minimum

Used for the cluster subnet, the DP access subnet, and the customer's subnet

*You must specify a minimum of 3 OCPU and 32GB memory.

Note

The following shapes are not supported for CDH clusters. They are only supported for ODH clusters.

VM.Standard.E4.Flex

VM.Standard3.Flex

VM.Optimized3.Flex

VM.DenseIO.E4.Flex

BM.Standard3.64

BM.Optimized3.36

BM.DenseIO.E4.128

BM.Standard.E4.128

Worker

VM.Standard2.1*

VM.Standard2.2*

VM. Standard2.4

VM. Standard2.8

VM. Standard2.16

VM.Standard2.24

VM.Standard.E4.Flex *

VM.Standard3.Flex*

VM.Optimized3.Flex*

VM.DenseIO.E4.Flex*

VM.DenseIO2.8

VM.DenseIO2.16

VM.DenseIO2.24

BM.Standard2.52

BM.DenseI2.52

BM.HPC2.36

BM.Standard3.64*

BM.Optimized3.36*

BM.DenseIO.E4.128*

BM.Standard.E4.128*

2 minimum

Used for the cluster subnet and the customer's subnet

* Be aware that VM.Standard2.1 and VM.Standard2.2 are small shapes and won't support running large workloads. For VM.Standard.E4.Flex, VM.Standard3.Flex, and VM.Optimized3.Flex you must specify minimum of 1 OCPU and 16GB memory.

Note

The following shapes are not supported for CDH clusters. They are only supported for ODH clusters.

VM.Standard.E4.Flex

VM.Standard3.Flex

VM.Optimized3.Flex

VM.DenseIO.E4.Flex

BM.Standard3.64

BM.Optimized3.36

BM.DenseIO.E4.128

BM.Standard.E4.128

Compute only worker

VM.Standard2.1*

VM.Standard2.2*

VM. Standard2.4

VM. Standard2.8

VM. Standard2.16

VM.Standard2.24

VM.Standard.E4.Flex *

VM.Standard3.Flex*

VM.Optimized3.Flex*

VM.DenseIO.E4.Flex*

VM.DenseIO2.8

VM.DenseIO2.16

VM.DenseIO2.24

BM.Standard2.52

BM.DenseI2.52

BM.HPC2.36

BM.Standard3.64*

BM.Optimized3.36*

BM.DenseIO.E4.128*

BM.Standard.E4.128*

2 minimum

Used for the cluster subnet and the customer's subnet

* Be aware that VM.Standard2.1 and VM.Standard2.2 are small shapes and won't support running large workloads. For VM.Standard.E4.Flex, VM.Standard3.Flex, and VM.Optimized3.Flex you must specify minimum of 1 OCPU and 16GB memory.

Compute only worker nodes are not supported for CDH clusters.

Note

The following shapes are not supported for CDH clusters. They are only supported for ODH clusters.

VM.Standard.E4.Flex

VM.Standard3.Flex

VM.Optimized3.Flex

VM.DenseIO.E4.Flex

BM.Standard3.64

BM.Optimized3.36

BM.DenseIO.E4.128

BM.Standard.E4.128

Edge

VM.Standard2.1*

VM.Standard2.2*

VM. Standard2.4

VM. Standard2.8

VM. Standard2.16

VM.Standard2.24

VM.Standard.E4.Flex *

VM.Standard3.Flex*

VM.Optimized3.Flex*

VM.DenseIO.E4.Flex*

VM.DenseIO2.8

VM.DenseIO2.16

VM.DenseIO2.24

BM.Standard2.52

BM.DenseI2.52

BM.HPC2.36

BM.Standard3.64*

BM.Optimized3.36*

BM.DenseIO.E4.128*

BM.Standard.E4.128*

2 minimum

Used for the cluster subnet and the customer's subnet

* Be aware that VM.Standard2.1 and VM.Standard2.2 are small shapes and won't support running large workloads. For VM.Standard.E4.Flex, VM.Standard3.Flex, and VM.Optimized3.Flex you must specify minimum of 1 OCPU and 16GB memory.

Note

Since the Edge node is specific to client application usecases, choose shape as per application requirement.

Edge nodes are not supported for CDH clusters.

Cloud SQL query server

VM. Standard2.4

VM. Standard2.8

VM. Standard2.16

VM.Standard2.24

VM.Standard.E4.Flex *

VM.Standard3.Flex*

VM.Optimized3.Flex*

VM.DenseIO.E4.Flex*

VM.DenseIO2.8

VM.DenseIO2.16

VM.DenseIO2.24

BM.HPC2.36

BM.Standard2.52

BM.Optimized3.36*

--

Not all shapes are available by default. To see what shapes are available by default through the Cloud console, see Find the Limits for Your Tenancy . To submit a request to increase your service limits, see Request an Increase for Big Data Service Nodes.

Allocation of Block Storage for Nodes with Standard VM Shapes

Nodes based on standard VM shapes use network-attached block storage.

Note

Block storage isn't supported for nodes based on DenseIO and HPC shapes.

All nodes have a boot volume of 150 GB.

Option Limits/Guidelines
Minimum initial block storage 150 GB
Default initial block storage * 150 GB
Minimum additional block storage 150 GB
Default additional block storage * 1 TB
Incremental step for (initial and additional) block storage 50 GB
Maximum block storage for a single node

48 TB

The 48 TB total results from 12 volumes of 4 TB each.

If you add block storage multiple times, the maximum remains 48 TB, but it may be spread across more than 12 volumes.

Maximum block volume size

4 TB

If you specify the maximum 48 TB, 12 drives of 4 TB each are created.

If you specify a lower number, enough 4 TB devices for that amount are created, and more devices are created as you add more storage.

The figures below show initial sizes only. because you can't add additional block storage to master or utility nodes.

Option Limits and Guidelines
Minimum initial block storage 150 GB
Default initial block storage 1 TB
Minimum additional block storage 150 GB
Default additional block storage 1 TB
Incremental step for (initial and additional) block storage 50 GB
Maximum block storage for a single node 32TB
Maximum block volume size 32 TB
MySQL placement For utility nodes move /var/lib/mysql to /u01 and create a symbolic link. This prevents filling up the boot volume.
Option Guidelines
Default initial block storage 2 TB
Minimum initial block storage 150 GB

Query server storage is used for temporary table space to perform heavy JOIN and GROUP BY operations. 2 TB is recommended for typical processing. For small environments, for example development, this number can be adjusted down.

For best performance, consider these factors:

  • I/O throughput
  • Networking between the compute device and block storage device.

See Block Volume Performance in the Oracle Cloud Infrastructure documentation.

The table below describes how Big Data Service allocates block volume storage for nodes of different sizes.

What Amount
Initial volume allocation for master nodes and utility nodes 1 large volume
Volume allocation for additional block storage for master nodes and utility nodes 1 large volume
Initial volume allocation for worker nodes.
  • Storage: Less than 6 TB.

    Volume size: 500 GB The last volume may be smaller than 500 GB.

  • Storage: 6 TB to 48 TB.

    Volume size: Divide evenly into 12 volumes, each of which is at least 500 GB.

  • Storage: Greater than 48 TB.

    Volume size: Not allowed.

Volume allocation for additional block storage for worker nodes

The minimum number of volumes that can accommodate the storage size, with a maximum volume size of 4 TB per volume. (The last volume may be smaller than 4 TB.)

It's recommended that you use edge nodes for staging.

Understand cluster profiles

Cluster profiles enable you to create optimal clusters for a specific workload or technology. After creating a cluster with a specific cluster profile, additional Hadoop services can be added to the cluster.

Cluster profile types

Oracle Big Data Service allows creating cluster for numerous cluster profile types.

Cluster profile Components (Secure and Highly Available) Components
HADOOP_EXTENDED1 Hive, Spark, HDFS, Yarn, ZooKeeper, MapReduce2, Ambari Metrics, Ranger, Hue, Oozie, Tez Hive, Spark, HDFS, Yarn, ZooKeeper, MapReduce2, Ambari Metrics, Hue, Oozie, Tez
HADOOP HDFS, Yarn, ZooKeeper, MapReduce2, Ambari Metrics, Ranger, Hue HDFS, Yarn, ZooKeeper, MapReduce2, Ambari Metrics, Hue
HIVE Hive, HDFS, Yarn, ZooKeeper, MapReduce2, Ambari Metrics, Ranger, Hue, Tez Hive, HDFS, Yarn, ZooKeeper, MapReduce2, Ambari Metrics, Hue, Tez
SPARK Spark, Hive, HDFS, Yarn, ZooKeeper, MapReduce2, Ambari Metrics, Ranger, Hue 2 Spark, Hive, HDFS, Yarn, ZooKeeper, MapReduce2, Ambari Metrics, Hue 2
HBASE HBase, HDFS, Yarn, ZooKeeper, MapReduce2, Ambari Metrics, Ranger, Hue HBase, HDFS, Yarn, ZooKeeper, MapReduce2, Ambari Metrics, Hue
TRINO Trino, Hive, HDFS, ZooKeeper, Ambari Metrics, Ranger, Hue 2 Trino, Hive, HDFS, ZooKeeper, Ambari Metrics, Hue 2

1 HADOOP_EXTENDED consists of components that you created clusters before cluster profiles were available.

2 Only Hive metastore component from Hive service is available.

Apache Hadoop versions in cluster profiles

The table below lists the Hadoop component versions included in cluster profiles corresponding to ODH version.

ODH 1.x
Cluster profile Version
HADOOP_EXTENDED HDFS 3.1, Hive 3.1, Spark 3.0.2
HADOOP HDFS 3.1
HIVE Hive 3.1
SPARK Spark 3.0.2
HBASE HBase 2.2
TRINO Trino 360
ODH 2.x
Cluster profile Version
HADOOP_EXTENDED HDFS 3.3, Hive 3.1, Spark 3.2
HADOOP HDFS 3.3
HIVE Hive 3.1
SPARK Spark 3.2
HBASE HBase 2.2
TRINO Trino 389

Creating a Cluster

Create an Big Data Service cluster from the Oracle Cloud Console.

Before you can create a cluster, you must have:

The Cluster creation wizard asks you to provide information about your network and to make choices based on your network. To prepare for those questions, have the name of your network, its compartment, and its regional subnet name ready.

To create a cluster:
  1. Open the navigation menu and click Analytics and AI. Under Data Lake, click Big Data Service.
  2. In the left panel, under Compartment, select a compartment to host the cluster.
  3. Click Create Cluster.
  4. Enter the following information:
    • Cluster Name: Enter a name to identify the cluster.

    • Cluster Admin Password: Enter a string to be used as the cluster password. You need this password to sign into Apache Ambari or Cloudera Manager depending on your cluster version and to perform certain actions on the cluster through the Cloud Console.

    • Confirm Cluster Admin Password: Reenter the password.

    • Secure and Highly Available (HA): Check this box to make the cluster secure and highly available. A secure cluster has the full Hadoop security stack, including HDFS Transparent Encryption, Kerberos, and Apache Sentry. This setting can't be changed for the life of the cluster.

    • Kerberos realm name: This field appears when you select the Secure and Highly Available (HA) check box. The default value is BDSCLOUDSERVICE.ORACLE.COM. However, you can provide a different value. Typically, the realm name is the same as your DNS domain name except that the realm name is in uppercase. This convention helps differentiate problems with the Kerberos service from the problems with the DNS namespace, while keeping a name that is familiar.

      A valid Kerberos realm name must consist of 2-32 ASCII characters and must be a combination of uppercase letters, numbers, dashes (-), and dots (.). It must also start and end with uppercase letters. If you plan to integrate your BDS cluster with an existing Active Directory server, you must ensure that the Kerberos realm name of your BDS cluster is different from the DNS names of the Active Directory domains.

    • Cluster Version: Select a distribution and a version of the Hadoop distribution to use for your cluster. Choose one of the following:

      • Select ODH 2.0, ODH 1.0, or ODH 0.9 (Oracle Distribution including Apache Hadoop) to use Oracle's implementation of Hadoop.
      • Select a version of the Cloudera Distribution Including Apache Hadoop (CDH) software to use for this cluster. The listed versions are fully supported. See the Cloudera documentation for descriptions of the features in each release.
    • Cluster Profile: Select the cluster profile for the cluster (available for ODH 2.0 and ODH 1.0 versions only). See Understand cluster profiles for more information.
  5. In the Hadoop Nodes section of the page, configure the types, shapes, and numbers of the compute instances (servers) to host the master and worker nodes of the cluster. For information about the choices you can make, see Understand Instance Types and Shapes. Not all shapes are available by default, although you can request those not listed. See Increasing Service Limits.

    Enter details for Master/Utility Nodes:

    • Choose Instance Type: Click the Virtual Machine box or the Bare Metal box, to indicate what type of compute instances you want for the master nodes.

    • Choose Master/Utility Node Shape: Select the shape for the master and utility nodes. See Compute Shapes and see Service Limits for details about the available shapes.

    • Block Storage Size per Master/Utility Node (in GB): Enter the block storage size, in gigabytes (GB); for each master and utility node.

    • Number of Master & Utility Nodes: A high-availability (HA) cluster always has 4 master/utility nodes, and a non-HA cluster always has 2 master/utility nodes. Therefore, this read-only field shows 4 nodes for an HA cluster or 2 nodes for a non-HA cluster.

  6. Enter details for Worker Nodes
    • Choose Instance Type: Click the Virtual Machine box or the Bare Metal box, to indicate what kind of compute instances you want.

    • Choose Worker Node Shape: Select the shape for the worker nodes. See Compute Shapes and Big Data Service Default Limits for details about the available shapes.

    • Block Storage Size per Worker Node: Enter the block storage size, in gigabytes (GB), for each worker node.

    • Number of Worker Nodes: Enter the number of worker nodes for the cluster, with a minimum 3 nodes.

  7. In the Network Settings section, complete the network details for your cluster.
    • Cluster Private Network: Enter a CIDR block for the cluster private network that will be created for the cluster.

      The cluster private network is created in the Oracle tenancy (not your customer tenancy), and it's used exclusively for private communication among the nodes of the cluster. No other traffic travels over this network, it isn't accessible by outside hosts, and you can't modify it once it's created. All ports are open on this network.

      In the CIDR Block field, enter a CIDR block to assign the range of contiguous IP addresses available for this private network, or accept the default 10.0.0/16. This CIDR block range cannot overlap the CIDR block range in your customer network, discussed in the next step.

    • Customer Network: Enter information to add the cluster to your Virtual Cloud Network (VCN) and a regional subnet in that VCN.

      Choose VCN in <compartment>: Accept the current compartment, or click Change Compartment to select a different one. Then select the name of an existing VCN in that compartment to use for the cluster. The VCN must contain a regional subnet.

      Choose Regional Subnet in <compartment>: Choose a regional subnet to use for the cluster.

      Important: If you plan to make any of the IP addresses in the subnet public (to allow access to a node from the internet), you must select a public subnet for the cluster. For more information, see VCNs and Subnets.

    • Network Options
      Deploy Oracle-managed Service gateway and NAT gateway (Quick Start): Select this option to simplify your network configuration by allowing Oracle to provide and manage these communication gateways. When you select this option, a service gateway and a Network Address Translation (NAT) gateway are deployed for private use by the cluster. These gateways are created in the Oracle tenancy and can't be modified after the cluster is created.
      • A NAT gateway enables nodes without public IP addresses to initiate connections to and receive responses from the internet but not to receive inbound connections initiated from the internet. See NAT Gateway.
      • A service gateway enables nodes without public IP addresses to privately access Oracle services, without exposing the data to an internet gateway or a NAT gateway. See Service Gateway.

      Follow these guidelines:

      • Choose this option to give all nodes in the cluster private network full outbound access to the public internet. When you select this option, you won't be able to limit that access in any way (for example by restricting egress to only a few IP ranges).

        If you select this option, your cluster won't be able to use service gateways or NAT gateways in your customer network.

      • If you don't choose this option, you must create gateways in your customer VCN. When you do this, you can also create security rules to limit egress to specified IP ranges.

      • If you map the private IP addresses of the cluster nodes to public IP addresses, then a NAT gateway isn't needed. See Map a Private IP Address to a Public IP Address.

        Use the gateways in your selected Customer VCN (Customizable): Select this option to permit the cluster to use the gateways in your customer VCN. You must create and configure these resources yourself.
        Note

        If you create your network by using one of the network creation wizards in the console, gateways are created for you, but you may have to configure them further to suit your needs. See Virtual Networking Quickstart. For complete information about setting up a gateway, see Service Gateway.

    • Encryption: Select one of the following:
      • Encrypt using Oracle-managed keys: Select to leave all encryption related matters to Oracle.

      • Encrypt using customer-managed keys: Select if you have access to and want to use a valid customer-managed encryption key. Select the following:

        • Choose Vault in <compartment>: Accept the current compartment, or click Change Compartment to select a different compartment. Then select the name of an existing vault in that compartment.
        • Choose Master encryption key in <compartment>: Accept the current compartment, or click Change Compartment to select a different compartment. Then select an existing master encryption key in that compartment.

      For additional information on creating and managing vaults, see To create a new vault and Managing Vaults. For additional information on creating and managing master encryption keys, see To create a new master encryption key and Managing Keys.

  8. Under Additional Options, do the following:
    • SSH Key: Enter an SSH public key in any of the following ways:

      • Select Choose SSH Key File, then either

        • Drag and drop a public SSH key file into the box,

        • or click select one... and navigate to and choose a public SSH key file from your local file system.

      • Select Paste SSH Key and paste the contents from a public SSH key file into the box.

    • Bootstrap script URL: Enter a publicly accessible URL for the bootstrap script. The script runs on all the cluster nodes after a cluster is created, when the shape of a cluster changes, or when you add or remove nodes from a cluster. You can use this script to install, configure , and manage custom components in a cluster.
  9. Tags: Use tags to help you organize and list resources. Enter tags as described in Tagging Overview.
  10. Click Create Cluster.

Creating a Secure and Highly Available Cluster

You make a cluster secure and highly available by setting an option when you create it. A secure cluster has the full Hadoop security stack, including HDFS Transparent Encryption, plus Kerberos and Apache Sentry.

Make a new cluster secure and highly available by selecting a Secure and Highly Available (HA) under General Settings in the Create Cluster wizard. You can't make an existing cluster secure and highly available if it wasn't created with those features, and you can't remove those features from an existing cluster. See Creating a Cluster for instructions.

Define Security Rules

An administrator must configure security rules to control network traffic to and from Big Data Service resources.

Background

In Oracle Cloud Infrastructure, two kinds of virtual firewalls are available for controlling traffic to and from your cloud resources. Security lists include security rules that apply to an entire subnet. Network security groups include security rules that apply to a defined set of resources that are organized into groups. Network security groups allow finer-grained control, while security lists are easier to set up and maintain.

Security lists and network security groups both include security rules. A security rule allows a particular type of traffic in or out of a Virtual Network Interface Card (VNIC).

Note

A VNIC is a networking component that enables a networked resource such as an instance (a node in Big Data Service) to connect to a virtual cloud network (VCN). The VNIC determines how the instance connects with endpoints inside and outside the VCN. Each VNIC resides in a subnet in a VCN. A security list defines a set of security rules that apply to all the VNICs in a subnet. A network security group defines a set of security rules that apply to a group of VNICs that you define.

It's important to understand the role of VNICs in your network architecture, but for the purposes of this documentation, it's usually sufficient to refer how security rules work in VCNs and subnets.

For more information see Security Rules.

Creating Security Rules in Security Lists

Typically, Big Data Service uses security lists. That means that you create security rules for a subnet, and any cluster in that subnet is subject to those rules. The following instructions tell how to create security rules in a security list defined for the subnet used by your cluster.

A security list can define both ingress rules (for incoming traffic) and egress rules (for outgoing traffic).

Each security rule specifies:
  • Direction (ingress or egress)
  • Stateful or stateless
  • Source type and source (ingress rules only)

For complete documentation about security rules, see Parts of a Security Rule.

The following sections contain specific details about creating ingress and egress rules for Big Data Service clusters.

Create Ingress Rules (and Open Ports)

You must open certain ports on Big Data Service clusters to allow access to services like Apache Ambari, Cloudera Manager, and Hue. Configure these ports in the security ingress rules that apply to a cluster.

To set ingress rules in a security list:
  1. Open the navigation menu and click Networking. Then click Virtual Cloud Networks.
  2. Under Compartment in the panel on the left, select the compartment that hosts your VCN.
  3. Click the name of the VCN used by the cluster. (The VCN that was associated with the cluster when the cluster was created.)
  4. Under Subnets, click the name of the subnet used by the cluster. (The subnet was associated with the cluster when the cluster was created.)
  5. Under Security Lists, click the name of a security list defined for the subnet. If you choose the Default security list for the subnet, it may already have some rules defined. For example, it may have a rule that allows incoming traffic on port 22, for Secure Shell (SSH) access.
  6. If the ingress rules aren't displayed, click Ingress Rules on the left side of the page, under Resources.
  7. Click the Add Ingress Rules button to configure the ingress rules for the subnet.
  8. In the Add Ingress Rules dialog box, set the following options to open port 22 for SSH access (if it isn't already open):
    • Stateless - Leave this box unchecked. This makes the rule stateful, which means that any response to the incoming traffic is allowed back to the originating host, regardless of any egress rules applicable to the instance.
    • Source Type - Select CIDR.
    • Source CIDR - Enter 0.0.0.0/0, which indicates that traffic from all sources on the internet is allowed.
    • IP Protocol - Select TCP.
    • Source Port Range - Accept the default All.
    • Destination Port Range - Enter 22, to allow access via SSH.
    • Description - Add an optional description.
  9. At the bottom of the dialog box, click +Another Ingress Rule, and enter the values for another rule. Do this for as many times as necessary, to create all the rules you need, and then click Add Ingress Rules.
    For a typical set of ingress rules for a cluster, create additional rules using the same values as above, but with different Destination Port Ranges:
    • Apache Ambari - port 7183
    • Cloudera Manager - port 7183
    • Hue - port 8888
    • Web Resource Manager - port 8090
    • Spark History Server - port 18088
    • Cloud SQL (if Cloud SQL is installed) - port 1521

    The ingress rules will look similar to the following:

    Stateless Source IP Protocol Source Port Range Destination Port Range Type and Code Allows Description
    No 0.0.0.0/0 TCP All 22 . TCP traffic for ports: 22 SSH ... SSH
    No 0.0.0.0/0 ICMP . . 3,4 ICMP traffic for: 3, 4 ...
    No 0.0.0.0/0 ICMP . . 3 ICMP traffic for: 3 ...
    No 0.0.0.0/0 TCP All 7183 . TCP traffic for ports: 7183 CM
    No 0.0.0.0/0 TCP All 8888 . TCP traffic for ports: 8888 Hue
    No 0.0.0.0/0 TCP All 8090 . TCP traffic for ports: 8090 Web Resource Manager
    No 0.0.0.0/0 TCP All 180888 . TCP traffic for ports: 180888 Spark History Server
    No 0.0.0.0/0 TCP All 1521 . TCP traffic for ports: 1521 Cloud SQL (query server)
Create Egress Rules

When creating a cluster, you have the option to use a NAT gateway or not. Whether or not you choose that option affects how you can control outbound traffic.

  • If you choose the NAT gateway option when creating a cluster, all nodes will have full outbound access to the public internet. You won't be able to limit that access in any way (for example by restricting egress to only a few IP ranges).

  • If you choose not to create a NAT gateway when creating a cluster, you can create a NAT gateway on the VCN you're using to access the cluster. You can also edit policies on this NAT gateway to limit egress to specified IP ranges.

  • If you map the VM IPs onto public IPs then a NAT gateway isn't needed.

Establishing Connections to Nodes with Private IP Addresses

By default, cluster nodes are assigned private IP addresses and are therefore not publicly available on the internet. You can make them available in any of the ways described in the following topics:

Map a Private IP Address to a Public IP Address

Big Data Service nodes are by default assigned private IP addresses, which aren't accessible from the public internet. One way to make a node accessible from the internet is to map a node's private IP address to a public IP address.

The instructions below use the Oracle Cloud Infrastructure Cloud Shell, which is a web browser-based terminal accessible from the Oracle Cloud Console. You'll gather some information about your network and your cluster nodes, and then you'll pass that information to commands in the shell. To perform this task, you must have a cluster running in a VCN in your tenancy, and that cluster must have a regional, public subnet.

Required IAM Privileges for Mapping Private to Public IP Address

You must have appropriate Oracle Infrastructure Identity and Access Management (IAM privileges) to map private to public IP addresses.

The tenancy administrator or a delegated administrator with the appropriate privileges must create a policy according to the following guidelines.

Group

The policy can assign privileges to any Big Data Service group, to give members of that group the rights to map IP addresses.

Verbs
The policy must contain policy statements with the following IAM verbs:
  • vnic_read
  • private_ip_read
  • public_ip_read
  • public_ip_delete
  • public_ip_create
  • public_ip_update
  • private_ip_assign_public_ip
  • private_ip_unassign_public_ip
  • public_ip_assign_private_ip
  • public_ip_unassign_private_ip
Resource

The policy must specify the tenancy or the <compartment_name> of the compartment containing the subnet used for the IP addresses.

Example
allow group bds_net_admins to vnic_read in tenancy
allow group bds_net_admins to private_ip_read in tenancy
allow group bds_net_admins to public_ip_read in tenancy
allow group bds_net_admins to public_ip_delete in tenancy
allow group bds_net_admins to public_ip_create in tenancy 
allow group bds_net_admins to public_ip_update in tenancy 
allow group bds_net_admins to private_ip_assign_public_ip in tenancy 
allow group bds_net_admins to private_ip_unassign_public_ip in tenancy 
allow group bds_net_admins to public_ip_assign_private_ip in tenancy
allow group bds_net_admins to public_ip_unassign_private_ip in tenancy
Gather Information About the Cluster
  1. Open the navigation menu and click Analytics and AI. Under Data Lake, click Big Data Service.
  2. On the Clusters page, click the name of your cluster, for example mycluster.
  3. On the Cluster Information tab of the Cluster Details page, under Customer Network Information, click the Copy link next to Subnet OCID. Then paste that OCID to an editor or a file, so you can retrieve it later in this process.
  4. On the same page, under List of Cluster Nodes, find the node you want to map.
    Node names are constructed like clusterndp where
    • cluster is the first 7 letters of the cluster name.
    • nd is the type of node: mn = master node, un = utility node, and wn = worker node.
    • p is the position in the list of the type of node identified by nd, where 0 is the first, 1 is the second, etc.

    For example, the name myclustun0 refers to the first (0) utility node (un) in a cluster named mycluster.

  5. In the IP Address column, find the private IP address for that node, for example, 192.0.2.1. Save the address so you can retrieve it later.
Map the Private IP Address to a Public IP Address
  1. In the Cloud Console, click the Cloud Shell Cloud Shell icon at the top of the page. It may take a few moments to connect and authenticate you.

  2. At the prompt, enter the following.

    export DISPLAY_NAME=<display-name>

    export SUBNET_OCID=<subnet-ocid>

    export PRIVATE_IP=<ip-address>

    oci network public-ip create --display-name $DISPLAY_NAME --compartment-id `oci network private-ip list --subnet-id $SUBNET_OCID --ip-address $PRIVATE_IP | jq -r '.data[] | ."compartment-id"'` --lifetime "RESERVED" --private-ip-id `oci network private-ip list --subnet-id $SUBNET_OCID --ip-address $PRIVATE_IP | jq -r '.data[] | ."id"'`

    The three export statements above set variables that are used in the oci network command that follows. The variables are:

    • <display-name> (optional) is a "friendly name" that will be attached to the reserved public IP address. This name is not pre-existing. It's created when running this command.

      For convenience, you might want to use the name of the node whose private IP address you're mapping, for example myclusun0, which is the name of the first utility node in a cluster named mycluster.

    • <subnet-ocid> is the OCID of the customer public subnet used by the cluster for example, ocid1.subnet.oc1.iad....

    • <ip-address> is the private IP address assigned to the node you want to map, for example, 192.0.2.1.

    Enter the command beginning with oci network public-ip create --compartment-id... exactly as it's shown above, with no breaks.

    For example:

    • $ export DISPLAY_NAME="myclustun0"
    • $ export SUBNET_OCID="ocid1.subnet.oc1.…"
    • $ export PRIVATE_IP="192.0.2.1"
    • $ oci network public-ip create --display-name $DISPLAY_NAME --compartment-id `oci network private-ip list --subnet-id $SUBNET_OCID --ip-address $PRIVATE_IP | jq -r '.data[] | ."compartment-id"'` --lifetime "RESERVED" - private-ip-id `oci network private-ip list --subnet-id $SUBNET_OCID --ip-address $PRIVATE_IP | jq -r '.data[] | ."id"'`
    The output returned is:
    • { "data": {
    •     "assigned-entity-id": "ocid1.privateip.oc1...",
    •     "assigned-entity-type": "PRIVATE_IP",
    •     "availability-domain": null,
    •     "compartment-id": "ocid1.compartment.oc1...",
    •     "defined-tags": {},
    •     "display-name": "publicip...",
    •     "freeform-tags": {},
    •     "id": "ocid1.publicip.oc1....",
    •     "ip-address": "203.0.113.1",
    •     "lifecycle-state": "ASSIGNED",
    •     "lifetime": "RESERVED",
    •     "private-ip-id": "ocid1.privateip....",
    •     "scope": "REGION",
    •     "time-created": "2020-04-13..."
    •    },
    •    "etag": "1234abcd"
    • }
  3. In the output returned, find the value for ip-address. In the above example, it's 203.0.113.1. That is the new reserved public IP address that is mapped to the private IP address for the node.

    To see the reserved public IP address in the console, click the navigation menu navigation menu. Under Networking, click Virtual Cloud Networks. Then, in the navigation list on the left, under Networking, click IP Management. The new reserved public IP address appears in the Reserved Public IP Addresses list. If you supplied a display name in the command you ran, above, that name will appear in the Name column. Otherwise, a name like publicipnnnnnnnnn is generated.

Delete a Public IP Address

To delete a public IP, you can use the following:

oci network public-ip delete --public-ip-id ocid1.publicip.oc1....

The value for --public-ip-id is shown in output returned by the previous command, as shown above: "id": "ocid1.publicip.oc1....",.

Alternatively, you can go to the Networking Reserved Public IP Addresses page in the Cloud Console and delete reserved public IPs there.

Open Ports to Make Services Available

Making the node publicly available isn't enough to make a service like Apache Ambari or Cloudera Manager available from the internet. You must also open the port for the service by adding an ingress rule to a security list. See Define Security Rules.

Use a Bastion Host to Connect to Your Service

You can use a bastion host to provide access to the a cluster's private network from the public internet.

A bastion host is a compute instance that serves as the public entry point for accessing a private network from external networks like the internet. Traffic must flow through the bastion host to access the private network, and you can set up security mechanisms on the bastion to handle that traffic.

Use Oracle Cloud Infrastructure Site-to-Site VPN to Connect to Your Service

Site-to-Site VPN provides a site-to-site IPSec VPN between your on-premises network and your virtual cloud network (VCN). The IPSec protocol suite encrypts IP traffic before the packets are transferred from the source to the destination and decrypts the traffic when it arrives.

For details for connecting to Big Data Service with VPN see Site-to-Site VPN.

Use Oracle Cloud Infrastructure FastConnect to Connect to Your Service

Use FastConnect to access public services in Oracle Cloud Infrastructure without using the internet, for example, access to Object Storage, or the Oracle Cloud Console and APIs. Without FastConnect, the traffic destined for public IP addresses would be routed over the internet. With FastConnect, that traffic goes over your private physical connection.

For details for connecting Big Data Service with Oracle Cloud Infrastructure FastConnect see FastConnect Overview.

Adding Worker and Edge Nodes to a Cluster

When you add worker, compute only worker, or edge nodes to a cluster, you expand both compute and storage. The new nodes use the same instance shape and amount of block storage as the existing worker nodes in the cluster.

To add nodes to a cluster:
  1. Open the navigation menu and click Analytics and AI. Under Data Lake, click Big Data Service.
  2. In the left panel, under Compartment, select the compartment that hosts your cluster.
  3. In the list of clusters, find the cluster to which you want to add worker nodes.
  4. On the cluster details page, click the Add Nodes button.
  5. In the Add nodes panel that appears, enter the following details:
    • Node type: Select the node type. The available options are as follows:
      • Worker: A cluster must have at least three worker nodes.
      • Compute only worker: You cannot add a compute only worker node while creating a cluster. Therefore, when you add a compute only worker node for the first time, you can update the shape and block storage size. After it is updated, these fields become read-only.
      • Edge: You cannot add an edge node while creating a cluster. Therefore, when you add an edge node for the first time, you can update the shape and block storage size. After it is updated, these fields become read-only.
    • Node shape: This read-only field displays the shape used for the existing worker nodes. This shape is used for all the nodes you add. For information about the shapes, see Understand Instance Types and Shapes.
    • Block storage per node : This read-only field displays the block storage used for the existing worker nodes. The same amount of storage is used for all the nodes you add.
    • Number of worker nodes: Enter the number of worker nodes or compute only worker nodes to be added to the cluster. A cluster can have from 3 to 256 worker nodes. An ODH cluster can have from 0 to 256 compute worker nodes.
    • Cluster admin password: Enter the administration password for the cluster.
  6. Click Add.

Adding Block Storage to Worker Nodes

Block storage is a network-attached storage volume that you can use like a regular hard drive. You can attach extra block storage to the worker nodes of a cluster.

Note

Nodes in a cluster can have remote, network-attached, block storage or local, direct-attached, Non-Volatile Memory Express (NVMe) storage. Remote block storage is flexible and economical, while local NVMe storage provides the highest performance. The default storage type is determined when the cluster is created, based on the shape chosen for the cluster. The high-performance bare metal nodes and dense I/O virtual machine nodes are created with NVMe storage. Other kinds of virtual machine nodes are created with block storage.

You can attach extra storage to any cluster. You can't remove storage.

To add a block volume to the cluster:

  1. Open the navigation menu and click Analytics and AI. Under Data Lake, click Big Data Service.
  2. In the left panel, under Compartment, select the compartment that hosts your cluster.
  3. In the list of clusters, find the cluster to which you want to add block storage.
  4. Click the actions menu, and select Add Block Storage.
  5. In the Add Block Storage dialog box, enter information, as follows:
    • Additional Block Storage per Node (in GB) - Enter a number to indicate how many gigabytes of block storage to add, between 150GB and 32TB, in increments of 50GB.
    • Cluster Admin Password - Enter the administration password for the cluster.
  6. Click Add.

Adding Cloud SQL to a Cluster

You can add Oracle Cloud SQL to a cluster so you can use SQL to query your big data sources.

Note

Cloud SQL is not included with Big Data Service. You must pay an extra fee for use of Cloud SQL.

When you add Cloud SQL support to a cluster, a query server node is added and big data cell servers are created on all worker nodes.

For information about using Cloud SQL with Big Data Service see Using Cloud SQL with Big Data.
Note

For clusters with Oracle Distribution including Apache Hadoop, Cloud SQL is only supported for non-HA clusters.
  1. Open the navigation menu and click Analytics and AI. Under Data Lake, click Big Data Service.
  2. In the left panel, under Compartment, select the compartment that hosts your cluster.
  3. In the list of clusters, find the cluster to which you want to add Cloud SQL.
  4. Click the actions menu, and select Add Cloud SQL.
  5. Enter details in the Add Cloud SQL dialog box, as follows:
    • Query Server Node Shape: Select a shape to be used for the query server node. For information about the available shapes, see Understand Instance Types and Shapes.
    • Query Server Node Block Storage (in GB): Enter a number to indicate how many gigabytes of block storage to add, between 1TB and 48TB, in increments of 50GB.
    • Cluster Admin Password: Enter your cluster administration password.
  6. Click Add.

Configuring an External Metastore

Data Catalog provides a highly available and scalable metastore for Hive implementations.

You can configure this Data Catalog metastore as an external metastore for your Big Data Service cluster.

Creating an external metastore

You can create one external metastore for your Big Data Service cluster.

Prerequisites:

To create an external metastore configuration for Big Data Service, you must have the permissions and policies to access and manage a valid Data Catalog metastore.

Note

When you create an external metastore for the Big Data Service cluster, the local metastore for the cluster is made inactive and the new external Data Catalog metastore is activated.
  1. Open the navigation menu and click Analytics and AI. Under Data Lake, click Big Data Service.
  2. In the left panel, under Compartment, select the compartment that hosts your cluster.
  3. In the list of clusters, click the name of your cluster.
  4. In the left panel, under Resources click Metastore Configurations.
  5. Click Create external metastore configuration.
  6. In the Create external metastore configuration panel, review the Metastore name of the external Data Catalog metastore that is available in the specified compartment for you to configure.
  7. Select the API key alias you specified when you created the Object Storage API key.
  8. Enter the API key passphrase. You specified this passphrase when you created the API key.
  9. Enter the Cluster admin password for the cluster. You specified this password when you created the cluster.
  10. Click Create.

After successful creation of the external metastore for the Big Data Service cluster, the local metastore for the cluster is made inactive and the new external Data Catalog metastore is activated.

Viewing metastore details
  1. Open the navigation menu and click Analytics and AI. Under Data Lake, click Big Data Service.
  2. In the left panel, under Compartment, select the compartment that hosts your cluster.
  3. In the list of clusters, click the name of your cluster.
  4. In the left panel, under Resources click Metastore Configurations.
  5. The default local metastore is listed. An external metastore is also listed if you have created it. Click the local or external metastore name to view additional details.
From the actions menu for each metastore, you can test the metastore configuration, activate a metastore, update API key, or delete the metastore. You can not delete the local metastore.
Testing the local or external metastore configuration
  1. Open the navigation menu and click Analytics and AI. Under Data Lake, click Big Data Service.
  2. In the left panel, under Compartment, select the compartment that hosts your cluster.
  3. In the list of clusters, click the name of your cluster.
  4. In the left panel, under Resources click Metastore Configurations.
  5. Click the actions menu for the metastore configuration you want to test and select Test configuration.
  6. In the Test configuration dialog, enter the Cluster admin password for your cluster and click Test configuration.
  7. Review the connection status and when done, click Close.
Updating API key for an external metastore
  1. Open the navigation menu and click Analytics and AI. Under Data Lake, click Big Data Service.
  2. In the left panel, under Compartment, select the compartment that hosts your cluster.
  3. In the list of clusters, click the name of your cluster.
  4. In the left panel, under Resources click Metastore Configurations.
  5. Click the actions menu for the external metastore configuration and select Update API key.
  6. In the Update API key panel, select the new API key.
  7. Enter the API key passphrase for the new API key.
  8. Enter the Cluster admin password for the cluster.
  9. Click Update.
Activating the local metastore

When you activate the local metastore, the external metastore becomes inactive.

  1. Open the navigation menu and click Analytics and AI. Under Data Lake, click Big Data Service.
  2. In the left panel, under Compartment, select the compartment that hosts your cluster.
  3. In the list of clusters, click the name of your cluster.
  4. In the left panel, under Resources click Metastore Configurations.
  5. Click the actions menu for the local metastore configuration and select Activate.
  6. In the Activate local metastore dialog, enter the Cluster admin password for the cluster.
  7. Click Activate.
When you activate the local metastore, the external metastore is automatically made inactive.
Deleting an external metastore

You can delete an inactive external metastore. The external metastore becomes inactive when you activate the local metastore.

Note

You can not delete the default local metastore for your cluster.
  1. Open the navigation menu and click Analytics and AI. Under Data Lake, click Big Data Service.
  2. In the left panel, under Compartment, select the compartment that hosts your cluster.
  3. In the list of clusters, click the name of your cluster.
  4. In the left panel, under Resources click Metastore Configurations.
  5. Click the actions menu for the inactive external metastore configuration and select Delete. You can not delete an active external metastore. To make the external metastore inactive, activate the local metastore.

Install Ext JS to Support Oozie UI

Oozie UI depends on the Ext JS JavaScript framework, and you must manually install Ext JS on your cluster.

Note

These steps apply for clusters using Cloudera Distribution including Apache Hadoop.

Ext JS is a JavaScript framework for building data-intensive, cross-platform web and mobile applications. Big Data Service requires Ext JS 2.2 placed in the Oozie webapp directory to make Oozie UI work. Ext JS is licensed under GPLv2.

To install Ext JS 2.2:

  1. Download ext2.2.zip from http://archive.cloudera.com/gplextras/misc/ext-2.2.zip.
  2. Copy the zip file to a cluster node where Oozie server is installed, and unzip it there.
  3. Place the unzipped directory, with name ext-2.2, in /usr/lib/oozie/embedded-oozie-server/webapp/ and verify that ownership of the directory is with oozie:hadoop.

Modifying a Cluster

Stopping and Starting Clusters

Beginning with BDS 3.0.18, you can stop and start clusters.

Stopping a cluster decreases the service cost of the cluster to 25% of the current service cost for the cluster. When workload requests increase, you can restart the cluster.

Stopping a Cluster

You can stop clusters when your workload requests decrease. This reduces the service cost of the cluster to 25% of the current service cost.

  1. Open the navigation menu and click Analytics and AI. Under Data Lake, click Big Data Service.
  2. In the left panel, under Compartment, select the compartment that hosts your cluster.
  3. Stop a cluster one of two ways.
    • From the cluster page, click the Action menu for cluster you want to stop, and then click Stop.
    • From the cluster details page:
      1. To access the cluster details page, double-click the cluster on the cluster page.
      2. Click Stop.
  4. Enter the Cluster admin password.
  5. To force the cluster to stop even when applications are running, select Force stop running applications on cluster.
  6. Click Stop.
    The status of the cluster changes to Inactive.

Starting a Cluster

You can start clusters when your workload requests increase. The service cost for the cluster returns to normal.

  1. Open the navigation menu and click Analytics and AI. Under Data Lake, click Big Data Service.
  2. In the left panel, under Compartment, select the compartment that hosts your cluster.
  3. Start a cluster one of two ways.
    • From the cluster page, click the Action menu for cluster you want to start, and then click Start.
    • From the cluster details page:
      1. To access the cluster details page, double-click the cluster on the cluster page.
      2. Click Start.
  4. Enter the Cluster admin password.
  5. Click Start.
    The cluster status changes to Active.

Renaming a Cluster

You can change the name of any cluster.

To rename the cluster:
  1. Open the navigation menu and click Analytics and AI. Under Data Lake, click Big Data Service.
  2. In the left panel, under Compartment, select the compartment that hosts your cluster.
  3. Click the action menu for cluster you want to rename, and select Rename Cluster.
  4. In the Rename Cluster dialog box, enter a new name for the cluster and click Rename.

Changing the Shape of Cluster Nodes

You can change the shapes of the nodes of a cluster after the cluster is created, with the following restrictions:

  • All the master nodes and utility nodes in a cluster must use the same shape, and all worker nodes must use the same shape. However, the master nodes and utility nodes can use a different shape than the worker nodes. Therefore, when you change the shapes of nodes, you can change the shape of all the master and utility nodes together, and you can change the shape of all the worker nodes together.
  • You can change the shapes only of nodes using standard shapes, and you can only change them to other standard shapes. For information about standard shapes, see Compute Shapes. For information about shapes supported in Big Data Service, see Plan Your Cluster Layout, Shape, and Storage.

To change the shape of a cluster:

  1. Open the navigation menu and click Analytics and AI. Under Data Lake, click Big Data Service.
  2. In the left panel, under Compartment, select the compartment that hosts your cluster.
  3. In the list of clusters, click the name of your cluster.
  4. Click Change Shape.
  5. In the Change Shape panel, do the following:
    • Choose node type - The available options are as follows:
      • Master/utility: Select this option to change the shape of all the master and utility nodes
      • Worker: Select this option to change the shape of all the worker nodes
      • Compute only worker: Select this option to change the shape of compute only worker nodes.
      • Edge: Select this option to change the shape of edge nodes.
      • Cloud SQL: Select this option to change the shape of all the Cloud SQL nodes, if installed
    • Existing Shape - This read-only field shows the current shape of the nodes of the type you selected for Choose your node type, above.
    • New Shape - Select a new shape for the nodes of the selected type.
    • Cluster Admin Password - Enter the admin password for the cluster. (The password was assigned when the cluster was created.)
  6. Click Change Shape.

Autoscaling a Cluster

You can create an autoscale configuration for a cluster so that the compute shapes and numbers of the worker nodes are automatically increased or decreased, based on the CPU utilization thresholds.

Autoscaling allows you to maintain optimum performance of your cluster, while keeping costs as low as possible. Autoscaling monitors your CPU utilization and automatically adjusts the CPU capacity, based on the configuration parameters you set.

Note

When a cluster is autoscaled, the new details should be reflected in Apache Ambari or Cloudera Manager. To register that change with Apache Ambari or Cloudera Manager, a new cluster admin password is created when you create an autoscale configuration. The password is deleted when the autoscale configuration is deleted.
How Autoscaling Works

The autoscale feature collects data about the CPU utilization of worker or compute only worker nodes in a cluster. Two autoscale trigger types are available:

  • Metrics: This configuration includes the parameters for scaling up (switching to the next larger compute shape) and scaling down (switching to the next smaller compute shape) or scaling out (adding more nodes to the cluster) and scaling in (removing nodes from the cluster). A scale-up or scale-out configuration specifies a duration and a percentage, so that when the average CPU usage exceeds the specified percentage for the specified duration, the node is scaled up or out. A scale-down or scale-in configuration specifies a duration and a percentage, so that when the average CPU usage falls below the specified percentage for the specified duration, the node is scaled down or in.

    The average usage is based on the entire duration specified in the configuration. That is, the autoscale action is triggered at the end of the specified duration. For example, if the scale-up configuration is set to 60% for 6 hours, then the average CPU usage for the entire six hours must exceed 60% for the six-hour duration. The usage might fall below or rise above 60% for brief periods in that six-hour window, but the scale up action is triggered only after the data for the entire six hours is evaluated and averaged, and that average exceeds the percentage specified in the configuration.

    If you want the cluster to autoscale more frequently, based on more frequent fluctuations in CPU activity, use shorter duration values. Legal values for scaling durations are 5–60 minutes or 1–24 hours. Enter hours as units of 60 minutes. For example, 60, 120, 180, 240, etc., to 1440 minutes.

    Autoscale durations are mapped to Oracle Cloud Infrastructure Monitoring Query Language (MQL) interval values, where the ranges of values allowed for interval are 1m-60m, 1h-24h, and 1d. (Notice that while the minimum MQL interval is one minute, the minimum Big Data Service interval is five minutes.) See the "Interval Query Component" section in Monitoring Query Language (MQL) Reference.

    Autoscale takes advantage of Oracle Cloud Infrastructure alarms, and the autoscale duration value is also used as the notification interval for the autoscale alarm. (See Managing Alarms.) If the conditions for an autoscale action are still in effect after another interval, then the alarm will trigger another autoscale.

  • Schedule: Oracle supports two types of schedule-based policies, schedule-based vertical scaling and schedule-based horizontal scaling.

    As part of schedule-based vertical scaling, you specify the target shape and the shape configuration (OCPU count and memory size).

    As part of schedule-based horizontal scaling, you specify the target node count.

    Note

    All scheduled-based triggers/conditions associated with a cluster must be more than four hours apart.

    Note

    Scheduled-based conditions are snoozed for 15 minutes (up to a maximum of two hours), if the cluster's lifecycle state is not active when the trigger fires.

If you use a flex shape with either type of autoscale configuration, you can add or remove the exact number of OCPUs and control the memory usage during each autoscale operation. With a flex shape, you can also set minimum and maximum boundaries. See Supported Node Shapes in the Oracle Cloud Infrastructure documentation for the available shapes.

Oracle recommends that you constantly tune your autoscale values to meet your needs. See the recommendations for tuning alarms in the "Routinely Tune Your Alarms" section in Best Practices for Your Alarms.

Prerequisites
Quota

Your tenancy must have a quota that allows you to scale up or scale out worker or compute only worker nodes. If not, the autoscaling operation will fail. See Viewing Your Service Limits, Quotas, and Usage.

Network

When your cluster was created, one of the following options was selected:

  • Deploy Oracle-managed Service gateway and NAT gateway (Quick Start)

    If the cluster was created with this option selected, you can configure and use autoscale.

  • Use the gateways in your selected Customer VCN (Customizable)

    If the cluster was created with this option selected:

Autoscaling Types

You can autoscale a cluster horizontally or vertically when specified metric thresholds are exceeded.

Horizontal scaling adds or removes nodes to your cluster. Vertical scaling changes the shape of a node in your cluster.

For example in case of vertical scaling, when thresholds are met the shapes of all the worker nodes in the cluster are automatically scaled up to the next higher VM.Standard shape, scaled down to the next lower VM.Standard shape, or for Flex shapes to the configured OCPU and memory values.

In case of horizontal scaling, when thresholds are met the number worker nodes in the cluster are automatically scaled out or scaled in depending on the configured rules.

ODH clusters support both vertical and horizontal scaling. Horizontal scaling applies only to compute only worker nodes.

CDH clusters support only vertical scaling.

Create an Autoscale Configuration

You can have one autoscale configuration per supported node type. Therefore, on a cluster with both worker and compute only worker nodes, you can have up to two autoscale policies.

To create an autoscale configuration:
  1. Open the navigation menu and click Analytics and AI. Under Data Lake, click Big Data Service.
  2. In the left panel, under Compartment, select the compartment that hosts your cluster.
  3. In the list of clusters, click the name of your cluster.
  4. In the left panel, under Resources click Autoscale Configurations.
  5. Click Create Autoscale Configuration.
  6. In the Create Autoscale Configuration panel, enter the details as described in the following table:
    Field Description
    Autoscale configuration name Enter a name for this configuration.
    Cluster admin password Enter the admin password for the cluster. (You assign an admin password when you create a cluster.)
    Node type Select the type of node. The available options are as follows:
    • Worker: Select this option to add block storage and compute to your cluster.
    • Compute only worker: Select this option to add only compute to your cluster.
    Autoscale type The available options are as follows:
    • Horizontal: Select this option to add or remove nodes from the cluster. This option is available only if you select Compute only worker node type.
    • Vertical: Select this option to change the shape of the cluster nodes.
    Trigger type Initiates the autoscale when the specified metrics are met or when they exceed the threshold.
    Performance Metrics Measures the average CPU utilization of the cluster nodes over a period of time.
    Scale-up rule This field appears when you select the Vertical autoscale type. The scale-up rule sets the conditions for scaling up the cluster (use a larger compute shape). Enter the following details:
    • Average CPU threshold percentage: Set a percentage of average CPU utilization, so that when the CPU operates at that average percentage (or higher) for the minimum duration in minutes, the cluster is scaled up.
    • Minimum duration in minutes: Set a duration, so that when the average CPU threshold percentage operates for that duration, the cluster is scaled up.
    If you selected a Flex shape, then also specify the following details:
    • Maximum limits: Enter the maximum number of OCPUs and memory the cluster can have.
    • Step size: Specify the number of OCPUs and memory that should be added to the cluster each time the autoscale is triggered.
    For example, if the scale-up rule is:
    • Threshold percentage = 80%
    • Minimum duration = 30 minutes
    and the shapes of the worker nodes are VM.Standard2.4, when the CPU utilization averages 80% or more for 30 minutes, the worker nodes are scaled up to VM.Standard2.8. For Flex shapes, the shape of worker nodes changes depending on the OCPU and memory specified.
    Scale-down rule This field appears when you select the Vertical autoscale type. The scale-down rule sets the conditions for scaling down the cluster (use a smaller compute shape). Enter the following details:
    • Average CPU threshold percentage: Set a percentage of average CPU utilization, so that when the CPU operates at that average percentage (or lower) for the minimum duration in minutes, the cluster is scaled down.
    • Minimum duration in minutes: Set a duration, so that when the average CPU threshold percentage operates for that duration, the cluster is scaled down.
    If you selected a Flex shape, then also specify the following details:
    • Minimum limits: Enter the minimum number of OCPUs and memory the cluster can have.
    • Step size: Specify the number of OCPUs and memory that should be removed from the cluster each time the autoscale is triggered.
    For example, if the scale-down rule is:
    • Threshold percentage = 20%
    • Minimum duration = 30 minutes
    and the shapes of the worker nodes are VM.Standard2.8, when the CPU utilization averages 20% or less for 30 minutes, the worker nodes are scaled down to VM.Standard2.4. For Flex shapes, the shape of worker nodes changes depending on the OCPU and memory specified.
    Scale-out rule This field appears when you select the Horizontal autoscale type.

    Horizontal scaling applies only to compute only worker nodes.

    The scale-out rule sets the conditions for scaling out the cluster. Enter the following details:
    • Average CPU threshold percentage: Set a percentage of average CPU utilization, so that when the CPU operates at that average percentage (or higher) for the minimum duration in minutes, the cluster is scaled out.
    • Minimum duration in minutes: Set a duration, so that when the average CPU threshold percentage operates for that duration, the cluster is scaled out.
    • Maximum number of nodes: Enter the maximum number of nodes that the cluster can have. When autoscale is triggered, the number of nodes specified in Step size are added to the cluster if the number of nodes in the cluster after addition is lesser than or equal to the number of nodes specified in this field.
    • Step size: The number of nodes to add to the cluster when autoscale is triggered.
    Scale-in rule This field appears when you select the Horizontal autoscale type.

    Horizontal scaling applies only to compute only worker nodes.

    The scale-in rule sets the conditions for scaling in the cluster (use a smaller compute shape). Enter the following details:
    • Average CPU threshold percentage: Set a percentage of average CPU utilization, so that when the CPU operates at that average percentage (or higher) for the minimum duration in minutes, the cluster is scaled out.
    • Minimum duration in minutes: Set a duration, so that when the average CPU threshold percentage operates for that duration, the cluster is scaled out.
    • Minimum number of nodes: Enter the minimum number of nodes that the cluster can have. When autoscale is triggered, the number of nodes specified in Step size are removed from the cluster if the number of nodes in the cluster after removal is greater than or equal to the number of nodes specified in this field.
    • Step size: The number of nodes to remove from the cluster when autoscale is triggered.
    For example, if the scale-in rule is:
    • Threshold percentage = 20%
    • Minimum duration = 30 minutes
    • Maximum number of nodes = 1
    • Step size = 1

    and the shapes of the worker nodes are VM.Standard2.8, when the CPU utilization averages 20% or more for 30 minutes with the maximum number of nodes are one and step size is one, the worker nodes are scaled in to VM.Standard2.4.

    Schedule-based Horizontal condition This field appears when you select scheduled-based Horizontal autoscale type.
    Note

    Scheduled-based Horizontal autoscaling applies only to compute only worker nodes.

    For schedule-based Horizontal autoscaling, enter the following details:

    • Days: Enter the days of the week that you want the condition to run.
    • Time: Enter a trigger time for the condition to begin. For example, 9:00 AM.
    • Time Optional: Enter second trigger time on the same day. For example, 4:00 PM.
    • Number of nodes: Enter the target number of nodes at first trigger.
    • Number of nodes Optional: Enter the target number of nodes at second trigger.

    For example, if the schedule-based Horizontal condition is:

    • Days = Monday, Tuesday, Friday
    • Time = 9:00 AM
    • Number of nodes = 10
    • Time Optional = 4:00 PM
    • Number of nodes Optional = 1

    In this example, there are 10 available nodes between 9:00 AM and 4:00 PM on the specified days. However, when the time reaches 4:00 PM, the nodes drop to one and remain at one until another trigger is reached.

    Scheduled-based Vertical condition This field appears when you select scheduled-based Verticle autoscale type.

    For schedule-based Vertical autoscaling, enter the following details:

    • Days: Enter the days of the week that you want the condition to run.
    • Time: Enter a trigger time for the condition to begin. For example, 9:00 AM.
    • Time Optional: Enter second trigger time on the same day. For example, 4:00 PM.
    • Target shape: Select a target shape from the list. If you select a flex shape, you must customize the following resources:
      • Number of OCPUs
      • Amount of memory (GB)
    • Target shape Optional: Select a second trigger target shape.

    For example, if the schedule-based Vertical condition is:

    • Days = Monday, Tuesday, Friday
    • Time = 9:00 AM
    • Target shape = VM.Standard.2.8
    • Time Optional = 4:00 PM
    • Targe shape Optional = VM.Standard.2.4

    In the example, the first trigger is set for 9:00 AM with a target shape of VM.Standard.2.8. The second trigger is set for 4:00 PM with a target shape of VM.Standard.2.4.

  7. Click Create.

    It takes a few minutes for the configuration to take effect. During this time, the cluster is in the Updating state.

    When an autoscale event is triggered, the worker nodes are updated on a rolling basis; that is, one node is updated at a time.

Edit an Autoscale Configuration
To edit an autoscale configuration:
  1. Open the navigation menu and click Analytics and AI. Under Data Lake, click Big Data Service.
  2. In the left panel, under Compartment, select the compartment that hosts your cluster.
  3. In the list of clusters, click the name of your cluster.
  4. In the left panel, under Resources click Autoscale Configurations.
  5. In the list of autoscale configurations, click the name of your autoscale configuration.
  6. Click the actions menu and select Edit.
  7. In the Edit Autoscale Configuration panel, change any of the values, as described in Create an Autoscale Configuration.

    It will take a few minutes for the configuration to take effect.

Delete an Autoscale Configuration
To delete an autoscale configuration:
  1. Open the navigation menu and click Analytics and AI. Under Data Lake, click Big Data Service.
  2. In the left panel, under Compartment, select the compartment that hosts your cluster.
  3. In the list of clusters, click the name of your cluster.
  4. In the left panel, under Resources click Autoscale Configurations.
  5. In the list of autoscale configurations, click the name of your autoscale configuration.
  6. Click the actions menu and select Delete.
  7. To confirm, in the Delete Autoscale Configuration dialog box, enter the autoscale configuration name, the cluster admin password, and then click Delete.

Using Object Storage API keys

Big Data Service uses the OCI API signing key mechanism to connect to Object Storage. You can access Object Storage buckets encrypted with user managed keys from the Big Data Service cluster nodes. For details on data encryption, see securing Object Storage.

Note

To use the Object Storage API keys, you must create a Big Data Service cluster with version 3.0.4 or later. The Big Data Service version is displayed on the Cluster Information tab of the Cluster Details page. .
Create access policy
A tenancy's administrator group users can manage API keys for any user. To allow other users to create and manage Object Storage API keys for themselves, create a policy using the following statement in the root compartment.
allow any-user to {USER_INSPECT, USER_READ, USER_UPDATE, USER_APIKEY_ADD, USER_APIKEY_REMOVE} in tenancy where request.principal.id = target.user.id
Create an Object Storage API key
  1. Open the navigation menu and click Analytics and AI. Under Data Lake, click Big Data Service.
  2. In the left panel, under Compartment, select the compartment that hosts your cluster.
  3. In the list of clusters, click the name of your cluster.
  4. In the left panel, under Resources click Object Storage API keys.
  5. Click Create key.
  6. In the Create API key panel, enter a key alias to uniquely identify this key in the cluster.
  7. Enter the OCID of the user who can use this API key. To retreive the user OCID, from the Console navigate to Identity & Security → Users. From the actons menu for the user, click Copy OCID.
  8. Enter and confirm a passphrase. This passphrase is used to encrypt the API key and cannot be changed later.
  9. Select a default region that is used to establish the Object Storage endpoint name.
  10. Click Create.
The API key is listed in the Object Storage API keys page. When the API key is successfully created, it's status changes to Active.
View the configuration file

You can view and copy the public key of the Object Storage API key from its configuraton file.

  1. Access the cluster details page of the cluster that has the API key you want to view.
  2. In the left panel, under Resources click Object Storage API keys.
  3. From the actions menu of the API key you want to view, click View configuration file.
  4. The public key details of the API key are displayed in the View configuration file dialog.
  5. View the configuration file details or copy the public key.
Test the connection to Object Storage
  1. Access the cluster details page of the cluster that has the API key you want to test.
  2. In the left panel, under Resources click Object Storage API keys.
  3. From the actions menu of the API key you want to test, click Test connection.
  4. Enter the Object Storage URI for the bucket you want to connect to in the URI format oci://MyBucket@MyNamespace/.
  5. Enter the passphrase of the API key. You specified this passphrase when you created the API key.
  6. Click Test connection. The status of the test connection is displayed.
Delete an Object Storage API key

When an Object Storage API key is deleted, all user access to run Objet Storage jobs on the Big Data Service clusters is revoked.

  1. Access the cluster details page of the cluster that has the API key you want to delete.
  2. In the left panel, under Resources click Object Storage API keys.
  3. From the actions menu of the API key you want to delete, click Delete.
  4. To confirm the deletion, enter the key alias of the key you want to delete.
  5. Click Delete.

Applying Tags to a Cluster

You can use Oracle Cloud Infrastructure tags to help organize your resources.

Tags are named key/values pairs that you can associate with resources, which you can use to organize and list the resources.

  1. Open the navigation menu and click Analytics and AI. Under Data Lake, click Big Data Service.
  2. In the left panel, under Compartment, select the compartment that hosts your cluster.
  3. In the list of clusters, click the name of your cluster.
  4. Click More Actions, and then Add Tags.
  5. In the Add Tags dialog box, enter information, as described in Resource Tags.
  6. Click Add Tags.

Restarting a Cluster Node

You can restart a node in a running cluster.

To restart a node:
  1. Open the navigation menu and click Analytics and AI. Under Data Lake, click Big Data Service.
  2. In the left panel, under Compartment, select the compartment that hosts your cluster.
  3. In the list of clusters, click the name of the cluster with the node you want to restart.
  4. On the Cluster Details page, under List of cluster nodes, click the action menu for the node you want to restart. Select Restart Node from the menu.
  5. In the Restart Node dialog box, enter the name of the node to restart, and click Restart.

Deleting a Cluster Node

You can delete a node in a running cluster.

Note

Removing a node might fail when decommissing it. There's an upper limit of 40 minutes, and if decommissing the node doesn't complete in the 40 minutes, the request fails. Time taken to decommission a node depends on the number of blocks in the node that need to be moved. Therefore, we recommend you decommission a node first, followed by removing the node from OCI console.

To decommission a node in Ambari, do the following:

  1. Access Apache Ambari.
  2. From the side toolbar, under Hosts, and then select the worker node to be decommissioned.
  3. From the list of components installed on that host, select Data Node.
  4. Click the ... Action icon, and then select Decommission.
To delete a node from the OCI console:
  1. Open the navigation menu and click Analytics and AI. Under Data Lake, click Big Data Service.
  2. In the left panel, under Compartment, select the compartment that hosts your cluster.
  3. In the list of clusters, click the name of the cluster with the node you want to delete.
  4. On the Cluster Details page, under List of cluster nodes, click the action menu for the node you want to delete. Select Delete Node from the menu.
  5. In the Delete Node dialog box, enter the name of the node to delete, and click Delete.
  6. Select the Force delete even when decommissioning fails checkbox if you want to delete the node even when decommissioning of the node fails.

Removing Cloud SQL from a Cluster

Oracle Cloud SQL can be added to a Big Data Service cluster, for an extra fee. If Cloud SQL has been added to a cluster, you can remove it, and you'll no longer be charged for Cloud SQL on the cluster.

Note

Removing Cloud SQL from a cluster terminates the query server node and deletes any files on that node. This is an irreversible action.

Removing Cloud SQL from the cluster:

  • Removes Cloud SQL cells from the cluster worker nodes
  • Terminates the query server node and deletes any files or work that you have on that host. (The VM is terminated.)
  • Has no impact on Hive metadata or the sources that Cloud SQL accesses.
  • Ends the billing for Cloud SQL. You no longer pay for Cloud SQL once it is removed.

To remove Cloud SQL from a cluster:

  1. Open the navigation menu and click Analytics and AI. Under Data Lake, click Big Data Service.
  2. In the left panel, under Compartment, select the compartment that hosts your cluster.
  3. Click action menu for the cluster you want to modify and select Remove Cloud SQL.
  4. In the Remove Cloud SQL dialog box, enter the cluster admin password and click Remove.

Bootstrap Script Helper Functions

You can run a bootstrap script on all the cluster nodes, after a cluster is created, when the shape of a cluster changes, or when you add or remove nodes from a cluster. You can use this script to install, configure, and manage custom components in a cluster.

Note

Logs are available in the /var/logs/oracle/bds/bootstrap/ directory of the mn0 node.
Bootstrap Shell Script Helper Functions

Shell script based bootstrap helper functions exist to update Ambari configurations.

Note

The custom bootstrap script is run as an opc user. To run a command that needs root access, add sudo.
Helper Function Functionality
$getClusterName Displays the cluster name.
$getAllNodesIps Displays a space-separated list of IPs of all the nodes.
$getUtilityNodesIps Displays a space-separated list of IPs of the utility nodes.
$getWorkerNodesIps Displays a space-separated list of IPs of the worker nodes.
$getMasterNodesIps Displays a space-separated list of IPs of the master nodes.
$getQueryServerNodesIps Displays a space-separated list of IPs of the query server nodes.
$eventType Displays the event type that initiated the bootstrap script. Possible event type values are CreateBdsInstance, AddWorkerNodes, StartBdsInstance, ChangeShape, on-demand.
Shell Script Examples
Example 1: Custom bootstrap script with helper functions to find the node types
#!/bin/bash
 
#Helper Method Sample
 
#Use Following Helper function to get list of IPs of all nodes and you can use this to run a command for all nodes
for node in $getAllNodesIps
do
    # setup common things for all nodes
    echo "Node name: $node"
done
 
#Cluster Name
 
echo "Cluster Name"+$getClusterName
 
#Master Nodes 
for node in $getMasterNodesIps
do
    # setup common things for master nodes
    echo "Master Node name: $node"
done
 
 
#Worker Nodes 
for node in $getWorkerNodesIps
do
    # setup common things for worker nodes
    echo "Worker Node name: $node"
done
 
#Utility Nodes
 
for node in $getUtilityNodesIps
do
    # setup common things for Utility nodes
    echo "Utility Node name: $node"
done
 
#QueryServer Nodes
for node in $getQueryServerNodesIps
do
    # setup common things for Query Server nodes
    echo "QueryServer Node name : $node"
done
 
 
 
echo "bootstrap script execution complete!!!"
Example 2: Custom bootstrap script for a cluster life cycle
To run a custom bootstrap script for a cluster life cycle operation such as create cluster, add node, or change shape, use the $eventType helper variable as shown in the following example:
#!/bin/bash
#Take action on certain Event Types
 
echo "Customer Bootstrap Script Execution for EventType :$eventType"
 
if [ $eventType == "CreateBdsInstance" ]
    then
       echo "setup things post Cluster Creation"
    fi
 
if [ $eventType == "AddWorkerNodes" ]
    then
       echo "setup things post ADD Node "
    fi
 
if [ $eventType == "ChangeShape" ]
    then
       echo "setup things post Change Shape"
    fi
Bootstrap Python Script Helper Functions

Python script based bootstrap helper functions exist to update Ambari configurations.

These helper functions help you:
  • Update the configuration of an Hadoop component in the cluster
  • Update the configuration that's applicable to a particular host or group of hosts
  • Obtain the configuration value and take actions based on the value
  • Export configuration from a config type (for example, HDFS core-site.xml)
Note

  1. You can execute these helper functions only from predefined execute method in the Python script.
  2. Python 2.7 is the supported python version in the Python script.
  3. You must manage the Python script at your own risk because it can lead to microservice shut down.
Category Helper Function Functionality
Config Groups createConfigGroup(service_name, hosts, config_group="Default") This helper function creates a config group for a service with a list of hosts. You must provide the service name, list of hosts, and config group name as input.
removeConfigGroup(service_name, config_group="Default") This helper function removes the config group in a service. You must provide the service name and config group name as input.
addHostsInConfigGroup(service_name, hosts, config_group="Default" This helper function adds hosts to the config group in a service with a list of hosts. You must provide service name, list of hosts, and config group name as input.
removeHostsInConfigGroup(service_name, hosts, config_group="Default" This helper function removes hosts to the config group in a service with la ist of hosts. You must provide service name, list of hosts, and config group name as input.
listConfigGroups(service_name) This helper function displays all the config groups in a service. You must provide service name.
getHostsInConfigGroup(service_name, config_group="Default" This helper function displays all the hosts in the config group in a service. You must provide config group and service name.
importConfigGroup(service_name, from_config_group, to_config_group) This helper function clones the configs from one config group to another config group in a service. You must provide from config group, to config group, and service name.
Config updateConfig(service_name, config_type, config_properties, config_group="Default")

This helper function updates the config properties to the config type file of config group in a service. You must provide config group, config type, config properties, and service name.

Note : config_properties is a map of config key and value pairs.

For example: helper.updateConfig(service_name="HDFS", config_type="core-site", config_properties={"fs.trash.interval": "400", "config_group": "test"})

removeConfig(service_name, config_type, config_name, config_group="Default") This helper function removes the config from the config type file of config group in a service. You must provide config group, config type, config name, and service name.

For example: helper.removeConfig(service_name="HDFS", config_type="core-site", config_name="fs.trash.interval")

getConfig(service_name, config_type, config_name, config_group="Default") This helper function displays the config property value of config name in the config type file of a config group in a service. You must provide config group, config type, config name, and service name.
restartStaleConfig() This helper function restarts the services where configs are stale.
exportConfig(service_name, config_type, config_group="Default") This helper function displays all the configs in config type file of config group in a service. You must provide config group, config type, and service name.
Shell execution runShellCommandOnAllNodes(command) This helper function executes shell command on all the cluster nodes. You must provide the command.
runShellCommandOnNode(command, host) This helper function executes shell command on the requested host. you must provide the command and host.
Utility help functions getClusterName() Displays cluster name.
getMasterNodesIps() Displays master nodes lps.
getWorkerNodesIps() Displays worker nodes lps.
getUtilityNodesIps() Displays utility node lps.
getQueryServerNodesIps() Displays query server nodes lps.
getComputeOnlyWorkerNodesIps() Displays compute only worker node lps.
getEdgeNodesIps() Displays Edge node lps.
getAllNodesIps() Displays all nodes lps.
getEventType() Displays the event type. Possible event type values are CreateBdsInstance, AddWorkerNodes, StartBdsInstance, ChangeShape, on-demand.
getLogger()

This helper function returns the logger instance which you can use to log info and error logs. For example, getLogger().info("log info msg")

list of supported service names = [HDFS, YARN, MAPREDUCE2, TEZ, HIVE, OOZIE, ZOOKEEPER, AMBARI_METRICS, RANGER, HUE, KERBEROS, ODHUTILS, SPARK3, HBASE, TRINO]

list of supported event types = ["CreateBdsInstance", "AddWorkerNodes", "StartBdsInstance", "ChangeShape"]

Python Script Examples
Example 1: Custom bootstrap python script with helper functions to update Ambari configs in Default config group
#!/usr/bin/env python2.7
 
def execute(helper):
    # custom logger
    logger = helper.getLogger()
     
    logger.info('Testing Config Helper functions')
 
    # Update config_properties in the Default config group of config_type file core-site in service HDFS
 
    helper.updateConfig(service_name="HDFS", config_type="core-site", config_properties={"fs.trash.interval": "400"})
 
    # Remove config property from Default config group of config_type core-site
 
    helper.removeConfig(service_name="HDFS", config_type="core-site", config_name="fs.trash.interval")
 
    # Get config value from Default config group of config_type file core-site in service HDFS
 
    config_value = helper.getConfig(service_name="HDFS", config_type="core-site", config_name="fs.trash.interval")
 
    # Export configs from Default config group of config_type file core-site in service HDFS
 
    helper.exportConfig(service_name="HDFS", config_type="core-site")
 
    # Restart stale config
 
    helper.restartStaleConfig()
Example 2: Custom bootstrap Python script with helper functions for shell command execution
#!/usr/bin/env python2.7
 
def execute(helper):
    logger = helper.getLogger()
 
    logger.info("Custom logger logs are available in '/var/logs/oracle/bds/bootstrap/' directory of mn0 node")
 
    logger.info("Execute get utility nodes ips")
 
    utility_node_ips = helper.getUtilityNodesIps()
 
    logger.info("Execute shell command on utility node")
 
    helper.runShellCommandOnNode(command='pwd', host=utility_node_ips[0])
 
    logger.info("Execute shell command for on-demand event type")
 
    event_type = helper.getEventType()
 
    if event_type == "on-demand":
        helper.runShellCommandOnNode(command='ls', host=utility_node_ips[0])
 
 
    logger.info("Create config group test in service HDFS")
 
    helper.createConfigGroup(config_group="test", service_name="HDFS", hosts=[])
 
    logger.info("Add Worker nodes as hosts to above created config group test in service HDFS")
 
    worker_node_ips = helper.getWorkerNodesIps()
 
    helper.addHostsInConfigGroup(service_name="HDFS", hosts=worker_node_ips, config_group="test")
 
    logger.info("Update config_properties in the config group test of config_type file core-site in service HDFS")
 
    helper.updateConfig(config_group="test", service_name="HDFS", config_type="core-site",
                        config_properties={"fs.trash.interval": "400"})
 
    logger.info("Restart stale configs")
 
    helper.restartStaleConfig()
Updating Bootstrap Script URL
  1. On the cluster details page of the cluster for which you want to update the bootstrap script URL, from the More Actions menu, select Update bootstrap script URL.
  2. In the Update bootstrap script URL dialog box that appears, in the Bootstrap script URL field, enter the updated URL.
    Note

    The updated script is run only when the shape of the cluster changes, or when you add or remove nodes from a cluster.
  3. Click Update.

Finding Available Updates

To find the available updates for a cluster, follow these steps:

  1. Select the cluster for which you want to view the updates and navigate to its details page.
    Note

    If there are any updates to be installed, a notification appears on the top right corner of the page.
  2. On the left side of the page, under Resources, click Updates.
    The Available updates section displays the list of updates to be installed. The Update history section displays the list of installed updates. For information about installing updates, see Installing Available Updates.

Installing Available Updates

To install the available updates for a cluster, follow these steps:

  1. Select the cluster for which you want to install the updates and navigate to its details page.
  2. On the left side of the page, under Resources, click Updates. A list of available updates is displayed.
  3. In the Available updates section, click Install against the patch version that you want to update.
  4. In the Install patch dialog box that appears, enter the cluster admin password and click Install.
    After the update is installed, the Cluster information box displays the latest version of the cluster.

Terminating a Cluster

You can terminate any cluster.

Caution

Terminating a cluster deletes the cluster and removes all the data contained in local storage or block storage. This is an irreversible action.

To terminate the cluster:

  1. Open the navigation menu and click Analytics and AI. Under Data Lake, click Big Data Service.
  2. In the left panel, under Compartment, select the compartment that hosts your cluster.
  3. Click the action menu for the cluster you want to terminate and select Terminate Big Data Cluster.
  4. In the Terminate Big Data Cluster dialog box, enter the name of the cluster and click Terminate.