Oracle Cloud Infrastructure Documentation

Getting Started with Big Data Service

Before you can create and use Big Data Service clusters in a new account, an administrator for the account must set up the Oracle Cloud Infrastructure environment to support them.

When you sign up for Oracle Cloud Infrastructure, Oracle creates a tenancy for the account. The tenancy is a secure and isolated partition within Oracle Cloud Infrastructure where you can create, organize, and administer your cloud resources.

Big Data Service uses Oracle Cloud Infrastructure Identity and Access Management (IAM) for authentication and authorization through Oracle Cloud Infrastructure. Create Identity and Access Management (IAM) users, groups, and policies to control access to resources used in your Big Data Service environment. Manage authentication and authorization to Big Data Service by using Oracle Cloud Infrastructure services and by using Linux operating system services and accounts.

To set up access controls for Big Data Service through IAM, an administrator does the following:

  • Optionally creates compartments in the tenancy, to organize and isolate resources and services. See Setting Up Your Tenancy .

  • Configure the Virtual Cloud Network (VNC) for the service, including subnets and network resources. Network access rules control access to the network and resources on the network. See Networking Overview.

  • Creates groups and adds users to the group. At a minimum, there should be an administrators group and a users group.

  • Creates policies to control access. Policies can be written, for example, to say which group(s) can perform what action(s) on which resource(s) in which compartment(s).

    See Overview of Oracle Cloud Infrastructure Identity and Access Management for an overview, as well as Getting Started with Policies.

Note

Every tenancy has one or more users in the Administrators group. These users are cloud administrators and can perform the above actions, even if no administrator is defined for your Big Data Service.

For information about users, groups, and policies defined through IAM, see Overview of Oracle Cloud Infrastructure Identity and Access Management and subsequent topics.

Creating IAM Users

You can apply policies to individual users, but it's best practice to create groups whose privileges are defined in polices and then add users to the groups.

Create Identity and Access Management (IAM) groups with access privileges appropriate to your needs.

For the simple case, use the existing Administrators group, whose members have complete administrative rights over the resources in the tenancy. You can also create a big-data-users group, whose members have read access to the cluster and appropriate rights for working with data. In more complex situations, you need multiple administrator groups whose access is restricted to certain compartments or to certain resources, such as networking or clusters and storage.

Big Data Service Administrators

While the cloud administrator has complete control over all Big Data Service resources in the tenancy, it's good practice to delegate cluster administration tasks to one or more Big Data Service administrators.

Among the tasks that an administrator must do are:

  • Create compartments in the tenancy, to organize and isolate resources (optional).
  • Set up a Virtual Cloud Network (VCN) in the tenancy, including subnets, gateways, and other networking resources.
  • Create groups, users, and policies in the tenancy, to control who has what kind of access to which resources.

For complete documentation on setting up a new tenancy, see Setting Up Your Tenancy in the Oracle Cloud Infrastructure documentation.

About the Types of Administrators

You can set up administrator groups and administrator users in a number of different ways, depending on your requirements. One useful strategy is to create three kinds of administrators:

  • Cloud administrator - A cloud administrator has full access privileges to all the resources in the tenancy, including those used by any other services in your tenancy. When Oracle creates a new tenancy, a default administrator is created for the tenancy, and that administrator is a cloud administrator. You can create additional cloud administrators.

    A cloud administrator is created through Oracle Cloud Infrastructure Identity and Access Management. See Overview of Oracle Cloud Infrastructure Identity and Access Management.

  • Big Data Service administrator - A Big Data Service administrator has privileges to manage one or more clusters and the resources associated with them, like groups, users, storage, and so forth. This kind of administrator might not have privileges on other services in the tenancy or might have limited privileges on shared resources, like networks. A Big Data Service administrator is created through Oracle Cloud Infrastructure Identity and Access Management.
  • Hadoop cluster administrator A Hadoop cluster administrator has permissions directly on the cluster, to perform management operations programmatically. A Hadoop administrator is created by using operating system commands. See Create a Big Data Service Administrators Group and Add a User.

Create a User to be the Big Data Service Administrator

To create a new Big Data Service administrator for a new service, a cloud administrator must create a user and then add the user to a Big Data Service administrators group.

To create the user:
  1. Open the navigation menu and click Identity & Security. Under Identity, click Users.
  2. Click Create User.
  3. In the Create User page, enter the following information:
    • Name - Enter a name for the user. You may want to use the user's email address.
    • Description - Enter a description.
    • Email - Enter the user's email address, which can be used to prompt the user to create a new password if it's been lost or forgotten.
  4. Click the Create button.

Create a Big Data Service Administrators Group and Add a User

Create a Big Data Service administrators group, whose members will be granted permissions to manage the cluster life cycle.

To create the group and add a user to the group:
  1. Open the navigation menu and click Identity & Security. Under Identity, click Groups.
  2. Click Create Group.
  3. In the Create Group panel, enter a name for the group, for example bds-admins, and a description, and then click Create.
  4. On the Identity > Groups page, click the name of the administrator group you just created, for example, bds-admins.
  5. On the Group Details page, under Group Members, click Add User to Group.
  6. In the Add User to Group dialog box, select a user to add to the group, and then click Add. If you haven't created the user who will be an administrator yet, go back to Create a User to be the Big Data Service Administrator to create the user, and then return to this step.

About the Operating System User Accounts

In addition to the access controls provided by Oracle Cloud Infrastructure, Big Data Service supports operating system user accounts, which allow direct access to the cluster through SSH.

Every Big Data Service cluster node is provisioned with the following operating system user accounts.

  • opc

    This is the system administrator account you use in conjunction with the sudo command to gain root user access to your nodes. By default, opc doesn’t allow connection using a password; however, you may choose to connect using a password by assigning a known password to opc or by creating another user with a known password.

  • root

    This is the root administrator for the system. You do not have direct access to this account. To perform operations that require root user access, execute sudo -s as the opc user. By default, root doesn’t require a password.

  • oracle

    This is an operating system and cluster user account that's used to run jobs on the cluster during the validation of the cluster. This account is used by the system and has a randomly generated password.

Creating IAM Policies

Big Data Service uses Oracle Cloud Infrastructure Identity and Access Management (IAM) for authentication and authorization through Oracle Cloud Infrastructure.

After you create Identity and Access Management (IAM) users and groups, use policies to control access to resources used in your Big Data Service environment.

Create IAM Policies to Control Access to Resources

Create Oracle Cloud Infrastructure Identity and Access Management (IAM) policies to grant privileges to users and groups to use and manage Big Data Service resources. Before you can create a cluster, you must also create a policy that grants the system access to networking resources.

Your service minimally requires:
  • A policy with the following policy statement, granted on the compartment with the network resources used by the cluster(s):

    allow group <admin-group> to manage virtual-network-family in compartment <compartment>

    That policy statement grants network administration rights to members of the <admin-group> .

  • A policy with the following policy statement, granted on the compartment that will contain the cluster(s):

    allow group <admin-group> to manage bds-instance in compartment <compartment>

    That policy statement grants cluster administration rights to members of the <admin-group> .

  • A policy with the following policy statement, also granted on the compartment that contains the network resources that will be used by the cluster(s):

    allow service bdsprod to {VNIC_READ, VNIC_ATTACH, VNIC_DETACH, VNIC_CREATE, VNIC_DELETE,VNIC_ATTACHMENT_READ, SUBNET_READ, VCN_READ, SUBNET_ATTACH, SUBNET_DETACH, INSTANCE_ATTACH_SECONDARY_VNIC, INSTANCE_DETACH_SECONDARY_VNIC} in compartment <compartment>

    That policy statement grants the system the rights to interact with various networking components.

  • You can also optionally create policies for users with lesser access privileges than administrators. For example, you can create a simple policy for non-admin users with a statement like:

    allow group <user-group> to read all-instances in compartment <compartment>

For more information about the entities you can specify in a Big Data Service policy, see Understand Big Data Service Resources and Permissions in IAM Policies.

In the above:

  • <admin-group> is an IAM group you created for Big Data Service administrators. It's possible to have different administrator groups, one for clusters and one for networking, but it many cases, all that's required is a single administrator group whose members can manage all Big Data Service resources.

  • <user-group> is an IAM group you created for Big Data Service users.

  • <compartment> is an Oracle Cloud Infrastructure compartment that contains your resources. You can have one compartment for clusters and related resources and another for network resources. However, you can use a single compartment for both.

    It's the in compartment <compartment> phrase at the end of the policy statement that determines which compartment the policy statement affects. If you want the policy statement to apply to all the resources in your tenancy, substitute in tenancy for in compartment <compartment>.

    The policy itself can reside in any compartment in the tenancy, since the in compartment <compartment> or in tenancy phrases determine the scope of the policy. However, be aware that an administrator must have administrator privileges in the compartment containing the policy to create, modify, or delete it.

  • bdsprod (in the policy statement that begins allow service bdsprod to ...) is the name of the service, which is used in access policies. This name doesn't change.

The following instructions describe a scenario with one administrator group, bds-admins, one users' group, bds-users, and one compartment, bds-compartment. The instructions tell how to create one policy for admin access to clusters, another for admin access to the network, another for user access to the clusters, and another to allow the system to use the networking resources, as described in the introduction above. But you can create multiple users and groups and use multiple compartments, as suits your needs.

To create the policies:

  1. Open the navigation menu and click Identity & Security. Under Identity, click Policies.
  2. In the left panel, under Compartment, select the compartment where you want the policies to reside.
  3. Click Create Policy.
  4. In the Create Policy page, enter the following information:

    • Name: Enter a name for the policy, for example, bds-net-admin.

    • Description: Enter a description for the policy.

    • Compartment: Select a compartment from the list, if you want to create the policy in a different compartment.

    • Policy Builder: Click the Show manual editor toggle on the Policy Builder box. Copy the following and paste it into the text box: 

      allow group bds-admins to manage virtual-network-family in compartment bds-learn
allow group bds-admins to manage bds-instance in compartment bds-learn
allow group bds-ua-users to read instances in compartment bds-learn
allow service bdsprod to {VNIC_READ, VNIC_ATTACH, VNIC_DETACH, VNIC_CREATE, VNIC_DELETE,VNIC_ATTACHMENT_READ, SUBNET_READ, VCN_READ, SUBNET_ATTACH, SUBNET_DETACH, INSTANCE_ATTACH_SECONDARY_VNIC, INSTANCE_DETACH_SECONDARY_VNIC} in compartment bds-learn
  5. Click Create.
  6. To review any policy, click its name.

For more information about creating groups, users, and policies, see Overview of Oracle Cloud Infrastructure Identity and Access Management in the Oracle Cloud Infrastructure documentation.

Understand Big Data Service Resources and Permissions in IAM Policies

Oracle Identity and Access Management (IAM) provides a flexible framework for writing policy statements that control how resources can interact with one another. IAM defines a number of standard resources, along with the permissions needed to interact with them. Big Data Service adds its own service-specific resources and permissions.

This topic describes the resources and permissions an administrator can use to create IAM policy statements for Big Data Service.

Resource Kinds and Permissions

Resource Family Resource Kind Permissions
bds-family bds-instances
  • BDS_INSPECT
  • BDS_READ
  • BDS_CREATE
  • BDS_UPDATE
  • BDS_DELETE
  • BDS_MOVE
bds-family bds-limits
  • BDS_CONSUMPTION_INSPECT

Operations to Permissions Map

The following table lists the IAM operations that are specific to Big Data Service. You can write an IAM policy that includes these operations, or you can write a policy that uses a defined verb that encapsulates these operations.

Operation API Operation Permission Required to Use the Operation
List all clusters in the specified compartment ListBdsInstances BDS_INSPECT
Create a cluster CreateBdsInstance BDS_CREATE
Show details about the specified cluster GetBdsInstance BDS_READ
Change the size of a cluster ChangeShape BDS_UPDATE
Update details for a cluster UpdateBdsInstance BDS_UPDATE
Delete the specified instance DeleteBdsInstance BDS_DELETE
Add block storage to the specified cluster AddBlockStorage BDS_UPDATE
Add worker nodes to the specified cluster AddWorkerNodes BDS_UPDATE
Restart a specified node of a cluster RestartNode BDS_UPDATE
Add Cloud SQL to the specified cluster AddCloudSql BDS_UPDATE
Remove Cloud SQL from the specified cluster RemoveCloudSql BDS_UPDATE
Move the cluster from one compartment to another ChangeBdsInstanceCompartment BDS_MOVE
List all autoscale configurations for the specified cluster ListAutoScalingConfigurations BDS_INSPECT
Add an autoscale configuration to the specified cluster AddAutoScalingConfiguration BDS_UPDATE
Show details about the specified autoscale configuration GetAutoScalingConfiguration BDS_READ
Update fields of an autoscale configuration UpdateAutoScalingConfiguration BDS_UPDATE
Delete an autoscale configuration RemoveAutoScalingConfiguration BDS_UPDATE
List all Big Data work requests in the specified compartment ListWorkRequests BDS_INSPECT
Show details about the specified work requests GetWorkRequest BDS_READ
Show logs for the specified work request ListWorkRequestLogs BDS_INSPECT
Show errors for the specified work request ListWorkRequestErrors BDS_INSPECT
Show resources used ListConsumptions BDS_CONSUMPTION_INSPECT
List API keys on the specified cluster ListBdsApiKeys BDS_READ
Create an API key on the specified cluster CreateBdsApiKey BDS_UPDATE
Get an API key on the specified cluster GetBdsApiKey BDS_READ
Delete an API key on the specified cluster DeleteBdsApiKey BDS_UPDATE
Test access to Object Store bucket using the specified API key TestBdsObjectStorageConnection BDS_READ

Operation-Specific Attributes

Note

For a given resource kind, you should have the same set of attributes across all operations (get, list, delete, and so on). The one exception is for a "create" operation, where you won't have the ID for that object yet, so you can't have a target.RESOURCE-KIND.id attribute for "create."

Resource Kind Name Type Source
bds-instances target.bds-instances.source-compartment.id Entity Request
bds-instances target.bds-instances.destination-compartment.id Entity Request

IAM Verbs for Use with Big Data Service

Resource Kind inspect read use manage
bds-instances BDS_INSPECT inspect +

BDS_READ

 read +

BDS_UPDATE

use +

BDS_CREATE

BDS_DELETE

BDS_MOVE
bds-limits BDS_CONSUMPTION_INSPECT . . .

Example 1 - Administrators with All Permission on Clusters

The following policy statement says that members of a group named bds-admins can inspect, read, update, create, delete, and move all clusters in a compartment named bds-learn.

allow bds-admins to manage bds-instances in compartment bds-dev

In the above statement:

  • bds-admins is a group created by an administrator.

  • manage specifies the operations that members of the bds-admins group can use. Manage is one of the verbs described in the "IAM Verbs for Use with Big Data Service" section above. It gives a user/group permission to use all of the operations provided by the inspect, read, and use verbs, plus a few operations specific to the manage verb:

    • The inspect verb includes the BDS_INSPECT operation.
    • The read verb includes the BDS_INSPECT and BDS_READ operations.
    • The use verb includes the BDS_INSPECT, BDS_READ, and BDS_UPDATE operations.
    • The manage verb includes the BDS_INSPECT, BDS_READ, BDS_UPDATE, BDS_CREATE, BDS_DELETE, and BDS_MOVE operations.

  • bds-dev is a compartment created by an administrator.

The following policy statement says that members of the bds-admins group can manage the Virtual Cloud Network (VCN) resources in the entire tenancy.

allow group bds-admins to manage virtual-network-family in tenancy

Example 2 - Users

The following policy statement says that members of a group named bds-users can inspect and read all clusters in the bds-learn compartment. (The verb read includes both inspect and read permissions.)

allow bds-users to read bds-instances in compartment bds-learn

More Information

For more information about IAM policies, see Overview of Oracle Cloud Infrastructure Identity and Access Management in the Oracle Cloud Infrastructure documentation. For details about writing policies, see Policy Syntax and Policy Reference.

Creating Network Resources

Before you can create and use Big Data Service clusters, you must create and configure a network. Oracle Cloud Infrastructure Networking service provides a wide range of features for establishing a secure networking topology for your Big Data Service.

For complete documentation about Oracle Infrastructure Networking, see Overview of Networking and the subsequent networking topics in the Oracle Cloud Infrastructure documentation. The following sections discuss networking details that are specific to Big Data Service.

Terminology

The term "network" may be used to refer to a Virtual Cloud Network (VCN) or a subnet in a VCN. When the difference is pertinent, the "VCN" or "subnet" is used.

"Instance," "host," and "node" may be used interchangeably. However, because the hosts that comprise a Hadoop cluster are called nodes, "nodes" is used throughout this documentation.

Understand Networking for Big Data Service

In Oracle Cloud Infrastructure, a network consists of at least one Virtual Cloud Network (VCN) with at least one subnet, along with Virtual Network Interface Cards (VNICs), gateways, route tables, security rules, and other virtual networking features. For a simple development environment, you may only need a single VCN with a single subnet in a single region, possibly with access to the public internet. For a complex production environment, you may want to connect your VCN to an on-premises network, and you may want to peer with other VCNs in other regions.

A network used for Big Data Service must meet the general requirements for any Oracle Cloud Infrastructure network, as described in Overview of Networking and the subsequent networking topics in the Oracle Cloud Infrastructure documentation. In addition to those requirements, consider the following information specific to Big Data Service:

Creating and Using Subnets

Subnets divide a VCN by assigning ranges of IP addresses that don't overlap with other subnets in the VCN. Consider the following when creating your network for Big Data Service:

A subnet must be regional and it may be public:

  • In Oracle Cloud Infrastructure, a subnet can exist in a single availability domain or across an entire region. A regional subnet is required for Big Data Service. Therefore, when you create the VCN for Big Data Service, you must create at least one regional subnet in it.

  • Cluster nodes are by default private. If you plan to make your cluster available for access from the public internet, you must use a public subnet. In that case, when you create the VCN, the regional subnet you create (see above) must also be public. See also "Making Nodes Accessible," below.

You specify which VCN and which subnet to use for a cluster when you create the cluster. See Creating a Cluster.

Making Nodes Reachable

As mentioned above, cluster nodes are private by default. Nodes are created with private IP addresses, and all ports are closed by default (with the exception of port 22, which is open for SSH access). Therefore, you must configure the network to allow access to the nodes.

Configuring Security Rules for Nodes

A security rule allows a particular type of traffic in or out of a VNIC (which connects nodes to the network). Each security rule specifies direction (ingress or egress), stateful or stateless, source type and source (for ingress rules), destination type and destination (for egress rules), IP protocol, source port, destination port, and ICMP type and code.

To allow network traffic to and from a cluster node, you must configure the security rules for the node. Do this for all nodes that you want to make reachable, whether from the public internet, from a private network, or from both.

See Define Security Rules in this documentation and Security Rules in the Oracle Cloud Infrastructure documentation.

Making Nodes Accessible from the Internet

To make nodes publicly accessible from the internet, you must:
  • Use a public subnet. See VCNs and Subnets in the Oracle Cloud Infrastructure documentation.
  • Create and map public IP addresses to the default private IP addresses of nodes you want to open to the internet. See Map a Private IP Address to a Public IP Address.
  • Configure security rules to allow traffic from the internet. See "Configuring Security Rules for Nodes," below.

See also Access to the Internet in the Oracle Cloud Infrastructure documentation.

About the Customer Network and the Cluster Private Network

Big Data Service clusters are dual-homed. The nodes of the cluster are connected to both a cluster private network in the Oracle tenancy and to a customer network in your customer tenancy.

About the Cluster Private Network

The cluster private network is a Virtual Cloud Network (VCN) is created in the Oracle tenancy when a cluster is created. Characteristics of this network are:

  • When you create a cluster, you're prompted to specify a CIDR block to allocate a range of IP addresses for the network. This CIDR block can't overlap the CIDR block of the customer private network.
  • The private IP addresses of the cluster nodes are assigned from the CIDR block of the private subnet in this VCN.

  • The network is used exclusively for private communication among the nodes of the cluster; for example, distributed data processing, service monitoring, etc. All ports are open by default.

  • You can choose to deploy a service gateway and a network address gateway on this network, but you can't otherwise configure gateways, routing tables, or security lists on this network to control network traffic to and from your cluster. See Networking Gateway Options When Creating a Cluster, below.

About the Customer Network

The customer network is a in your customer tenancy. The VCN must already exist (and must have a regional subnet) before a cluster can be created. Details about this network are:

  • When you create a cluster, you're prompted to choose an existing VCN and subnet to associate with the cluster.

  • The subnet you choose for the cluster must be a regional subnet. If you want to make any of the nodes available to traffic from the public internet, you must choose a public subnet. If you're using IPSec VPN or Oracle Cloud Infrastructure FastConnect to connect to your on-premises network, you can use a private subnet, but that means traffic through the public internet won't be allowed.
  • You can configure gateways, routing tables, and security lists on this network to control network traffic to and from your cluster.
  • In your customer VCN, some ports are open for Hadoop components to communicate with each other. We recommend that you encrypt the network communication between these ports using encryption algorithms, such as AES 256.

Networking Gateway Options When Creating a Cluster

When you create a cluster, you must choose between these two options:
  • Choose Deploy Oracle-managed Service gateway and NAT gateway (Quick Start) to deploy a service gateway and a NAT gateway in the cluster private network.
    • A NAT gateway enables nodes without public IP addresses to initiate connections to and receive responses from the internet but not to receive inbound connections initiated from the internet. See NAT Gateway.
    • A service gateway enables nodes without public IP addresses to privately access Oracle services, without exposing the data to an internet gateway or a NAT gateway. See Service Gateway.

    When you select this option, you won't be able to limit that access in any way (for example by restricting egress to only a few IP ranges). When you choose this option:

    • The service gateway and the NAT gateway are used for all the operations described above, for the lifetime of the cluster. You can't change it after the cluster has been created, and any service gateways or NAT gateways in your customer network will be ignored.
    • This NAT gateway gives all nodes in the cluster private network full outbound access to the public internet.
    • You can't further restrict traffic that's directed to the NAT gateway or the service gateway. For example, you can't redirect traffic to or from specific IP addresses.

  • Choose Use the gateways in your selected Customer VCN (Customizable) to use a service gateway and a NAT gateway in your customer network.

    When you choose this option:

    • You have complete control over the routing of network traffic to and from your cluster.

    • You must create and configure the gateways yourself. See Service Gateway.

      If you create your network by using one of the network creation wizards in the console, some gateways are created for you, but you may have to configure them to suit your needs. See Virtual Networking Quickstart.

    • You must create and configure security rules to restrict traffic through the gateways.
    • You can change the configuration any time.
    • If you map the private IP addresses of the cluster nodes to public IP addresses, a NAT gateway isn't needed. See Map a Private IP Address to a Public IP Address.

Increasing Service Limits

Like other Oracle Cloud Infrastructure services, Oracle limits the number of resources you can use in your tenancy. You can request that those limits be increased.

About Oracle Cloud Infrastructure Limits

When you sign up for Oracle Cloud Infrastructure, a set of service limits is configured for your tenancy. The service limit is the quota or allowance set on a resource. For example, your tenancy is allowed a maximum number of compute instances per availability domain. These limits are generally established with your Oracle sales representative when you purchase Oracle Cloud Infrastructure. If you didn't establish limits with your Oracle sales representative, or, if you signed up through the Oracle Store, default or trial limits are set for your tenancy. These limits may be increased for you automatically based on your Oracle Cloud Infrastructure resource usage and account standing. You can also request a service limits increase.

For more information about limits, see Service Limits.

Find the Limits for Your Tenancy

To find the limits for all the resources in your tenancy:

  1. Open the navigation menu, and click Governance and Administration. Under Governance, click Limits, Quotas and Usage.
    Note

    You must have permission to view limits and usage. See "To view your tenancy's limits and usage" under Service Limits.

  2. From the Service dropdown, select Big Data to see how many OCPUs are available for each cluster node type.

Big Data Service Default Limits

Among the limits set on your tenancy are limits on the number of Big Data Service cluster nodes you can create. More specifically, you're restricted to a certain number of nodes of a certain shape.

The following table shows the default limits to various cluster shapes. These are your limits if you didn't make other arrangements when you bought your subscription and if you haven't already asked for an increase.

Resource Monthly Universal Credits Pay-as-You-Go
VM.Standard2.1 12 instances (12 OCPUs) 8 instances (8 OCPUs)
VM.Standard2.2 12 instances (24 OCPUs) 8 instances (16 OCPUs)
VM.Standard2.4 12 instances (48 OCPUs) 8 instances (32 OCPUs)
VM.Standard2.8 8 instances (64 OCPUs) Contact us
VM. Standard2.16 8 instances (128 OCPUs) Contact us
VM.Standard2.24 8 instances (192 OCPUs) Contact us

VM.DenseIO2.8

VM.DenseIO2.16

VM.DenseIO2.24

BM.HPC2.36

BM.DenseIO2.52

BM.Standard2.52

 Contact us Contact us

About the Units Shown for Big Data Service Limits

In practice, you increase the number of nodes, or instances, in a cluster. ("Nodes" and "instances" mean the same thing in this context. Oracle Cloud Infrastructure services usually use the term "instance," but Big Data Service follows the Hadoop convention of using the term "node.")

However, the limits are usually expressed as a number of Oracle Compute Units (OCPUs). Each type of Big Data Service node shape has a set number of OCPUs. The number after the decimal in the node shape name indicates the number of OCPUs in a single node of that shape. For example, a VM.Standard2.1 node has one OCPU, a VM.Standard2.4 node has four OCPUs, and a BM.DenseIO2.52 node has 52 OCPUs.

For example, if your subscription uses monthly universal credits, the default limit for node shape VM.Standard2.4 is 48 OCPUs, which equals 12 nodes. The calculation is as follows: 48 OCPUs service limit divided by 4 OCPUs per node equals 12 nodes.

Request an Increase for Big Data Service Nodes

  1. Open the navigation menu, and click Governance and Administration. Under Governance, click Limits, Quotas and Usage.
  2. Click the request service limit increase link.
  3. On the Request Service Limits Updates page, under Primary Contact Details, enter your name and email.

  4. Under Resource Limit Update (this will be your new limit), select the following:

    • Service Category: Select Big Data.
    • Resource: Select the node shape whose limits you want to increase, for example VM.Standard2.16.
    • <region> Region Limit: Select the new limit you want. (Notice that there is a message under this field that lists your current limit and current usage for the selected node shape.)

  5. If you want to increase limits for other node types, click the Additional Request button and repeat the process from above.

  6. Under Reason for Request, enter your reason, and the click Submit Request.

For more information, see Requesting a Service Limit Increase.