Accessing and Monitoring Clusters

Use the Big Data pages in the Oracle Cloud Console to view and manage the service and to create and manage clusters.

Note

You can't create clusters until you've set up your Oracle infrastructure environment to support Big Data. See the Set up your infrastructure tasks in Big Data's Typical Workflow.

Note

The Big Data pages give you access to your clusters, but you'll use other Oracle Cloud Console pages to manage other aspects of your service, for example the Networking pages for your network and the Identity pages for identity and access management.

To open Big Data in the Oracle Cloud Console:

  1. Open the Oracle Cloud Console. See Signing in to the Console.
  2. Open the navigation menu and click Analytics and AI. Under Data Lake, click Big Data Service.

Viewing Clusters

You can see all the clusters in a compartment in your account on the Clusters page.

To see the clusters, open Big Data in the console. See Accessing and Monitoring Clusters.

The table under Clusters lists all the Big Data clusters in your account, with the following information about each:

  • Name - Click a name in the Name column to show details about the cluster.
  • State - The state of the cluster such as Creating, Active, or Deleted.
    Note

    Suspending clusters is not supported in this release.
  • Number of Nodes - The number shows the combined total number of master nodes and worker nodes in the cluster.
  • Secure and Highly Available - This option is set when a cluster is created. If a cluster was created as secure and highly available, it remains secure and highly available throughout the lifetime of the cluster.
  • Created - Shows when the cluster was created.

Click the View Cluster Details icon at the end of the row for a cluster to open a menu with these commands:

Viewing Cluster Details

You can review details about metrics, work requests, and nodes in a cluster.

To view details about a cluster:
  1. Open the navigation menu and click Analytics and AI. Under Data Lake, click Big Data Service.
  2. In the left panel, under Compartment, select the compartment that hosts your cluster.
  3. In the list of clusters, click the Name of your cluster.
    The Cluster Information box includes the following information about the cluster:
    • Cluster OCIDs - The Oracle Cloud Infrastructure IDs.
    • Compartment - The Oracle Cloud Infrastructure compartment containing the cluster.
    • Total Number of Nodes - The combined total of master/utility nodes and worker nodes.
    • Secure and Highly Available - True if the cluster is secure and highly available. See Creating a Secure and Highly Available Cluster.
    • Created - The date and time the cluster was created.
    • Cloud SQL Installed - True if Cloud SQL Support has been added to the cluster. See Adding Cloud SQL to a Cluster.
    • Cluster Version - The version of Oracle Distribution including Apache Hadoop (ODH) or Cloudera Distribution Including Hadoop (CDH) used for this cluster.
    • Ambari URL or CM URL - The URL for Apache Ambari if your created a cluster with Oacle Distribution including Apache Hadoop or Cloudera Manager if you created a cluster with Cloudera Distribution including Apache Hadoop.

    On the left side of the page, under Resources, three links are displayed: Nodes, Cluster Metrics, and Work Requests.

  4. Select Nodes to display information about the nodes in the cluster.
    Click the name of a node on this page to view information about the node, including the IP address of the node.
  5. Select Cluster Metrics to display metrics for the cluster.
  6. Select Work Requests to display the work requests for the cluster. Work requests describe cluster management activities, like create, add nodes, terminate, etc.

Viewing Cluster Node Details

View details about cluster nodes from the Big Data Cluster Detailspage.

To view details about the nodes in your cluster:
  1. Open the navigation menu and click Analytics and AI. Under Data Lake, click Big Data Service.
  2. In the left panel, under Compartment, select the compartment that hosts your cluster.
  3. In the list of clusters, click the name of the cluster whose nodes you want to examine.
  4. In the list of cluster nodes, click the name of a node to examine.
  5. On the Node Details page, review the details.
    The Node Information box contains this information:
    • Node OCID - The Oracle Cloud ID (OCID) of the node.
    • Node Type - Master, utility, or worker node.
    • Associated BDS Cluster - The name of the Big Data Service cluster containing this node.
    • Shape - The Oracle Infrastructure Compute shape used for this node.
    • Attached Block Storage - The amount of block storage attached to this node.
    • Image - The OCID of the image used for this node.
    • SSH Fingerprint - A sequence of bytes that identifies the SSH public key associated with the node.
    • Launched - When the node was started.

    The Network Information section contains this information:

    • Private IP Address - The private IP address of this node.
    • Subnet - The Virtual Cloud Network (VCN) subnet used for this node.
    • Availability Domain - The Availability Domain in which the cluster that contains the node resides.
    • Fault Domain - The fault domain in which the cluster that contains the node resides.

Viewing Cluster Metrics

You can monitor the health, capacity, and performance of your Big Data resources by using metrics, alarms, and notifications.

Resources: Big Data clusters and cluster nodes.

Required IAM Policy

To monitor resources, you must be given the required type of access in a policy written by an administrator, whether you're using the Cloud console or the REST API with an SDK, CLI, or other tool. The policy must give you access to the monitoring services as well as the resources being monitored. If you perform an action and get a message that you don’t have permission or are unauthorized, confirm with your administrator the type of access you've been granted and which compartment you should work in. For information on user authorizations for monitoring and notifications, see the Authentication and Authorization section for the following services: Monitoring and Notifications.

Available Metrics: oci_big_data_service

The metrics listed in the following table are automatically available for any Big Data cluster that you create. You don't need to enable monitoring on the resource to get these metrics.

Big Data metrics include the following dimensions :

RESOURCEID

The Oracle Cloud ID (OCID) of the Big Data cluster (for cluster metrics)

The Oracle Cloud ID (OCID) of the Big Data node (for node metrics)

RESOURCETYPE

BigDataCluster (for cluster metrics)

BigDataClusterNode (for node metrics)

Metric Metric Display Name Unit Description Resource Type
HdfsSpaceUsed HDFS Space Used Bytes Total HDFS space used on the cluster Cluster
HdfsSpaceFree HDFS Space Free Bytes Total free HDFS space on the cluster Cluster
YarnJobsCompleted Yarn Jobs Completed Jobs/Min Number of YARN jobs completed on this cluster Cluster
SparkJobsCompleted Spark Jobs Completed Jobs/Min Number of Spark jobs completed on this cluster Cluster
DiskUtilization Disk Utilization Bytes Disk space utilized Node
MemoryUtilization Memory Utilization Bytes Total memory used Node
NetworkBytesIn Network Bytes In Bytes/Min Network bytes in per minute Node
NetworkBytesOut Network Bytes Out Bytes/Min Network bytes out per minute Node

View Default Metric Charts in the Console

Default metric charts use predefined service queries. You can select resources of interest and update the interval, statistic, and time range.

To view default metric charts for a cluster:
  1. Open the navigation menu and click Analytics and AI. Under Data Lake, click Big Data Service.
  2. In the left panel, under Compartment, select the compartment that hosts your cluster.
  3. In the list of clusters, click the name of your cluster.
  4. In the left panel, under Resources, click Cluster Metrics. The Cluster Metrics page displays a default set of charts for the selected cluster.
To view default metric charts for a cluster node:
  1. Open the navigation menu and click Analytics and AI. Under Data Lake, click Big Data Service.
  2. In the left panel, under Compartment, select the compartment that hosts your cluster.
  3. In the list of clusters, click the name of the cluster whose nodes you want to examine.
  4. In the list of cluster nodes, click the name of a node to examine.
  5. In the list of cluster nodes, click the name of a cluster node.
  6. The Node Metrics section displays a default set of charts for the selected node.
To view default metric charts for all clusters in a compartment:
  1. Open the navigation menu and click Observability and Management. Under Monitoring, click Service Metrics.
  2. For Compartment, select the compartment that contains the clusters that you're interested in.
  3. For Metric Namespace, select oci_big_data_service. The Service Metrics page dynamically updates the page to show charts for each metric that is emitted by the selected metric namespace.

Optionally, you can specify other dimensions to filter your displayed metrics. For more information, see To filter results and To select different resources.

For more information about monitoring and notifications in Oracle Cloud Infrastructure, see Monitoring Overview and Notifications Overview.

Create Metric Queries in the Console

To create a query for a customized view of your metrics:
  1. Open the navigation menu and click Observability and Management. Under Monitoring, click Metrics Explorer.
    The Metrics Explorer page displays an empty chart with fields to build a query.
  2. Fill in the fields for a new query.

    For more information, see Building Metric Queries.

Connecting to a Cluster Node using SSH

To connect to a Big Data cluster node via a command shell, use Secure Shell (SSH).

An SSH key pair is created when a cluster is created, and the public key is installed on all nodes of the cluster. See Creating a Cluster. For information about creating other key pairs, see Managing Key Pairs on Linux Instances.

Prerequisites
To use SSH to connect to a cluster, you must:
  • Have access to the private SSH key that's associated with a public key assigned to the cluster.

    Note also that permissions on the private key file must allow you read/write/execute access, but prevent other users from accessing the file. For example, to set appropriate permissions, you might enter chmod 600 ~/.ssh/my_keys/my_host_key_filename. If permissions are not set correctly and the private key file is accessible to other users, the ssh utility will simply ignore the private key file.

  • Know the public IP address of the node you want to connect to. You can find the IP address by looking at the Node Details page in the Oracle Cloud Console. See Viewing Cluster Node Details.
  • Ensure port 22 is open. See Define Security Rules.
Connecting to the Cluster by Using SSH at the command line:

To connect to a node in a public subnet:

  1. Use the following command to set the file permissions so that only you can read the file:
    $ chmod 400 <private_key>

    <private_key> is the full path and name of the file that contains the private key associated with the cluster you want to access.

  2. Use the following SSH command to access the cluster.
    $ ssh –i <private_key> <username>@<public-ip-address>

    <private_key> is the full path and name of the file that contains the private key associated with the instance you want to access.

    <username> is the default name for the cluster. The default user name is opc.

    <public-ip-address> is the public IP address of the cluster node you want to access.

Note that if the SSH private key isn't stored in the file or in the path that the ssh utility expects (for example, the ssh utility might expect the private key to be stored in ~/.ssh/id_rsa), you must explicitly specify the private key filename and location in one of two ways:
  • Use the -i option to specify the filename and location of the private key. For example, ssh -i ~/.ssh/my_keys/my_host_key_filename opc@192.0.2.254
  • Add the private key filename and location to an SSH configuration file, either the client configuration file (~/.ssh/config) if it exists, or the system-wide client configuration file (/etc/ssh/ssh_config). For example, you might add the following:Host 192.0.2.254 IdentityFile ~/.ssh/my_keys/my_host_key_filename

For more about the SSH utility’s configuration file, enter man ssh_config

Connect to Nodes in Private Subnets By Using SSH

Worker nodes in private subnets have private IP addresses only (they do not have public IP addresses). They can only be accessed by other resources inside the VCN. Oracle recommends using bastion hosts to control external access (such as SSH) to worker nodes in private subnets. A bastion host is in a public subnet, has a public IP address, and is accessible from the internet. For more information about bastion hosts, see the white paper Bastion Hosts: Protected Access for Virtual Cloud Networks and Bastion documentation.

Connect By Using PuTTY on Microsoft Windows
  1. Open putty.exe.
  2. In the Category pane, expand Window, and then select Translation.
  3. In the Remote character set drop-down list, select UTF-8. The default locale setting on Linux-based instances is UTF-8, and this configures PuTTY to use the same locale.
  4. In the Category pane, select Session and enter the following:

    • Host Name (or IP address):

      <username>@<public-ip-address>

      <username> is the default name for the instance. The default user name is opc.

      <public-ip-address> is your instance public IP address that you retrieved from the Console

    • Port: 22

    • Connection type: SSH

  5. In the Category pane, expand Connection, expand SSH, and then click Auth.
  6. Click Browse, and then select your private key.
  7. Click Open to start the session

    If this is your first time connecting to the instance, you might see a message that the server's host key is not cached in the registry. Click Yes to continue the connection.

Running Administrative Tasks on the Instance

When you’re logged in as the default user, opc, you can use the sudo command to run administrative tasks, such as creating users and groups that will be used to access the cluster.

Connecting to Services on a Cluster using Load Balancer

The Oracle Cloud Infrastructure Load Balancing service provides automated traffic distribution from one entry point to multiple backend servers. You can use a load balancer with Big Data to provide the entry point for accessing services like Apache Ambari, Cloudera Manager, Hue, and Big Data Studio (the Big Data notebook application).

About Load Balancers

A load balancer includes one or more backend sets to route incoming traffic to backend servers. The backend set is a logical entity that includes:

  • A list of backend servers, to which traffic is directed by the load balancer. In Big Data Service, the cluster nodes are the backend servers. A backend server is typically identified by an IP address and a port.
  • A load balancing policy, which tells the load balancer how to distribute incoming traffic to the backend servers.
  • A health check policy, to test the health and availability of backend servers.
  • Optional Secure Sockets Layer (SSL) handling, to establish an encrypted link between a client and a server.
  • Optional session persistence configuration. Session persistence isn't used in the scenario described below.

A load balancer also has one or more listeners, to accept incoming traffic. Listeners route traffic to backend sets, which route traffic to backend servers.

Overview of This Task

These instructions tell how to create a load balancer that can be used as the front end for accessing Apache Ambari or Cloudera Manager, Hue, and Big Data Studio from a web browser. You are able to connect to the services by using the load balancer's IP address or hostname, plus the port where a service runs, for example:

  • https://<load-balancer-ip-or-hostname>:7183 for Apache Ambari or Cloudera Manager
  • https://<load-balancer-ip-or-hostname>:8889 for Hue
  • https://<load-balancer-ip-or-hostname>:30000 for Big Data Studio.

You can create a public load balancer (in a public subnet), which allows access from the public internet; or you can create a private load balancer (in a private subnet), which allows access from your private VCN. For a private load balancer, you must set up private access from the VCN, for example by using a bastion host or by using Oracle FastConnect service or Oracle IPSec VPN service. However, for simplicity, these instructions tell how to create a public load balancer.

Specifically, you do the following:

  • Create a load balancer.
  • Create three backend sets, one for Apache Ambari or Cloudera Manager, one for Hue, and one for Big Data Studio.
  • Add one backend server to each backend set. A backend server is the cluster node where the service runs, plus the port where the service listens for requests.
  • Create three listeners, one for Apache Ambari or Cloudera Manager, one for Hue, and one for Big Data Studio.
  • Configure SSL for the backend sets and for the listeners, thus implementing end-to-end SSL.

For complete documentation about Oracle Cloud Infrastructure load balancers, see Overview of Load Balancing.

Prerequisites
  • A Big Data cluster running in a Virtual Cloud Network (VCN) with an internet gateway and a public regional subnet (for a public load balancer).

  • Security rules that allow incoming traffic to the ports where services run in the cluster: Apache Ambari or Cloudera Manager (port 7183), Hue (port 8889), and Big Data Studio (port 30000). See Define Security Rules.

  • Admin access to manage load balancers. For examples, see "Let network admins manage load balancers" in Common Policies.

  • SSH access to the cluster. You must be able to connect to nodes on your cluster to download self-signed SSL certificates that are stored on the cluster. To do this, prior to creating a load balancer, you must set up your environment to allow that access. For example you can use Oracle FastConnect or Oracle IpSec VPN, you can set up a bastion host, or you can map private IPs to public IP addresses. See Establishing Connections to Nodes with Private IP Addresses.

  • A DNS domain (optional if using self-signed SSL certificates or required if using SSL certificates from a certificate authority (CA)). For information about using DNS in Oracle Cloud Infrastructure, see Overview of the DNS Service.

  • SSL certificates, used for authentication.

    • Self-signed certificates from the cluster are required.

    • Certificates issued by a trusted certificate authority are recommended. You must have a DNS domain (as mentioned above) to obtain certificates from a certificate authority.

    This requirement is explained more fully in step one, below.

  • A list of the private IP addresses of the nodes where the services run on your cluster.

    • In a highly available (HA) cluster, the services run on the following nodes:

      • Apache Ambari or Cloudera Manager runs on the first utility node.
      • Hue and Big Data Studio run on the second utility node.
    • In a non-HA cluster, all the services run on the first (and only) utility node.

STEP 1 (Ambari): Obtain SSL Certificates for Apache Ambari

If you created your cluster with Cloudera Manger, then skip this step.

This topic tells you how to implement end-to-end SSL for your load balancer. The load balancer will accept SSL encrypted traffic from clients, and it will encrypt traffic to the backend servers. To do this, you must associate SSL certificate bundles with the listeners and with the backend servers.

  • For listeners, you can use signed SSL certificates issued by a trusted certificate authority or you can use self-signed certificates that you download from the cluster. Oracle recommends that you use signed SSL certificates from a certificate authority to provide the highest level of security.
    Note

    This topic uses the phrases "CA-signed SSL certificate" or just "CA-signed certificate" to refer to signed SSL certificates obtained from a certificate authority.
  • For backend sets, use the self-signed certificates that you download from the cluster.

Obtain one of the following certificates:

  • Certificate-Authority (CA)-signed SSL certificates
  • Self-signed SSL certificates

You can purchase CA-signed SSL certificates from a trusted certificate authority such as Symantec, Thawte, RapidSSL, or GeoTrust. See the documentation from your chosen certificate authority for information about obtaining and managing certificates.

To obtain SSL certificates from a certificate authority, you usually have to have a DNS domain. For information about using DNS in Oracle Cloud Infrastructure, see Overview of the DNS Service.

Self-signed SSL certificates are included in your cluster. They're located in the following directory in each utility node:

/var/lib/ambari-server/keys

List the contents of the /var/lib/ambari-server/keys directory.

[opc@<your-cluster-name>-un0 (or -un1) ~]$ ls /var/lib/ambari-server/keys
...
https.crt
https.key
...

From the /var/lib/ambari-server/keys directory you can find:

  • Public certificate: https.crt
  • Private SSL key: https.key

Note

Each node has a unique https.key and https.crt file. The file names are the same in each node, but the contents are different for each node.
Note

Download the certificates before you proceed to the next step. For an HA cluster, download the SSL certificates and keys from the first and second utility nodes. For a non-HA cluster, download the SSL certificate and key from the first (and only) utility node.

For more information about using SSL with load balancers, see SSL Certificate Management.

STEP 1 (Cloudera): Obtain SSL Certificates for Cloudera Manager

If you created your cluster with Apache Ambari, then skip this step.

This topic tells you how to implement end-to-end SSL for your load balancer. The load balancer will accept SSL encrypted traffic from clients, and it will encrypt traffic to the backend servers. To do this, you must associate SSL certificate bundles with the listeners and with the backend servers.

  • For listeners, you can use signed SSL certificates issued by a trusted certificate authority or you can use self-signed certificates that you download from the cluster. Oracle recommends that you use signed SSL certificates from a certificate authority to provide the highest level of security.
    Note

    This topic uses the phrases "CA-signed SSL certificate" or just "CA-signed certificate" to refer to signed SSL certificates obtained from a certificate authority.
  • For backend sets, use the self-signed certificates that you download from the cluster.

Obtain one of the following certificates:

  • Certificate-Authority (CA)-signed SSL certificates
  • Self-signed SSL certificates

You can purchase CA-signed SSL certificates from a trusted certificate authority such as Symantec, Thawte, RapidSSL, or GeoTrust. See the documentation from your chosen certificate authority for information about obtaining and managing certificates.

To obtain SSL certificates from a certificate authority, you usually have to have a DNS domain. For information about using DNS in Oracle Cloud Infrastructure, see Overview of the DNS Service.

Self-signed SSL certificates are included in your cluster. They're located in the /opt/cloudera/security/x509 directories of your utility nodes:

Example contents of an /opt/cloudera/security/x509 directory:

[opc@<your-cluster-name>un0 ~]$ ls -1 /opt/cloudera/security/x509/
agents.pem
hostname.key
hostname.pem
hue.pem
node_<your-cluster-name>mn0.subnetxxx.clustervcn.oraclevcn.com.pem
node_<your-cluster-name>mn1.subnetxxx.clustervcn.oraclevcn.com.pem
node_<your-cluster-name>un0.subnetxxx.clustervcn.oraclevcn.com.pem
node_<your-cluster-name>un1.subnetxxx.clustervcn.oraclevcn.com.pem
node_<your-cluster-name>wn0.subnetxxx.clustervcn.oraclevcn.com.pem
node_<your-cluster-name>wn1.subnetxxx.clustervcn.oraclevcn.com.pem
node_<your-cluster-name>wn2.subnetxxx.clustervcn.oraclevcn.com.pem
node.cert
node.hue.key
node.key

where:

  • Cluster name: <your-cluster-name>
  • Node name: <your-cluster-name><mn,un,wn><0,1>
  • mn represents master node
  • un represents utility node
  • wn represents worker node
  • 0 means the first node

From the /opt/cloudera/security/x509 directory you can find:

  • Public certificate: /opt/cloudera/security/x509/<utility_node_certificate>.pem.

    Example: node_<your-cluster>un0.subnetxxx.clustervcn.oraclevcn.com.oraclevcn.com.pem

    The <utility_node_certificate>.pem files for all the nodes are stored in the above directory on all the nodes. That is, you can access and download all the files from a single node.

  • Private SSL key: /opt/cloudera/security/x509/node.hue.key.

Note

Each node has a unique node.hue.key file. The file names are all the same, but the contents are different for each node.
Note

Download the certificates before you proceed to the next step. For an HA cluster, download the SSL certificates and keys from the first and second utility nodes. For a non-HA cluster, download the SSL certificate and key from the first (and only) utility node.

For more information about using SSL with load balancers, see SSL Certificate Management.

STEP 2: Create the Load Balancer
  1. Open the navigation menu and click Networking. Then click Load Balancers.
  2. In the left panel, under Compartment, select a compartment select the compartment where you want to create the load balancer. Then click Create Load Balancer.
  3. For Select Load Balancer Type, click Load Balancer and then click Create Load Balancer.
  4. On the Add Details page of the Create Load Balancer wizard, enter the following information:

    • Load Balancer Name: Enter a name to identify the load balancer.

    • Choose Visibility Type: Click Public to create a load balancer that will be accessible from the public internet.

      You could choose instead to select Private to create a load balancer that will be accessible only from your private network, but for simplicity these instructions describe how to create a public load balancer.

    • Choose Total Bandwidth: Select Small.

    • Virtual Cloud Networking in <compartment>: Click the Select a virtual cloud network list and select the VCN where your cluster is running. If the network is in a different compartment, click Change Compartment and select the compartment from the list.

    • Subnet in <compartment>: Click the Select a subnet list and select a public subnet in your VCN to use for the load balancer. (A public subnet is required for a public load balancer.) If the subnet is in a different compartment, click Change Compartment and select the compartment from the list.

    • Use Network Security Groups to Control Traffic: Leave this box unchecked.

  5. Click Next.
  6. On the Choose Backends page of the wizard, enter the following information to create a backend set (with health policy) for Apache Ambari or Cloudera Manager:

    • Specify a Load Balancing Policy: Accept the default Weighted Round Robin.

    • Select Backend Servers: Skip this option. You'll add a backend server later.
    • Specify Health Check Policy: Enter the following for the health check policy for this Apache Ambari or Cloudera Manager backend set:

      • Protocol: Select HTTP.

      • Port: Enter 7183, which is the port on which Apache Ambari or Cloudera Manager listens.

      • URL Path (URI): Keep the default forward slash (/).

    • Use SSL: Leave this box unchecked. You'll configure SSL for this backend set later.
  7. Click Next.

  8. On the Configure Listener page, enter the following information:

    • Listener Name: Enter a name for the listener, for example cm-listener.

    • Specify the type of traffic your listener handles: Select HTTP. You'll change this to HTTPS later.

    • Specify the Port Your Listener Monitors for Ingress Traffic: Enter 7183.

  9. Click Submit. When the large load balancer status icon load balancer status icon at the top of the Load Balancer Details page is green, you can continue with the steps below. It may take a few minutes to create the load balancer.

STEP 3: Create Certificate Bundles

In this step, you'll save your SSL files as bundles, which you'll use later to configure SSL for backend sets and listeners. You'll create different bundles, depending on the type of cluster and the source of your SSL certificates. The use cases are:

  • Use case A: HA cluster that uses a CA-signed certificate
  • Use case B: Non-HA cluster that uses a CA-signed certificate
  • Use case C: HA cluster that uses only the self-signed certificates from the cluster
  • Use case D: Non-HA cluster that uses only a self-signed certificate from the cluster

Each use case requires one to three certificate bundles. The table below shows all possible bundles, although you only have to create the ones required for your use case. The instructions following the table tell you which bundles to create for your use case.

Note

The names used for the bundles in the table below are user-supplied names. You can provide different names when you create the bundles, but the names below are used throughout this topic for clarity.

Node names are node0 for the first node and node1 for the second node.

Certificate Bundles
Bundle Contents
lb-ca-signed-cert
  • CA-signed SSL certificate, obtained from a certificate authority, for the listener on the load balancer host
  • CA certificate from your certificate authority
  • Private key associated with SSL certificate, above
node0-self-signed-cert
  • Self-signed SSL certificate for the first utility node, downloaded from the cluster
  • Private key downloaded from first utility node
node1-self-signed-cert
  • Self-signed SSL certificate for the second utility node, downloaded from the cluster
  • Private key downloaded from second utility node
Note

In several steps while creating the load balancer, you'll be prompted to enter certificates and keys. The Cloud Console provides alternative ways to add them. For example, the options for adding an SSL certificate are:

  • Choose SSL Certificate File: If you select this option, you can either drag a file from your local file system directly into the SSL Certificate box, or you can click select one and navigate to select the file on your file system.

  • Paste SSL Certificate: If instead you select this option, open the appropriate file, copy the entire contents, and paste it into the SSL Certificate box.

Similar options are presented for CA Certificate and Private Key, and the same instructions apply. When those options appear in the console, the instructions below simply say, "add the certificate..." or "add the key..."

To create the bundles:

  1. On the left side of the Load Balancer Details page, under Resources, click Certificates.

    Refer to the Steps to Complete for Each Use Case table, below, to find your use case. Then refer to the Complete These Steps column to find which of the subsequent steps you must follow to create the bundle or bundles for your use case.

    Notice that:

    • Use case A requires three bundles; use cases B and C each require two bundles; and use case D requires only one bundle.
    • All the use cases require one or two bundles with self-signed certificates. Use cases A and B also require a bundle with a CA-signed certificate.

    Steps to Complete for Each Use Case
    Use Case Type of Cluster CA-signed certificate? Complete These Steps
    A HA Yes

    2: Create a bundle with a CA-signed certificate

    3: Create a bundle with the self-signed certificate from the first utility node

    4: Create a bundle with the self-signed certificate from the second utility node

    B Non-HA Yes

    2: Create a bundle with a CA-signed certificate

    3: Create a bundle with the self-signed certificate from the first utility node

    C HA No

    3: Create a bundle with the self-signed certificate from the first utility node

    4: Create a bundle with the self-signed certificate from the second utility node

    D Non-HA No

    3: Create a bundle with the self-signed certificate from the first utility node

  2. Create a bundle with a CA-signed certificate. You must create this bundle if you have either of the following use cases:

    • Use Case A: HA cluster with a CA-signed certificate
    • Use Case B: Non-HA cluster with a CA-signed certificate

    Create the bundle:

    1. Click Add Certificate.

    2. On the Add Certificate page, enter the following information:

      • Certificate Name: Enter lb-ca-signed-cert (or a name of your choice).
      • SSL Certificate: Add your CA-signed SSL certificate.

      • Specify CA Certificate: Check this box, and then add your CA certificate.

      • Specify Private Key: Check this box, and then add the private key associated with the SSL certificate you added in the SSL Certificate field, above.

    3. Click Add Certificate, and then click Close in the Work Request Submitted dialog box. It may take a few moments for the bundle to be added to the Certificates table.
  3. Create a bundle with the self-signed certificate from the first utility node. You must create this bundle for all the use cases:

    • Use Case A: HA cluster with a CA-signed certificate
    • Use Case B: Non-HA cluster with a CA-signed certificate
    • Use Case C: HA cluster with only the self-signed certificates from the cluster
    • Use Case D: Non-HA cluster with only a self-signed certificate from the cluster

    Create the bundle:

    1. Remain on the Certificates page. Click Add Certificate.

    2. On the Add Certificate page, enter the following information:

      • Certificate Name: Enter node0-self-signed-cert (or a name of your choice).
      • SSL Certificate: Add the self-signed SSL certificate for the first utility node that you downloaded from the cluster.

      • Specify CA Certificate: Check this box, and then add the same self-signed SSL certificate that you just added for SSL Certificate, directly above.

      • Specify Private Key: Check this box, and then add the private key associated with the SSL certificate you added in the SSL Certificate field, above, that is, the node.hue.key you downloaded from the first utility node.

    3. Click Add Certificate, and then click Close in the Work Request Submitted dialog box. It may take a few moments for the bundle to be added to the Certificates table.
  4. Create a bundle with the self-signed certificate from the second utility node. You must create this bundle if you have either of the following use cases:

    • Use Case A: HA cluster with a CA-signed certificate
    • Use Case C: HA cluster with only self-signed certificates from the cluster

    Create the bundle:

    1. Remain on the Certificates page. Click Add Certificate.

    2. On the Add Certificate page, enter the following information:

      • Certificate Name: Enter node1-self-signed-cert (or a name of your choice).
      • SSL Certificate: Add the self-signed SSL certificate for the second utility node that you downloaded from the cluster.

      • Specify CA Certificate: Check this box, and then add the same self-signed SSL certificate that you just added for SSL Certificate, directly above.

      • Specify Private Key: Check this box, and then add the private key associated with the SSL certificate you added in the SSL Certificate field, above, that is, the node.hue.key you downloaded from the second utility node.

    3. Click Add Certificate, and then click Close in the Work Request Submitted dialog box. It may take a few moments for the bundle to be added to the Certificates table.
STEP 4: Configure the Backend Set for Apache Ambari or Cloudera Manager
  1. On the left side of the Certificates page, under Resources, click Backend Sets. The backend set you created in STEP 2: Create the Load Balancer is displayed in the Backend Sets table, with a name like bs-lb-<date-timestamp>, for example, bs_lb_2020-0928-1136.

  2. Click the action menu at the end of the row containing this backend set, and select Edit.

  3. Enter the following information.

    • Name: Read only. This name was created for you by the wizard.
    • Traffic Distribution Policy: Accept the default Weighted Round Robin.
    • Use SSL: Select this box, then, under Certificate Name, select node0-self-signed-cert. Use this bundle for all use cases. It's the bundle containing the self-signed SSL certificate for the first utility node of the cluster.

    • Verify Peer Certificate: Check this box.
    • Verify Depth: Set to 1, or select another number to indicate maximum depth for certificate chain verification.
    • Session Persistence: Accept the default Disable Session Persistence.
  4. Click Update Backend Set, and then click Close in the Work Request Submitted dialog box. When complete, a cipher suite name is added to the Cipher Suite field for the backend set. It may take a few moments.

STEP 5: Create a Backend Set for Hue
  1. Remain on the Backed Sets page and click Create Backend Set again. On the Create Backend Set page, enter the following information.
    • Name: Provide a name, for example, hue-backend-set.

    • Traffic Distribution Policy: Accept the default Weighted Round Robin.

    • Use SSL: Select this box, then, under Certificate Name, select the certificate bundle for your use case:

      • For HA clusters, select node1-self-signed-cert.

        This is the bundle containing the self-signed SSL certificate for the second utility node of the cluster.

      • For non-HA clusters, select node0-self-signed-cert.

        This is the bundle containing the self-signed SSL certificate for the first utility node of the cluster.

    • Verify Peer Certificate: Check this box.
    • Verify Depth: Set to 1, or select another number to indicate maximum depth for certificate chain verification.
    • Session Persistence: Accept the default Disable Session Persistence.
    • Health Check: Enter the following information:

      • Protocol: Select TCP.

      • Port: Enter 8889, which is the port on which Hue listens.

  2. Click Create Backend Set, and then click Close in the Work Request Submitted dialog box. It may take a few moments for the backend set to be added to the Backend Sets table.
STEP 6: Create a Backend Set for Big Data Studio
  1. Remain on the Backend Sets page and click Create Backend Set again. On the Create Backend Sets page, enter the following information.

    • Name: Provide a name, for example, data-studio-backend-set.
    • Traffic Distribution Policy: Accept the default Weighted Round Robin.
    • Use SSL: Select this box, then, under Certificate Name, select the certificate bundle for your use case:

      • For HA clusters, select node1-self-signed-cert.

        This is the bundle containing the self-signed SSL certificate for the second utility node of the cluster.

      • For non-HA clusters, select node0-self-signed-cert.

        This is the bundle containing the self-signed SSL certificate for the first utility node of the cluster.

    • Verify Peer Certificate: Check this box.
    • Verify Depth: Set to 1, or select another number to indicate maximum depth for certificate chain verification.
    • Session Persistence: Accept the default Disable Session Persistence.
    • Health Check: Enter the following information:

      • Protocol: Select HTTP.

      • Port: Enter 30000, which is the port on which Big Data Studio listens.

      • URL Path (URI): Enter a forward slash (/).

  2. Click Create Backend Set, and then click Close in the Work Request Submitted dialog box. It may take a few moments for the backend set to be added to the Backend Sets table.

STEP 7: Add a Backend Server for Apache Ambari or Cloudera Manager
  1. Remain on the Backend Sets page. In the Backend Sets table, click the name of the backend set for Apache Ambari or Cloudera Manager, for example bs_lb_2020-0928-1136. (Remember, the wizard assigned this name to the first backend set.)
  2. On the left side of the Backend Set Details page, under Resources, click Backends. Then click Add Backends.
  3. On the Add Backends page, select IP Addresses at the top of the page, and enter the following information:
    • IP Address: Enter the private IP address of the first utility node of your cluster, for example 10.2.0.106.
    • Port: Enter 7183, which is port where Apache Ambari or Cloudera Manager listens.
    • Weight: Accept the default value 1.
  4. Click Add, and then click Close in the Work Request Submitted dialog box. It may take a few moments for the backend server to be added to the Backends table.
STEP 8: Add a Backend Server for Hue
  1. Click Backend Sets in the breadcrumbs at the top of the page to return to the Backend Sets page. In the Backend Sets table, click the name of the backend set you created for Hue, for example hue-backend-set.
  2. On the left side of the Backend Set Details page, under Resources, click Backends. Then click Add Backends.
  3. On the Add Backends page, select IP Addresses at the top of the page, and enter the following information:
    • IP Address: Enter the private IP address of the second utility node for an HA cluster or the first (and only) utility node for a non-HA cluster, for example 10.2.0.100.
    • Port: Enter 8889, which is the port on which Hue listens.
    • Weight: Accept the default value 1.
  4. Click Add, and then click Close in the Work Request Submitted dialog box. It may take a few moments for the backend server to be added to the Backends table.
STEP 9: Add a Backend Server for Big Data Studio
  1. Click Backend Sets in the breadcrumbs at the top of the page to return to the Backend Sets page. In the Backend Sets table, click the name of the backend set you created for Big Data Studio, for example data-studio-backend-set.

  2. On the left side of the Backend Set Details page, under Resources, click Backends. Then click Add Backends.

  3. On the Add Backends page, select IP Addresses at the top of the page, and enter the following information:

    • IP Address: Enter the private IP address of the second utility node for an HA cluster or the first (and only) utility node for a non-HA cluster, for example 10.2.0.106.
    • Port: Enter 30000, for the port where Big Data Studio listens.
    • Weight: Accept the default value 1.
  4. Click Add, and then click Close in the Work Request Submitted dialog box. It may take a few moments for the backend server to be added to the Backends table.

STEP 10: Configure the Listener for Apache Ambari or Cloudera Manager
  1. Click Load Balancer Details in the breadcrumbs at the top of the page. On the left side of the Load Balancer Details page, under Resources, click Listeners. Notice that the Listeners table includes the listener you created for Apache Ambari or Cloudera Manager in STEP 2: Create the Load Balancer, for example, cm-listener.
  2. Click the action menu at the end of the row containing the listener, and select Edit.

  3. On the Edit Listener page, enter the following information:
    • Name: Read only.

    • Protocol: Select HTTP.

    • Port: Enter 7183, which is the port on which Apache Ambari or Cloudera Manager listens.

    • Use SSL: Select this box, then, under Certificate Name, select the certificate bundle for your use case:

      • If you're using CA-signed SSL certificates, select: lb-ca-signed-cert

        This is the bundle containing the CA-signed SSL certificate. Use this for both HA and non-HA clusters.

      • If you're using the self-signed certificate from the cluster only, select: node0-self-signed-cert

        This is the bundle containing the self-signed SSL certificate for the first utility node of the cluster. Use this for both HA and non-HA clusters.

    • Verify Peer Certificate : Leave this box unchecked.

    • Backend Set: From the list, select the backend set you created for Apache Ambari or Cloudera Manager in STEP 2: Create the Load Balancer, for example, bs_lb_2020-0928-1136. (Remember, the wizard assigned this name when you created it.)

  4. Click Create Listener, and then click Close in the Work Request Submitted dialog box. It may take a few moments for the listener to be added to the Listeners table.
STEP 11: Create a Listener for Hue
  1. Click Backend Sets in the breadcrumbs at the top of the page to return to the Backend Sets page.
  2. On the left side of the page, under Resources, click Listeners.
  3. Click Create Listener.
  4. On the Create Listener page, enter the following information:
    • Name: Enter a name for the listener, for example, hue-listener.

    • Protocol: Select HTTP.

    • Port: Enter 8889, which is the port on which Hue listens.

    • Use SSL: Select this box, then, under Certificate Name, select the certificate bundle for your use case:

      • If you're using a CA-signed certificate, select lb-ca-signed-cert.

        This is the bundle containing the CA-signed SSL certificate for the load balancer host.

      • If you're using only self-signed certificates with an HA cluster, select: node1-self-signed-cert,

        This is the bundle containing the self-signed SSL certificate for the second utility node of the cluster.

      • If you're using only the self-signed certificate with a non-HA cluster, select node0-self-signed-cert.

        This is the bundle containing the self-signed SSL certificate for the first utility node of the cluster.

    • Verify Peer Certificate : Leave this box unchecked.

    • Backend Set: From the list, select the backend set you created for Hue in STEP 5: Create a Backend Set for Hue, for example, hue-backend-set.

  5. Click Create Listener, and then click Close in the Work Request Submitted dialog box. It may take a few moments for the listener to be added to the Listeners table.
STEP 12: Create a Listener for Big Data Studio
  1. Remain on the Listeners page. Click Create Listener again.

  2. On the Create Listener page, enter the following information:

    • Name: Enter a name for the listener, for example, data-studio-listener.

    • Protocol: Select HTTP.

    • Port: Enter 30000, which is the port on which Big Data Studio listens.

    • Use SSL: Select this box, then, under Certificate Name, select the certificate bundle for your use case:

      • If you're using a CA-signed certificate, select lb-ca-signed-cert.

        This is the bundle containing the CA-signed SSL certificate for the load balancer host.

      • If you're using only self-signed certificates with an HA cluster, select: node1-self-signed-cert,

        This is the bundle containing the self-signed SSL certificate for the second utility node of the cluster.

      • If you're using only the self-signed certificate with a non-HA cluster, select node0-self-signed-cert.

        This is the bundle containing the self-signed SSL certificate for the first utility node of the cluster.

    • Verify Peer Certificate : Leave this box unchecked.

    • Backend Set: From the list, select the backend set you created for Big Data STEP 6: Create a Backend Set for Big Data Studio Studio in, for example, data-studio-backend-set.

  3. Click Create Listener, and then click Close in the Work Request Submitted dialog box. It may take a few moments for the listener to be added to the Listeners table.

STEP 13: Access the Cluster

It may take a few minutes for the backend sets and listeners to be ready to receive requests. On the left side of the page, under Resources, click Backend Sets. In the Backend Sets table, check the status in the Health column. When the status for all the backend sets are OK OK, you can use the load balancer.

Tip

If it's taking a long time for the health check, consider shortening the interval. Click the action menu at the end of the row containing a backend set, and select Update Heath Check. Change Interval in Ms to 1000 (the minimum interval) and Timeout in Ms to 500. Repeat for each backend set. You can change the settings later if you want the health checks to be performed less often.

To open the services included in this load balancer:

  1. Find the IP address or the hostname used for your load balancer.

    • IP address:

      The IP address is listed in the Load Balancer Information panel at the top of the load balancer pages in the console.

    • DNS hostname:

      After the load balancer is created and it's been given an IP address, you or another administrator must add a DNS entry to your DNS name servers, to resolve your desired hostname (for example, bds-frontend.mycompany.com) to the public IP address of the load balancer. Then, the services registered in the load balancer will be accessible by using that hostname, for example, bds-frontend.mycompany.com:7183 for Apache Ambari or Cloudera Manager.

      For information about using DNS in Oracle Cloud Infrastructure, see Overview of the DNS Service.

  2. In a web browser, enter the address as follows:

    • To use the load balancer's IP address: https://<load-balancer-ip>:<port>.

    • To use the load balancer's hostname in a domain: https://<hostname>:<port>.

    That is, for Apache Ambari or Cloudera Manager:
    • https://<load-balancer-ip>:7183
    • https://<hostname>:7183
    For Hue:
    • https://<load-balancer-ip>:8889
    • https://<hostname>:8889
    For Big Data Studio:
    • https://<load-balancer-ip>:30000
    • https://<hostname>:30000