1. Using Oracle Infrastructure Monitoring
  2. Discovery
  3. Add Hadoop Cluster

Add Hadoop Cluster

You can add Hadoop Cluster entities using the Add Entity UI or using the cloud agent command line interface (omcli) with the appropriate JSON files.

Step 1: Prepare Hadoop Cluster for monitoring.

Prerequisites

By default, Hadoop runs in non-secure mode in which no actual authentication is required.

By configuring Hadoop to run in secure mode, each user and service needs to be authenticated by Kerberos in order to use Hadoop services.

To perform Kerberos authentication, the Cloud Agent requires the following:

  1. krb5.conf file. This file can be found at /etc/krb5.conf

  2. Username and password

The Cloud Agent can use only one krb5.conf at a time. If a single Agent needs to perform Kerberos authentication with more than one domain, these details should be defined in a single krb5.conf file.

Step 2: Decide how you want to add the Hadoop Cluster.

You can add Hadoop Cluster entities using one of two ways:

  • Add them from UI
  • Use the agent's omcli add_entity command with the appropriate JSON files

Adding Entities from the UI

  1. From the Management Cloud main menu, select Administration, Discovery, and then Add Entity. The Add Entity page displays.
  2. Select the Hadoop Clusters Entity Type.
  3. Enter the following UI properties.
    Hadoop UI Fields
    • Discover Using Credentials: Discover Hadoop using Hadoop credentials (on by default).
    • Entity Name: Name of this entity displayed in the Oracle Management Cloud console.
    • Metric URL for NameNode 1: Name Node 1 URL with port..
    • Metric URL for NameNode 2: Name Node 2 URL with port.
    • Metric URL for Resource Manager 1: Resource Manager Node 1 URL with port.
    • Metric URL for Resource Manager 2: ResourceManager Node 2 URL with port.
    • Cloud Agent: Cloud agent monitoring the host where Hadoop is installed.

    Monitoring Credentials

    SSL Trust Store

    • Store Location: Path of the Truststore file.
    • Store Password: The keystore password to access the JKS file.
    • Store Type: Currently, only JKS is supported.

    Alias

    • Alias Name: Your alias name.
    • Password: Your alias password.
    • Path of krb5.conf file: Full path to the krb5.conf file.

See Add Entities from the Console for detailed instructions on using the Add Entity UI.

Using omcli and the Appropriate JSON Files

  1. Download and extract the required JSON file(s) from the master JSON zip file. See the table below for the specific JSON files you'll need.
  2. Edit the file(s) and specify the requisite properties shown below.
    Hadoop JSON Files and Properties

    Note:

    Hadoop uses primary and secondary nodes to maintain availability. If one node goes down, the secondary node is used to fetch information. For both nodes, the primary and secondary roles can be switched at any given time. For this reason, you specify two nodes and two Resource Manager URLs without specifying whether they are primary or secondary.

    Definition File: hadoop_credless.json

    Use this file if Hadoop was configured with no credentials.

    • Name: Hadoop entity name.
    • displayName: Hadoop entity display name.
    • omc_nn1_metric_url (Name Node 1): Name Node 1 URL with port.
    • omc_nn2_metric_url (Name Node 2): Name Node 2 URL with port.
    • omc_rm1_metric_url (Resource Manager 1): Resource Manager Node 1 URL with port.
    • omc_rm2_metric_url (Resource Manager 2): ResourceManager Node 2 URL with port.

    Example URL

    http://<HOSTNAME>:<PORT>/ (Rest API URL for
          Invocation)

    Definition File:hadoop_creds.json

    • name: Hadoop entity name.
    • displayName: Hadoop entity display name.
    • credentialRefs: Hadoop credential information (hadoopTrustStore and hadoopSPNEGOCredentials) defined in the hadoop_credentials_input.json file.
    • omc_nn1_metric_url: (Name Node 1): Name Node 1 URL with port.
    • omc_nn2_metric_url: (Name Node 2): Name Node 2 URL with port.
    • omc_rm1_metric_ur:l (Resource Manager 1): Resource Manager Node 1 URL with port.
    • omc_rm2_metric_ur:l (Resource Manager 2): ResourceManager Node 2 URL with port.

    Example URL

    http://<HOSTNAME>:<PORT>/ (Rest API URL for Invocation)

    Credential File:hadoop_credentials_input.json

    • hadoopTrustStore consists of the following user-defined properties:
      • StoreLocation: Path of Truststore file.
      • StorePassword: Truststore Password
    • hadoopSPNEGOCredentialsconsists of the following user-defined properties:
      • Alias: Alias name.
      • Password: Alias password.
      • KRB5Conf: Path of krb5.conf file.
  3. Add the entity using omcli.
    omcli add_entity agent DEFINITION_FILE [-credential_file CREDENTIAL_FILE [-encryption_method_gpg]]
  4. Verify the status of the newly added entity. 
    omcli status_entity agent DEFINITION_FILE

See step 4. Adding Entities to Your Service of Add Entities Using JSON Files for more information.

Step 3: (Optional but recommended) Set up alerts.

To enable lights-out monitoring, you can set up alert rules to generate alerts and send notifications if your entities have performance issues.

See Set Up Alert Rules and Set Up Alert Thresholds and Notifications.

Troubleshooting

If you run into any issues regarding discovery or monitoring of Hadoop Clusters, see the following: