Exadata Prerequisites

These are Exadata prerequisites that must be completed prior to discovery and monitoring.

Create a Database Server ILOM Service Processor User

For the Enterprise Manager Agent to communicate with an ILOM service processor, there must be a specific user ID established on the ILOM service processor.

Note:

Adding the specific user ID requires administrator level privilege on the ILOM service processor.

The specific ILOM user ID can be added in the ILOM service processor web interface, ILOM CLI, or with the ipmitool command. This example uses ILOM CLI.

For security reasons, the password to the ILOM service processor root user ID does not appear in the ILOM CLI commands in this example.

  1. Log in to the Service Processor as root:
    # ssh root@[Service Processor IP]
Password:
  2. Change to the users directory:
    # cd /SP/users
  3. Create the oemuser user and password:
    # create oemuser

Creating user...
Enter new password: ********
Enter new password again: ********

Created /SP/users/oemuser
  4. Change to the new user's directory and set the role:
    # cd oemuser
/SP/users/oemuser

set role='cro'
Set 'role' to 'cro'
  5. Test the ILOM user ID created in step 3 by listing the last 10 system events:

    • For Exadata X2 through X4:

      # ipmitool -I lan -H <ilom_hostname> -U oemuser -P <oemuser password> -L USER sel list last
      10\

    • For Exadata X5 and above (requires the -I lanplus command option): 

      # ipmitool -I lanplus -H <ilom_hostname> -U oemuser -P <oemuser password> -L USER sel list last 10
  6. Repeat steps 1 through 5 for the rest of the compute node ILOM service processors in your Oracle Database Machine.

Create an ExaCLI or RESTful API User

Enterprise Manager can monitor Exadata Storage Servers using cellcli, exacli, or RESTful API. In order to monitor and/or manage Exadata Storage Servers through exacli or the RESTful API, a user must be created and provided for Exadata Database Machine discovery. For more information on exacli or RESTful API user creation, see the Cell node sections in both Example 9-9 and Example 9-10 in Creating Users for Use with ExaCLI in Oracle Exadata Database Machine Maintenance Guide. These examples provide the steps to create both the celladministrator user and the cellmon user. Review the following best practice guidelines/requirements to determine if the celladministrator user is required:

  • As a best practice of using the least privileged user for an operation, always create and use the cellmon user for the monitoring credentials.

  • Create the celladministrator user only if one or both of these activities are required:

    • Set up of SNMP subscriptions for the Exadata Storage Servers

    • IORM administration

  • During discovery, for the Exadata Storage Server credentials, the following are the best practice recommendations/requirements:

    • Supply the cellmon user credentials if selecting exacli or RESTful API for monitoring

    • If setting up SNMP subscriptions, supply the celladministrator credentials for the SNMP setup (the celladministrator credential is required)

      Note:

      If SNMP subscriptions are not configured, then Enterprise Manager will not monitor key events such as hardware failures.

  • While making IORM change through Enterprise Manager, use the celladministrator credential.

Create an Exascale Monitoring User

Starting with Exadata System Software version 24.1, Enterprise Manager offers the capability to monitor Oracle Exadata Exascale. To monitor Exascale, create a user on the cluster with appropriate privileges. The user authentication uses public-private key pairs.

For the purpose of this example and throughout this document, the monitoring user will be esmonitor. See Oracle Exadata Exascale User-Specific Administration in Exascale User's Guide.

For use with Exascale, the public and private keys must be in PEM format. To create a key pair, use standard utilities such as openssl or the ESCLI utility on the Exadata compute node:

  1. Create a directory for your monitoring key pair:
    mkdir /home/oracle/eskey
  2. Protect this directory from access by other users:
    chmod 700 /home/oracle/eskey
  3. Generate your key using ESCLI:
    /opt/oracle/dbserver/dbms/bin/escli
    mkkey --private-key-file /home/oracle/eskey/esmonitor-priv.pem --public-key-file /home/oracle/eskey/esmonitor-pub.pem
  4. Protect your key from unauthorized access:
    chmod 600 /home/oracle/eskey/esmonitor-priv.pem /home/oracle/eskey/esmonitor-pub.pem
  5. Provide your public key to the Exadata Exascale administrator to provision your monitoring user.
  6. As the Exadata Exascale administrator, create the monitoring user, grant monitoring privileges and configure with the user's public key:
    escli --wallet <admin wallet location> ----ctrl <ExaCTRL server"port>
    mkuser esmonitor --id esmonitor
  7. Grant privileges to monitor the cluster and all vaults in the cluster:
    chuser esmonitor --privilege cl_monitor
    chuser esmonitor ---privilege +vlt_read
  8. Associate the public key with the user:
    chuser esmonitor --public-key-file1 esmonitor-pub.pem

Create SNMPv3 Users

If SNMPv3 will be used for Oracle Exadata Database Machine monitoring, ensure that the necessary SNMPv3 users are created on the components prior to discovering the Exadata to fully leverage monitoring through Enterprise Manager.

Note:

Oracle does not recommend to use MD5 as the AuthProtocol while creating SNMPv3 user for security reasons.

Create SNMPv3 Users on Compute Nodes and Storage Servers

The commands required to create SNMPv3 users on compute nodes and storage servers are similar, but make use of different command line interfaces and object names. In the following examples, run cellcli to get to the interactive prompt if on a storage server, and run dbmcli to get to the interactive prompt on a compute node. Specify the appropriate object name, cell and dbserver for compute node. The instructions differ between Exadata System Software releases. Please see the details for the respective Exadata System Software version in the sections below.

Note:

For additional information on these commands, please see the following references:
  • For information on CellCLI and the alter cell command, see Using the CellCLI Utility in the Oracle Exadata System Software User’s Guide.
  • For information on DBMCLI and the alter dbserver command, see Using the DBMCLI Utility in the Oracle Exadata Database Machine Maintenance Guide.
Create Individual SNMPv3 Users on Exadata 19.3 and Above

Exadata System Software version 19.3 and above supports maintaining SNMPv3 users individually. Use the following command to create an SNMPv3 user.

CLI> alter <cell|dbserver> snmpuser.<username> =(authprotocol=SHA,authpassword=<password>,privprotocol=AES,privpassword=<password>)

In the above command,

  • name: The user name

  • * is the only allowed value for the password

  • authProtocol: The authentication protocol. The supported protocol is SHA. The system prompts for the authentication password. The authentication password must have 8 to 12 alphanumeric characters.

  • privProtocol: Encryption protocol. The options supported are none, AES, and DES. The default value is none when privProtocol attribute is not specified. The system prompts for an encryption password if the encryption protocol is specified. The password must be exactly 8 alphanumeric characters long, and is case sensitive.

To change the password for an existing user:

CLI> alter <cell|dbserver> snmpuser.<username> =(authprotocol=SHA,authpassword=<newpassword>)

To delete a user:

CLI> alter <cell|dbserver> snmpuser.<username> =''
Create All SNMPv3 Users on Exadata 19.2. and Below

Exadata System Software versions 19.2 and below require maintenance of SNMPv3 users as a complete set. Use the following command to create an SNMPv3 user.

Note:

Be sure to include the details of all SNMPv3 users while creating a new user as the set will be replaced with this command. 
CLI> alter <cell|dbserver> snmpUser=((name=<username>, authProtocol=SHA, authPassword=<password>, privProtocol=DES, privPassword=<password>)[,<repeat_with_details_as_necessary_for_additional_users>])

Create SNMPv3 Users on Cisco Ethernet Switches

The below commands configure an SNMP user with authentication and privacy parameters on the Cisco Ethernet switches running NX-OS including the admin switch, and if this is a Exadata X8M, the RoCE switches.

Starting with Enterprise Manager 13.5 Release Update 13, while configuring SNMP, Enterprise Manager does not support AUTH protocol DES and PRIV protocol MD5 in Cisco Ethernet switches running NX-OS.

The passphrase can be any case-sensitive, alphanumeric string up to 64 characters.

switch# configure terminal
switch(config)#
switch(config)# snmp-server user <username> auth sha <passphrase> priv aes-128 passphrase

The below command displays information about one or more SNMP users.

switch(config)# show snmp user

Once all the configuration changes are done, the below command will save the configuration in persistent memory .

switch(config)# copy running-config startup-config
Create SNMP Community on Cisco Ethernet Switches

The below commands configure an SNMP community on the Cisco Ethernet switches running NX-OS including the admin switch, and if this is a Exadata X8M, the RoCE switches. 

switch# configure terminal
switch(config)#
switch(config)# snmp-server community <community> ro

The below command displays information about one or more SNMP communities.

switch(config)# show snmp community

Ensure that the ACL is not assigned to the community string.

Once all the configuration changes are done, the below command will save the configuration in persistent memory .

switch(config)# copy running-config startup-config

Create an SNMPv3 User in InfiniBand Switches

If the Exadata to be discovered is an IB Exadata and monitoring is to be done using SNMP v3, follow the below commands to create an SNMPv3 user on the IB switches:

  1. Log in to the ILOM CLI of the switch as ilom-admin user.
  2. Create the SNMPv3 user account by using the following create command:
    > create /SP/services/snmp/users/<v3user> privacyprotocol=<privacvy_protocol> privacypassword=<privacy_password> authenticationprotocol=<authentication_protocol> authenticationpassword=<authentication_password>

    For example,

    > create /SP/services/snmp/users/v3adminuser privacyprotocol=AES privacypassword=mypassword1 authenticationprotocol=SHA authenticationpassword=mypassword2

The user is created as part of the discovery flow of the switch. This is required only if you want to explicitly add an SNMPv3 user.

Note:

To edit the SNMPv3 user account, run the following set command:
> set /SP/services/snmp/users/<v3user> privacyprotocol=<privacy_protocol> privacypassword=<privacy_password> authenticationprotocol=<authentication_protocol> authenticationpassword=<authentication_password>

Run the following command to display information about one or more SNMP users:

> show -d properties /SP/services/snmp/users/<username>
Configure SNMPv1 on InfiniBand Switch Targets (Optional)

It is recommended to monitor InfiniBand Switch using SNMPv3 for better security. For information, see Create an SNMPv3 User in InfiniBand Switches.

If SNMP v1 communication is required, perform the following steps to configure a community string on each InfiniBand Switch:

  1. Log in to the InfiniBand Switch ILOM web interface using the URL https://<ib_switch_hostname> as root.

    Note:

    Try using Internet Explorer if the console does not display all fields/values in your browser of choice.

  2. Click ILOM Administration, then Management Access, and finally SNMP.
  3. Ensure the following values are set:
    State=Enabled
Port=161
Protocols=v1,v2c,v3

    If you need to make changes, make sure you click Save.

  4. Click ILOM Administration, then Management Access, then SNMP, and finally the Communities link. The page scrolls to the SNMP Communities table. In the SNMP Communities table, click Add.

    A new window opens.

  5. Type the community name into the Community Name field. Select the permissions from the Permission drop-down menu. Click Save.

    The community is created, and the window closes.

  6. Click Alert Management.
  7. If not already listed, for each Agent that monitors the InfiniBand Switch target, select an empty alert (one that has the Destination Summary 0.0.0.0, snmp v1, community 'public') and click Edit. Provide the following values:
    Level = Minor
Type = SNMP Trap
Address = [agent compute node hostname]
Destination Port = [agent port]
SNMP Version = v1
Community Name = public

    Click Save.

Enable SNMPv3 on PDUs

Follow the below steps to enable SNMPv3 on the PDUs.

  1. Access the PDU metering unit from a system on the network.
  2. Click on the Net Configuration link and log in as an admin user.
  3. Select the SNMP-Access tab.
  4. Click the SNMP v3 Enable checkbox to enable SNMP v3.
  5. Click Submit.

Create an SNMPv3 User on PDUs

Follow the below steps to create an SNMPv3 user on the PDUs.

  1. Access the PDU metering unit from a system on the network.
  2. Click on the Net Configuration link and log in as an admin user.
  3. Select the SNMP-Access tab.
  4. In the SNMPv3 table, perform the following
    1. Enter the SNMPv3 UserName.
    2. Select the Security Level auth / priv.
    3. Select SHA as the Auth Algorithm.
    4. Enter the Auth Password.
    5. Select AES as the Privacy Algorithm.
    6. Enter the Privacy Password.
    7. Select Enable.
  5. Click Submit.
Create SNMP Community on PDUs

Follow the below steps to create an SNMP community on the PDUs.

  1. Access the PDU metering unit from a system on the network.
  2. Click on the Net Configuration link and log in as an admin user.
  3. Select the SNMP-Access tab.
  4. In the NMS (SNMPv1/v2) table, perform the following
    1. Enter the SNMP IP address or host name of the first monitoring agent.
    2. Select the privacy level for read-write community as private or public.
    3. Select the privacy level for read-only community as private or public.
    4. Select the type of access from the menu.
    5. Select Enable.
  5. Click Submit.

Verify Software Versions

Verify the software versions for Exadata Storage Server Software, InfiniBand Switch, Cisco Ethernet Switch, PDU Firmware, and Grid Infrastructure/DB Cluster.

Exadata Storage Server Software

See Supported Hardware and Software Configurations for the Oracle Exadata Plug-in for specific supported Exadata Software releases. To verify the software version on the Exadata Storage Server, ssh to the Exadata Storage Server as the root, celladmin, or cellmonitor user. Run: 

# cellcli -e 'list cell detail'

Look for releaseVersion in the output.

InfiniBand Switch

To verify the version of the InfiniBand switch firmware in your environment:

  1. Log on to the management interface for the InfiniBand Switch (using SSH).
  2. Run the following command:
    # nm2version

    The output should be similar to this:

    # nm2version
Sun DCS 36p version: 2.2.13-2

    This example shows a supported configuration for deploying the plug-in to monitor.

  3. If the nm2version command returns output similar to this:
    # nm2version
NM2-36p version: 1.0.1-1

    Then you must upgrade your InfiniBand switch firmware. Follow the instructions listed in Exadata Database Machine and Exadata Storage Server Supported Versions (KB153930) in My Oracle Support.

Cisco Ethernet Switch

Use the show version command to list the BIOS version, NXOS version, and the model details.

Log on to the management interface for the Cisco Switch (using SSH). Run the following command:

#show version

Here's an example output to the command:

Software
  BIOS: version 05.38
  NXOS: version 7.0(3)I7(8)
  BIOS compile time: 06/12/2019
  NXOS image file is: bootflash:///nxos.7.0.3.I7.8.bin
  NXOS compile time: 3/3/2020 20:00:00 [03/04/2020 04:49:49]

Hardware
  cisco Nexus9000 C9336C-FX2 Chassis
  Intel(R) Xeon(R) CPU D-1526 @ 1.80GHz with 24571608 kB of memory.
  Processor Board ID FDO230701AK

PDU Firmware

The PDU firmware version must be 2.10 or later. The current version can be obtained by logging into the web interface of the PDU. On the left side of the screen, click Module Info to view the PDU firmware version.

Software updates for the PDU are available at:

https://updates.oracle.com/Orion/PatchDetails/process_form?patch_num=12871297

Grid Infrastructure/DB Cluster

Grid Infrastructure/DB Cluster is required to be up and running before discovery.

Verify Names Resolution

The Enterprise Manager OMS server(s) require direct network access to each of the compute nodes. If the names of the compute nodes are not registered in the OMS nodes' DNS, then they will have to be manually entered in the /etc/hosts file for each OMS.

Each compute node should be verified to be able to resolve the host names of the ILOM servers, PDUs, Exadata Storage Servers, and InfiniBand and Cisco switches. Again, if the names of those components are not registered in DNS, then entries can be added to the /etc/hosts file of each compute node.

To manage the Exadata Database Machine components from Enterprise Manager, it is necessary for your local machine to be able to resolve the host name of Enterprise Manager.

To access any of the Exadata Database Machine components directly from your local machine, it is also necessary for your local machine to be able to resolve the names of those components.

Verify Firewall Configuration

To verify the firewall configuration:

  1. Enable ping

    In many secure network environments, it is normal for the ping service to be disabled. Enterprise Manager uses ping to establish the basic availability and status of the Exadata Database Machine components.

    • The compute nodes need to have the ping service and port enabled from the OMS Server(s).

    • All other Exadata Database Machine components (ILOM servers, PDU's, Exadata Storage Servers, InfiniBand switches, and Cisco switch) need to have the ping service and port enabled from the compute nodes (where the agents are running).

    Note:

    The ping traffic overhead is minimal. The agent pings the targets every five minutes.

  2. Open Database Ports

    The database listener ports must be opened for the Enterprise Manager OMS server(s). Note that Exadata Database Machine databases will use SCAN listeners; so, ports will need to be opened for the base compute node, the compute node virtual IP, and scan listeners addresses.

    For example, if an Exadata Database Machine quarter rack has been configured with two compute nodes - exadbnode1.example.com and exadbnode2.example.com - and the listeners are using port 1521, then port 1521 will have to be opened to the Enterprise Manager Server for the following addresses:

    • The compute node host names - exadbnode1.example.com and exadbnode2.example.com

    • The virtual IPs for each compute node - exadbnode1-vip.example.com and exadbnode1-vip.example.com

    • The scan listener hostname - scan-exadatadb

  3. Open Enterprise Manager Upload Port

    The Enterprise Manager agents require access to the Enterprise Manager Servers upload service, normally configured on port 4889 for HTTP uploads and 4900 for HTTPS. To verify the ports assigned, run the following command on the OMS server command line.

    $ emctl status oms -details

    These ports will need to be opened for each of the compute nodes.

  4. Open Agent Ports

    The OMS server(s) will need to be able to connect to the Enterprise Manager agent HTTP/HTTPS port on each compute node. The agent port defaults to 3872. If port 3872 is not available, the next available port starting from port 1830 is used.

    To identify the port used:

    • Run the following command on the compute node command line:

      $ emctl status agent

    • Alternatively, you can look for the value of the EMD_URL property in the emd.properties file the following directory: <AGENT_HOME>/agent_inst/sysman/config

  5. Open SSH Ports (port 22)

    The Enterprise Manager agents require ssh access to the Exadata Database Machine components they monitor. As the agents will run on the compute nodes the ssh ports, 22, on each of the Exadata Storage Servers, ILOMs, PDUs, InfiniBand switches, and Cisco switch will need to be opened for each of the compute nodes.

  6. Allow UDP Traffic (SNMP Ports) (Port 162)

    All Exadata Database Machine components need to be able to send SNMP traps to the agents running on the compute nodes. SNMP uses the UDP protocol so the agent port and port 162 need to be opened for UDP traffic between the Exadata Storage Servers, ILOMs, InfiniBand Switches, Cisco Switch, and the agent.

Table 2-1 Firewall Ports

Component Ping service and port SNMP* SSH (port 22) Notes

PDU

From remote agent

Yes

Yes

Compute node ILOM

From remote agent

Yes

Yes

  1. Remote agent needs to be able to SSH to dom0.

  2. Need SNMP port open on dom0.

dom0

From remote agent

Yes

Yes

Exadata Storage Server

From remote agent

Yes

Yes

InfiniBand Switch

From remote agent

Yes

Yes

Cisco Switch

From remote agent

Yes

Yes

OMS

Yes

Upload http/https port - usually 3872

Agent

The OMS server(s) will need to be able to connect to the Enterprise Manager agent HTTP/HTTPS port on each compute node. The agent port defaults to 3872. If port 3872 is not available, the next available port starting from port 1830 is used.

User Roles

To manage the Exadata Database Machine in Enterprise Manager, you should consider creating Enterprise Manager roles and assigning them to the appropriate Enterprise Manager users. Creating these roles restricts the privileges that each user has, for example in deleting the plug-in or accessing reports. See Oracle Exadata Database Machine Administration.