2 Installation and Deployment

This chapter provides the instructions for deployment of the Oracle Exadata plug-in. The following topics are discussed:

Oracle Exadata Plug-in Deployment Prerequisites

Before deploying the Oracle Exadata plug-in, make sure the following prerequisites are met:

Create a Database Server ILOM Service Processor User

For the Enterprise Manager agent to communicate with an ILOM service processor, there must be a specific user ID established on the ILOM service processor.

Note:

Adding the specific user ID requires administrator level privilege on the ILOM service processor.

The specific ILOM user ID can be added in the ILOM service processor web interface, ILOM CLI, or with the ipmitool command. This example uses ILOM CLI.

For security reasons, the password to the ILOM service processor root user ID does not appear in the ILOM CLI commands in this example.

  1. Log in to the Service Processor as root:

    # ssh root@[Service Processor IP]
Password:

  2. Change to the users directory:

    # cd /SP/users

  3. Create the oemuser user and password:

    # create oemuser

Creating user...
Enter new password: ********
Enter new password again: ********

Created /SP/users/oemuser

  4. Change to the new user's directory and set the role:

    # cd oemuser
/SP/users/oemuser

set role='cro'
Set 'role' to 'cro'

  5. Test the ILOM user ID created in step 3 by listing the last 10 system events:

    • For Exadata X2 through X4:

      # ipmitool -I lan -H <ilom_hostname> -U oemuser -P oempasswd -L USER sel list last 10

    • For Exadata X5-2 (requires the -I lanplus command option):

      # ipmitool -I lanplus -H <ilom_hostname> -U oemuser -P oempasswd -L USER sel list last 10

  6. Repeat steps 1 through 5 for the rest of the compute node ILOM service processors in your Oracle Database Machine.

Verify Software Versions

Verify the following software versions:

Exadata Storage Server Software

The supported version is Exadata Storage Server Software 11g Release 2 (see Exadata Software Support for specific release version details). To verify the cell software version on the Exadata cell, ssh to the Exadata cell as the root, celladmin, or cellmonitor user. Run:

# cellcli -e 'list cell detail'

Look for "releaseVersion" in the output.

ILOM ipmitool

Enterprise Manager requires a minimal ipmitool software version of 1.8.10.4 or later for Oracle Solaris and 1.8.10.3 for Oracle Linux. To view the software version:

  • For Oracle Linux, run the following command as the root user on one of the database servers in the cluster:

    # dcli -g ~/dbs_group -l root ipmitool –V

    Note:

    The dbs_group file contains the list of compute node hostnames, one on each line. If the file does not exist, you can create it before running the dcli command.

  • For Oracle Solaris, run the following command on each of the database servers in the cluster:

    # /opt/ipmitool/bin/ipmitool -V

InfiniBand Switch

To verify the version of the InfiniBand switch firmware in your environment:

  1. Log on to the management interface for the InfiniBand Switch (using ssh).

  2. Run the following command:

    # nm2version

    The output should be similar to this:

    # nm2version
Sun DCS 36p version: 1.1.3-2

    This example shows a supported configuration for deploying the plug-in to monitor.

  3. If the nm2version command returns output similar to this:

    # nm2version
NM2-36p version: 1.0.1-1

    Then you must upgrade your InfiniBand switch firmware. Follow the instructions listed in My Oracle Support (MOS) Document 888828.1:

    https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=REFERENCE&id=88882

PDU Firmware

The PDU firmware version must be 1.04 or later. The current version can be obtained by logging into the web interface of the PDU. On the left side of the screen, click Module Info to view the PDU firmware version.

Software updates for the PDU are available at:

https://updates.oracle.com/Orion/PatchDetails/process_form?patch_num=12871297

KVM Application

The KVM Application software version must be 1.2.8 or later. The current version can be obtained by logging into the web interface of the KVM. On the left side of the screen under Unit View, Appliance, Appliance Settings, click Versions to view the Application software version.

Software updates for KVM are available at:

http://www.avocent.com/Support_Firmware/MergePoint_Unity/MergePoint_Unity_Switch.aspx

Grid Infrastructure/DB Cluster

Grid Infrastructure/DB Cluster is required to be up and running before discovery.

Verify Names Resolution

The Enterprise Manager OMS server(s) require direct network access to each of the compute nodes. If the names of the compute nodes are not registered in the OMS nodes' DNS, then they will have to be manually entered in the /etc/hosts file for each OMS.

Each compute node should be verified to be able to resolve the hostnames of the ILOM servers, PDU's, storage cell nodes, and InfiniBand and Cisco switches. Again, if the names of those components are not registered in DNS, then entries can be added to the /etc/hosts file of each compute node.

To manage the Exadata Database Machine components from Enterprise Manager Cloud Control 13c, it is necessary for your local machine to be able to resolve the host name of Cloud Control 13c.

To access any of the Exadata Database Machine components directly from your local machine, it is also necessary for your local machine to be able to resolve the names of those components.

Verify Firewall Configuration

To verify the firewall configuration:

  1. Allow ping

    In many secure network environments, it is normal for the ping service to be disabled. Enterprise Manager uses ping to establish the basic availability and status of the Exadata Database Machine components.

    • The compute nodes need to have the ping service and port enabled from the OMS Server(s).

    • All other Exadata Database Machine components (ILOM servers, PDU's, storage cell nodes, InfiniBand switches, and Cisco switch) need to have the ping service and port enabled from the compute nodes (where the agents are running).

    Note:

    The ping traffic overhead is minimal. The agent pings the targets every five minutes.

  2. Open Database Ports

    The database listener ports must be opened for the Enterprise Manager OMS server(s). Note that Exadata Database Machine databases will use SCAN listeners; so, ports will need to be opened for the base compute node, the compute node virtual IP, and scan listeners addresses.

    For example, if an Exadata Database Machine quarter rack has been configured with two compute nodes - exadbnode1.example.com and exadbnode2.example.com - and the listeners are using port 1521, then port 1521 will have to be opened to the Enterprise Manager Server for the following addresses:

    • The compute node hostnames - exadbnode1.example.com and exadbnode2.example.com

    • The virtual IPs for each compute node - exadbnode1-vip.example.com and exadbnode1-vip.example.com

    • The scan listener hostname - scan-exadatadb

  3. Open Enterprise Manager Upload Port

    The Enterprise Manager Cloud Control 13c agents require access to the Enterprise Manager Servers upload service, normally configured on port 4889 for HTTP uploads and 4900 for HTTPS. To verify the ports assigned, run the following command on the OMS server command line.

    $ emctl status oms -details

    These ports will need to be opened for each of the compute nodes.

  4. Open Agent Ports

    The OMS server(s) will need to be able to connect to the Enterprise Manager Cloud Control 13c Agent HTTP/HTTPS port on each compute node. The Agent port defaults to 3872. If port 3872 is not available, the next available port starting from port 1830 is used.

    To identify the port used:

    • Run the following command on the compute node command line:

      $ emctl status agent

    • Alternatively, you can look for the value of the EMD_URL property in the emd.properties file the following directory:

      <AGENT_HOME>/agent_inst/sysman/config

  5. Open SSH Ports (port 22)

    The Enterprise Manager Cloud Control 13c Agents require ssh access to the Exadata Database Machine components they monitor. As the Agents will run on the compute nodes the ssh ports, 22, on each of the storage cells, ILOMs, PDUs, KVMs, InfiniBand switches, and Cisco switch will need to be opened for each of the compute nodes.

    Note:

    The emkit configures ssh access but still requires the ports to be opened first.

  6. Allow UDP Traffic (SNMP Ports) (Port 162)

    All Exadata Database Machine components need to be able to send SNMP traps to the Agents running on the compute nodes. SNMP uses the UDP protocol so the Agent port and port 162 need to be opened for UDP traffic between the Storage Cells, ILOMs, InfiniBand Switches, Cisco Switch, and the Agent.

Table 2-1 Firewall Ports

Component Ping service and port SNMP* SSH (port 22) Notes

PDU

From remote agent

Yes

Yes

  

Compute node ILOM

From remote agent

Yes

Yes

  1. Remote agent needs to be able to SSH to dom0.

  2. Needs to access ipmitool.

  3. Need SNMP port open on dom0.

dom0

From EM OMS server

Yes

Yes

  

Cell

From remote agent

Yes

Yes

  

InfiniBand Switch

From remote agent

Yes

Yes

  

Cisco Switch

From remote agent

Yes

Yes

  

KVM

From remote agent

  

Yes

May not be required, if X5 does not have KVM

OMS

    

Yes

Upload http/https port - usually 3872

Agent

      

The OMS server(s) will need to be able to connect to the Enterprise Manager Cloud Control 12c Agent HTTP/HTTPS port on each compute node. The Agent port defaults to 3872. If port 3872 is not available, the next available port starting from port 1830 is used.

* SNMP uses the UDP protocol; so, the Agent port and port 162 need to be opened for UDP traffic

Oracle SuperCluster Prerequisites

Table 2-2 shows the required versions of OneCommand and Config-O-Matic (COM) supported by the Oracle SuperCluster hardware configurations.

Table 2-2 Oracle SuperCluster Prerequisites

Hardware Configuration OneCommand Version Config-O-Matic Version

M6-32

databasemachine.xml only

14.063 and later

1.6.4 and later

T5-8

databasemachine.xml

14.042 and later

1.5.8 COM and later

T5-8

catalog.xml (non-Java OneCommand)

n/a

1.5.4 COM and earlier

T4-4

catalog.xml only

Note: Because the T4-4 Oracle SuperCluster is at its "end of life," this entry is only for existing systems.

n/a

1.1.6 COM and earlier

User Roles

To manage the Exadata Storage Server, you need to create roles and then assign roles to administrators. Creating these roles restricts the privileges that each user has, for example in deleting the plug-in or accessing reports. See Chapter 5, "Oracle Exadata Database Machine Administration."

Install Enterprise Manager Agent

Note:

Enterprise Manager Exadata discovery supports the use of either management network hostname or client network hostname for the compute nodes. When installing the Enterprise Manager agent on the compute nodes, you should use the same hostname as used in Oracle Clusterware.

You can identify the hostname of the nodes in the cluster by running the olsnodes command on one of the compute nodes. It is recommended that a fully qualified hostname, including the domain name, be used when specifying an Enterprise Manager agent hostname.

There are two methods for installing the Enterprise Manager Agent. Chose the one that works best for your environment:

Notes:

  • Make sure any prerequisites are met. See Agent Installation Prerequisite - Solaris 11 Only.

  • The Enterprise Manager agent must be deployed to all compute nodes of the Database Machine. The host target name of the compute node must be the fully qualified host name, for example, dbm1db01.mydomain.com.

  • Non-fully qualified hostname (for example, dbm1db01) or IP address must not be used for the host target name of the compute node.

  • The same version of the Enterprise Manager agent and the same version of the Exadata plug-ins should be deployed on all compute nodes within the same Database Machine.

Agent Installation Prerequisite - Solaris 11 Only

If you are running Exadata Storage Server Software Release 11g Release 2 (11.2.3.1) on Oracle Solaris, follow the steps below before installing the agent.

  1. Log in to the compute node host as agent user and execute the ppriv $$ command. This command displays the E (Effective), P (Permitted), I (Inherited), and L (Limited) privileges.

    Generally, E, P, and I have the same set of basic privileges and for any normal user.

  2. Assuming that output of the above command shows basic privileges, add the priv_sys_net_config privilege to the agent user so that InfiniBand commands can be executed.

    Log in as root and run:

    # usermod -K defaultpriv=basic,priv_sys_net_config <agent_user>

    This command adds sys_net_config to the agent user.

  3. Log in as the agent user again and run the ppriv $$ command to verify if the privileges are set. The output will show the sys_net_config privilege along with the existing (basic) set of privileges.

Install Agents and Set Up Exadata Monitoring Using Automated Kit

The Oracle Enterprise Manager Cloud Control 13c Setup Automation kit is available for download from My Oracle Support as a patch. The kit simplifies the process of deploying the agents on each of the compute nodes considerably, allowing agents to be deployed to all of the compute nodes, in one go, from one of the compute nodes, using one simple command. Instructions for using the kit to deploy agents on the compute nodes are provided in the patch README.txt.

To download the Automation kit for your platform, see Doc ID 1440951.1 in My Oracle Support:

https://support.oracle.com

Oracle Enterprise Manager Cloud Control 13c uses a holistic approach to manage the Exadata Database Machine and provides comprehensive lifecycle management from monitoring to management and ongoing maintenance for the entire engineered system.

The patch README provides instructions for how to install Management Agents on an Oracle Exadata Database Machine and point them to an existing Cloud Control environment or install a new stand alone Enterprise Manager environment and deploy the Management Agents pointing to this new environment. This standalone environment can be used when you do not want to take the corporate environment offline to apply the patches needed to monitor Oracle Exadata Database Machine.

The following procedures described in the README can be performed at any time, such as during initial configuration of Oracle Exadata Database Machine, or later:

  • Preparing a corporate Oracle Enterprise Manager Cloud Control server to monitor Oracle Exadata Database Machine

  • Installing Oracle Enterprise Manager Cloud Control 13c on a standalone server to monitor Oracle Exadata Database Machine

  • Installing Oracle Enterprise Manager Cloud Control 13c Management Agents on Oracle Exadata Database Machine

  • Removing Oracle Enterprise Manager Cloud Control 13c Management Agents from Oracle Exadata Database Machine when using a standalone server

  • Removing Oracle Enterprise Manager Cloud Control 13c from a standalone server

Install Enterprise Manager Agent Using the Push Method

The installation instructions were captured on a Quarter Rack configuration (for example, 2 compute nodes, 3 cells).

  1. Add the Exadata Database Machine compute nodes as host targets to Oracle Enterprise Manager Cloud Control 13c. From the Enterprise Manager home page, click the Setup menu (upper right corner), Add Target, then Add Targets Manually.

  2. On the Add Host Targets: Host and Platform screen, specify a session name. Then identify the fully qualified hostnames and select the platform.

    Note:

    If the Agent software is not available for your platform, go to the Extensibility page and download it first.

    Figure 2-1 shows how the selection should look for the Linux x86-64 platform.

    Figure 2-1 Add Host Targets

    Add Host Targets Sample

  3. Click Next to add the details for the host.

  4. On the Installation Details screen, provide the following information:

    • Installation Base Directory

    • Instance Directory

    • Named Credential

    • For Port, leave this field blank. As part of the installation process, an available port will be selected automatically.

  5. Click Next to review the details about the host.

  6. Click Deploy Agent to start the agent deployment process.

  7. As the deployment process continues, remote prerequisite checks are automatically checked. If there are no issues, you will be presented with an Agent Deployment Summary with an indication that the agent deployment has passed. Figure 2-2 shows an example of a successful agent deployment.

    Figure 2-2 Agent Deployment Summary

    Agent Deployment Summary Example

Important:

If the root.sh was not executed during deployment, then make sure to execute it on all compute nodes.

Installing Oracle Management Agent in Silent Mode Option

You can install Oracle Management Agent in silent mode as an alternative to installing it using the Add Host Target Wizard. Silent mode requires you to use a response file for providing the installation details and a deployment script for silently installing the Management Agent using the information supplied in the response file.

See the Installing Oracle Management Agent in Silent Mode chapter in the Oracle Enterprise Manager Cloud Control Advanced Installation and Configuration Guide for more information:

http://docs.oracle.com/cd/E24628_01/install.121/e24089/install_agent_usng_rsp.htm#CEGGACJE

Manually Deploy Exadata Plug-in

Under the following conditions, you may need to manually deploy the Exadata plug-in on each of the compute nodes:

  • For a new or fresh installation: Deploy the plug-in manually if you did not deploy the agent using Automation kit or push the latest version of the agent through OMS.

  • For an upgrade to an existing installation: Deploy the Exadata plug-in manually if an older version of the Exadata plug-in has been deployed to the agent already and you would like to upgrade to the latest version of the Exadata plug-in.

To determine if the Exadata plug-in is deployed on each compute node and what version it is, you have two options:

  • From a terminal window, run the following command:

    emctl listplugins agent

    Note:

    The emctl listplugins agent command must be run on the compute node using the emctl in the agent installation directory.

  • From Enterprise Manager Cloud Control, click the Setup menu (upper right corner), Extensibility, and then Plug-ins.

To manually deploy the Exadata plug-in:

  1. From the Enterprise Manager home page, click the Setup menu (upper right corner), Extensibility, and then Plug-ins.

  2. On the Plug-ins page, select Oracle Exadata from the Name list.

    Note:

    The selected version should be for Release 12.1.0.1.0, 12.1.0.2.0, or 12.1.0.3.0. Oracle recommends that you deploy the latest version of the Exadata and Database plug-ins to the agent.

  3. With Oracle Exadata selected, click Deploy On, then Management Agent.

  4. On the Deploy Plug-in on Management Agent pop-up, click Add. A search pop-up will appear to allow you to search for targets to add. In the Target Type drop-down, select Agent, then click Search.

    Select a Target name from the results list and click Select. Repeat for each agent target.

  5. After you have added the agents, click Next on the Deploy Plug-in on Management Agent screen to review and verify the agent information.

  6. Click Deploy to deploy the plug-in to the agents.

  7. Once the agent has deployed to all the agents, a confirmation screen will appear. Click OK to dismiss the pop-up or Show Status to display the status of the agent in the Enterprise Manager Deployment Activities screen.