Configure Storage Cell SNMP for Enterprise Manager Monitoring
Configure the Compute Node ILOM SNMP for Enterprise Manager Monitoring
Accessing Oracle Support Workbench for Exadata Storage Server
Note:
You must remove the SNMP notification on the cell, InfiniBand switch, ILOM, Cisco switch, PDU, and KVM manually if you remove these targets from Enterprise Manager.
Starting with Exadata plug-in Release 12.1.0.3.0, when the Exadata Database Machine target is deleted through Enterprise Manager Cloud Control, there is an option to remove the SNMP notification from the cells and InfiniBand switches.
Note:
View a video of how to monitor your Exadata environment:
https://apex.oracle.com/pls/apex/f?p=44785:24:0::NO:24:P24_CONTENT_ID,P24_PREV_PAGE:5664,1
Note:
If you do not need to monitor KVM, then you can skip this section.
The traps from the KVM will be sent to UDP port 162 on the monitoring agent host. For the Enterprise Manager Agent to receive traps from the KVM, SNMP trap forwarding must be set up on the host where the Enterprise Manager Agent is running so that the snmptrapd
daemon can receive traps using port 162 and forward the trap to the agent receivelet's listening port.
Ideally, you should perform the steps on all compute nodes in preparation for a scenario when monitoring is moved to a compute node that was not previously being used to monitor the KVM targets.
Exadata Storage Cell SNMP configuration is performed using the cellcli
command and can be run in batch using dcli
from a compute node.
Note:
For Enterprise Manager Cloud Control 12c Bundle Pack 1 (BP1):
The SNMP trap subscription for Exadata Storage Cells is done automatically after finishing the guided discovery process.
For Exadata plug-in Release 12.1.0.2.0 and later:
During the discovery process, you can optionally provide the necessary root
credentials to subscribe for SNMP traps from Exadata Storage Cells. If you have done so, then you can skip the remaining steps of this section and proceed with Configure and Verify SNMP for InfiniBand Switch Targets.
While the public
string is used for the SNMP examples in the following sections, it is supported to use any valid SNMP community string.
Because of a limitation with the V1 protocol and JDMK API, if an IPv6 component sends SNMPv1 traps, then the sender's IP address is always 0.0.0.0
. Therefore, if the Exadata Storage Server environment is IPv6, then the snmpv1
subscription should not be allowed.
Use an SNMP V3 subscription for IPv6-based cell targets because of the limitation in SNMP V1 protocol.
ALTER CELL
CommandWhile using the ALTER CELL
command, all existing subscribers should be specified along with the new subscriber being added. Similarly, you can also modify the notificationPolicy
or notificationMethod
attributes.
While using the ALTER CELL
command, the host=
and community=
attribute values should be quoted, and type=
is NOT quoted.
If you are using the DCLI utility to set up SNMP alerting, then any command containing punctuation, which will be interpreted by the local shell, must be enclosed with double quotation marks. If the command includes the following characters, then outer quotation marks and escape characters are required:
The backslash (\) is the escape character that allows the characters to be passed to the CellCLI utility without being interpreted by the remote shell.
Check the current SNMP configuration using the following cellcli
commands:
Open a new terminal and verify whether the SSH connectivity was successfully established:
$ ssh -l cellmonitor <cell_ipaddress> cellcli -e 'list cell detail'
If you are not prompted for any password, then you can assume that the connectivity is established.
If you are asked to confirm whether you want to continue connecting, specify Yes.
To remove the subscription, use the ALTER CELL
command again by excluding the host name that you want to remove from the snmpsubscriber
list.
Note:
The SNMP receivelet listens on a single address and port for all monitored targets. The port is the UDP port with the same number as the TCP port used in the EMD_URL
.
By default, the SNMP receivelet listens on all addresses; if the property SnmpRecvletListenNIC
is set in the emd.properties
file, the receivelet will attempt to resolve the value as either a name or IP address, and listen on only that address.
This parameter is independent of AgentListenOnAllNICs
and EMD_URL
because in some installations, the Agent may need to communicate with the OMS and with managed targets on different networks.
The SNMP configuration for Enterprise Manager monitoring of InfiniBand Switches is done automatically as part of the Enterprise Manager guided discovery process. It is good practice, however, to verify that SNMP configuration has been successful.
Note:
For Enterprise Manager Cloud Control 12c Bundle Pack 1 (BP1):
The SNMP Trap setup for Infiniband Switches is done automatically after finishing the guided discovery process.
For Exadata plug-in Release 12.1.0.2.0 and later:
During the discovery process, you can optionally provide the necessary root
credentials to set up SNMP trap for the InfiniBand Switch. If you have done so, then you can skip the remaining steps of this section and proceed with Configure the Compute Node ILOM SNMP for Enterprise Manager Monitoring.
To configure (if necessary) and verify the SNMP configuration for an InfiniBand Switch:
You can set up SNMP for InfiniBand Switch targets using the Enterprise Manager Cloud Control console:
Perform steps 1-5 for both the Monitoring Agent and Backup Monitoring Agent of the InfiniBand switch target.
Note:
These instructions apply only for the Exadata Database Machine and are not applicable to the Oracle SuperCluster.
The compute node ILOM targets are responsible for displaying a number of disk failure alerts for their respective compute node that are received as SNMP traps. For Enterprise Manager to receive those traps, the /opt/oracle.cellos/compmon/exadata_mon_hw_asr.pl
script must be run to configure SNMP subscriptions for the agents (both primary and backup agents) that have been configured to monitor the compute node ILOM targets. This step is applicable to Exadata plug-in Release 12.1.0.2.0 and later.
The exadata_mon_hw_asr.pl
script is run as the root user with the -set_snmp_subscribers
parameter to add SNMP subscribers. For example:
# /opt/oracle.cellos/compmon/exadata_mon_hw_asr.pl -set_snmp_subscribers "(host=hostname1.mycompany.com,port=3872,community=public,type=asr,fromip=11.222.33.444),(host=hostname2.mycompany.com,port=3872,community=public,type=asr,fromip=12.345.67.890)" Try to add ASR destination Host - hostname1.mycompany.com IP - 11.222.33.44 Port - 3872 Community - public From IP - 22.333.44.555 Try to add ASR destination Host - hostname2.com IP - 11.111.11.111 Port - 3872 Community - public From IP - 22.333.44.555
The script needs to be run on each compute node:
The host
values should be the hostnames of the agents configured to monitor the compute node ILOM target associated with the compute node.
The fromip
values should be the IP address of the compute node that the compute node ILOM target is associated with.
For example, if you have an X2-2 machine with compute node targets edbm01db01
through edbm01db08
and associated compute node ILOM targets edbm01db01-c
through edbm01db08-c
, then you would need to run the script once on each compute node - therefore, the script would be run eight times in total.
On compute node edbm01db01
, the host
and port
values would be the hostnames and ports of the agents monitoring compute node ILOM target edbm01db01-c
and the fromip
value would be the IP address of the compute node itself, edbm01db01
.
On compute node edbm01db02
, the host
and port
values would be the hostnames and ports of the agents monitoring compute node ILOM target edbm01db02-c
and the 'fromip' value would be the IP address of the compute node itself, edbm01db02, ... and so on.
This is a good example of where Manual selection of Management Agents for targets is useful. If the first two compute nodes are always the Monitoring Agent and Backup Monitoring Agent, then it is easy to work out the values needed for -set_snmp_subscribers
parameters, the host
and port
values would be the same for all compute nodes.
Note:
The exadata_mon_hw_asr.pl
script, overwrites any existing SNMP subscriptions. While setting the SNMP subscribers, make sure that current subscribers are included in the new list of subscribers.
It is possible to use the exadata_mon_hw_asr.pl
script to get the current set of subscribers using the -get_snmp_subscribers
parameter.
For example:
# /opt/oracle.cellos/compmon/exadata_mon_hw_asr.pl -get_snmp_subscribers -type=asr
Suppose the current list is:
(host=hostname1.mycompany.com,port=162,community=public,type=asr,fromip=11.222.33.444), (host=hostname2.mycompany.com,port=162,community=public,type=asr,fromip=11.222.33.444)
Then new subscriptions can be added using the following command:
# /opt/oracle.cellos/compmon/exadata_mon_hw_asr.pl -set_snmp_subscribers "(host=asrhostname1.mycompany.com,port=162,community=public,type=asr,fromip=11.222.33.444), (host=asrhostname2.mycompany.com,port=162,community=public,type=asr,fromip=11.222.33.444), (host=hostname1.mycompany.com,port=3872,community=public,type=asr,fromip=11.222.33.444), (host=hostname2.mycompany.com,port=3872,community=public,type=asr,fromip=11.222.33.444)"
After adding the new subscribers, run the command exadata_mon_hw_asr.pl
script with the -get_snmp_subscribers
parameter to get the list of SNMP subscribers and verify the new SNMP subscriptions were added successfully. For example:
# /opt/oracle.cellos/compmon/exadata_mon_hw_asr.pl -get_snmp_subscribers -type=asr (host=asrhostname1.mycompany.com,port=162,community=public,type=asr,fromip=10.10.10.226), (host=asrhostname2.mycompany.com,port=162,community=public,type=asr,fromip=10.10.10.226), (host=hostname1.mycompany.com,port=3872,community=public,type=asr,fromip=10.10.10.226), (host=hostname2.mycompany.com,port=3872,community=public,type=asr,fromip=10.10.10.226)
For compute nodes running Management Server, the call to the exadata_mon_hw_asr.pl
script is replaced with dbmCLI
commands, which are similar to cellCLI
commands on the Storage Cells.
Oracle Exadata Database Machine X5-2 (up to X6) runs the Management Server on compute nodes.
For Virtualized Exadata, these commands need to be executed in Dom0
.
The Cisco Ethernet Switch must be configured to allow the Agents that monitor it to be able to both poll the switch and to receive SNMP alerts from the switch. To allow this, perform the following steps (swapping the example switch name dm01sw-ip
with the name of the Cisco Ethernet Switch target being configured):
Run the snmpwalk
command line utility or equivalent tool to verify the Cisco Switch configuration.
Run the following commands to fetch and display the data from the Cisco switch:
$ snmpget –v 1 –c <community_string> <hostname_of_cisco_switch> 1.3.6.1.4.1.9.2.1.56.0 $ snmpget –v 2c –c <community_string> <hostname_of_cisco_switch> 1.3.6.1.4.1.9.2.1.56.0
Note:
If a timeout message is displayed as an output for the above command, then it means that the Cisco Switch is not yet configured correctly.
To enable Enterprise Manager to collect metric data and raise events for the PDU target, you must configure the PDU to accept SNMP queries from the Agents that monitor the PDU target. Also, appropriate threshold values for different phase values needs to be set on the PDU.
This section assumes that this is a first time configuration of the PDU. SNMP must be enabled and the trap section completed. Granting SNMP access to a different monitoring Agent IP address is an example where only the "Trap Host Setup" section needs to be changed.
For details on configuring the PDU thresholds settings, see the "Configuring Oracle Exadata Database Machine" chapter in your Exadata Database Machine Owner's Guide. This guide is pre-loaded along with other Exadata user documentation onto your Oracle Exadata Database Machine.
Use the snmpwalk
command line utility or equivalent tool to verify the PDU configuration.
Run the following command to fetch and display the data from PDU:
snmpget –v 1 –c <community_string> <hostname_of_pdu> 1.3.6.1.4.1.2769.1.2.3.1.1.1.0
Note:
If a timeout message is displayed as an output for the above command, then it means that the PDU is not yet configured correctly.
The KVM needs to be configured to send SMNP traps to the Agents that monitor it. See the following sections to set up SNMP for KVM targets:
Verify the KVM SNMP Configuration (Base SNMP Configuration) - Linux
Verify the KVM SNMP Configuration (Base SNMP Configuration) - Solaris
Verify the KVM SNMP Configuration (SNMP Forwarding to Agent)
Note:
The SNMP forwarder setup described in Set Up SNMP Trap Forwarding on Compute Node is required on the compute node agent that monitors the KVM target. If the SNMP forwarder is not set up, the agent will not receive traps from the KVM device.
The traps from the KVM device will be sent to UDP port 162 on the monitoring agent host. Refer to Set Up SNMP Trap Forwarding on Compute Node for details to set up the trap forwarder to forward traps received on port 162 to the Enterprise Manager agent receivelet port.
Run the snmptrapd
command line utility to verify the KVM configuration. Make sure that the steps outlined in Set Up SNMP Trap Forwarding on Compute Node has been performed.
Follow the steps below on the monitoring agent host for the KVM target:
Log in as root
.
Run the following command:
# service snmptrapd stop # /usr/sbin/snmptrapd -d -Lsd -p /var/run/snmptrapd.pid
Reboot the KVM to generate SNMP traps. To reboot the KVM:
Log in to the KVM.
Click Overview on the left side of the screen under Unit View, Appliance.
Click Reboot.
In the confirmation window, click OK to confirm KVM reboot.
Check that snmptrapd
logs messages in the /var/log/messages
file from the IP address of the KVM. The following example shows the messages for the KVM's IP address (10.1.1.240):
Jul 30 14:51:05 dbm1db02 snmptrapd[65800]: Received 72 bytes from UDP: [10.1.1.240]:32768 . Jul 30 14:51:05 dbm1db02 snmptrapd[65800]: 0000: 30 82 00 44 02 01 00 04 06 70 75 62 6C 69 63 A4 0..D.....public. . Jul 30 14:51:05 dbm1db02 snmptrapd[65800]: 0016: 82 00 35 06 09 2B 06 01 04 01 D1 32 12 01 40 04 ..5..+.....2..@. . Jul 30 14:51:05 dbm1db02 snmptrapd[65800]: 0032: 0A F5 13 F0 02 01 06 02 01 02 43 03 6D 19 4C 30 ..........C.m.L0 . Jul 30 14:51:05 dbm1db02 snmptrapd[65800]: 0048: 82 00 15 30 13 06 0B 2B 06 01 04 01 D1 32 12 02 ...0...+.....2.. . Jul 30 14:51:05 dbm1db02 snmptrapd[65800]: 0064: 06 01 04 04 72 6F 6F 74 ....root . Jul 30 14:51:05 dbm1db02 snmptrapd[65800]:
If step 4 is confirmed, the KVM has been configured correctly. Run the following commands:
# service snmptrapd stop # service snmptrapd start
Run the snmptrapd
command line utility to verify the KVM configuration. Make sure that the steps outlined in Set Up Compute Node Agent - Solaris have been performed.
Follow the steps below on the monitoring agent host for the KVM target:
Log in as root
and run the following command:
# svcadm disable snmptrapd # /usr/sbin/snmptrapd -d -Lsd -p /var/run/snmptrapd.pid
Reboot the KVM to generate SNMP traps. To reboot the KVM:
Log in to the KVM.
Click Overview on the left side of the screen under Unit View, Appliance.
Click Reboot.
In the confirmation window, click OK to confirm KVM reboot.
Check that the snmptrapd
daemon logs messages in the /var/log/messages
file from the IP address of the KVM. The following example shows the messages for the KVM's IP address (10.1.1.240):
Jul 30 14:51:05 dbm1db02 snmptrapd[65800]: Received 72 bytes from UDP: [10.1.1.240]:32768 Jul 30 14:51:05 dbm1db02 snmptrapd[65800]: 0000: 30 82 00 44 02 01 00 04 06 70 75 62 6C 69 63 A4 0..D.....public. Jul 30 14:51:05 dbm1db02 snmptrapd[65800]: 0016: 82 00 35 06 09 2B 06 01 04 01 D1 32 12 01 40 04 ..5..+.....2..@. Jul 30 14:51:05 dbm1db02 snmptrapd[65800]: 0032: 0A F5 13 F0 02 01 06 02 01 02 43 03 6D 19 4C 30 ..........C.m.L0 Jul 30 14:51:05 dbm1db02 snmptrapd[65800]: 0048: 82 00 15 30 13 06 0B 2B 06 01 04 01 D1 32 12 02 ...0...+.....2.. Jul 30 14:51:05 dbm1db02 snmptrapd[65800]: 0064: 06 01 04 04 72 6F 6F 74 ....root Jul 30 14:51:05 dbm1db02 snmptrapd[65800]:
If step 3 is confirmed, the KVM has been configured correctly. Stop the snmptrapd
process currently running:
# ps -ef | grep snmptrapd # kill -9 <snmptrapd process ID>
Start the snmptrapd
daemon:
# svcadm enable snmptrapd
After performing the steps in Verify the KVM SNMP Configuration (Base SNMP Configuration) - Linux or Verify the KVM SNMP Configuration (Base SNMP Configuration) - Solaris, follow the steps below to validate that the monitoring Agent can receive the SNMP traps generated by the KVM and convert the received traps to Enterprise Manager events.
You can access the Oracle Support Workbench for the current Exadata Storage Server to access diagnostic data for problems and incidents related to the cell.
To access the Support Workbench for a single Exadata cell, follow these steps:
You can create a Database Machine dashboard to monitor the performance and usage metrics of the Database Machine system, its sub-components, as well as all the database system components residing on the Database Machine.
For Exadata Database Machine plug-in Release 12.1.0.3.0 and later, create the Database Machine Dashboard:
The generated report is accessible only by the Enterprise Manager user who creates it. To make the report public: