Oracle® Enterprise Manager

System Monitoring Plug-In Installation Guide for Oracle Engineered System Healthchecks

Release 12.1.0.5.0

E27420-05

August 2015

The Oracle Engineered System Healthchecks plug-in processes the XML output from the Exachk tool, which is included as part of Oracle Enterprise Manager system monitoring. The Exachk tool provides functionality for system administrators to automate the assessment of Engineered Systems for known configuration problems and best practices.

This document covers the following topics:

1 Oracle Engineered System Healthchecks Plug-in Contents

The Oracle Engineered System Healthchecks plug-in bundle consists of the following files:

  • Healthchecks Plug-in XML parser utility: A set of Perl scripts that parse the resulting XML file generated from the Exachk tool and returns the output in em_result format. This format can be processed by the Enterprise Manager Metric Engine.

  • A metadata XML file: defines the new Target Type and Metrics.

  • A Default Collections XML file: defines the schedule (that is, the time interval for the data collection of metrics). It also contains the logic for raising the alerts depending on results from the Exachk tool, and the messages to be shown for alerts.

  • A Report SQL file: defines a new Report, which will be created during the jar deployment.

  • A Messages dlf file: contains the risk, recommendation, benefit, and fail message to be displayed in case of a healthcheck failure.

  • The mpcui files: used for home page customization to define the content of the home page and the menus provided on this page.

1.1 Audience

The Oracle Engineered System Healthchecks plug-in is for users who perform administrative and problem-resolution tasks on Oracle Engineered Systems.

2 Supported Versions

The Oracle Engineered System Healthchecks plug-in supports the following software and platform versions:

  • Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.3 and higher.

  • Oracle Linux release 5.3 or later for:

    • Linux x86

    • Linux x86-64 (64-bit)

  • Oracle Solaris 11 or later for:

    • x86-64 (64-bit)

    • SPARC64 (64-bit)

3 Supported Hardware

The Oracle Engineered Systems Healthchecks plug-in supports Engineered Systems hardware, including all variants of Exadata Database Machine, SPARC SuperCluster, and Exalogic Elastic Cloud.

Note:

Support for Exalytics systems is currently not available.

4 Plug-in Prerequisites

The following prerequisites must be met before you can deploy the plug-in:

5 Downloading the Plug-in

You can download plug-ins in online or offline mode. Online mode refers to an environment where you have Internet connectivity, and can download the plug-in directly through Enterprise Manager from My Oracle Support. Offline mode refers to an environment where you do not have Internet connectivity, or where the plug-in is not available from My Oracle Support.

See the Managing Plug-ins chapter in the Oracle Enterprise Manager Cloud Control Administrator's Guide for details on downloading the plug-in in either mode:

http://docs.oracle.com/cd/E24628_01/doc.121/e24473/plugin_mngr.htm#CJGBEAHJ

6 Deploying the Plug-in

You can deploy the plug-in to an Oracle Management Service instance using the Enterprise Manager Cloud Control console, or using the EM Command Line Interface (EMCLI). While the console enables you to deploy one plug-in at a time, the command line interface mode enables you to deploy multiple plug-ins at a time, thus saving plug-in deployment time and downtime, if applicable.

See the Managing Plug-ins chapter in the Oracle Enterprise Manager Cloud Control Administrator's Guide for instructions on deploying the plug-in:

http://docs.oracle.com/cd/E24628_01/doc.121/e24473/plugin_mngr.htm#CJGCDHFG

7 Discovering Targets

After successfully deploying the plug-in, follow these steps to add the plug-in target to Cloud Control for central monitoring and management:

  1. Log in to Enterprise Manager Cloud Control.

  2. Click Setup, then Add Targets, and finally Add Targets Manually.

  3. On the Add Targets Manually page:

    1. Select Add Targets Declaratively by Specifying Target Monitoring Properties.

    2. From the Target Type drop-down, select the Oracle Engineered System Healthchecks target type.

    3. Click the Search icon search icon and select a monitoring agent from the pop-up window.

    4. Click Add Manually.

  4. On the Add Oracle Engineered System Healthchecks page, specify the name for the target instance in the Target Name field. Set the following values for the instance properties:

    • Exachk Results Directory: specify the path of the exachk output directory.

    • Max interval allowed (in days) between consecutive Exachk runs: 31

      Where 31 is the default, auto-filled value of days between consecutive runs of the Exachk tool. You can leave this value as is to use the default of 31 days; otherwise, you can change the value to the customized time span you need (such as 20, 45, 60, etc.).

      Note:

      A set of Global Properties is available. You can use these optional fields to include additional metadata about the target instance you are creating.

  5. Click OK.

Note:

After you deploy and configure the plug-in to monitor one or more targets in the environment, you can customize the monitoring settings of the plug-in. This alters the collection intervals and threshold settings of the metrics to meet the particular needs of your environment. If you decide to disable one or more metric collections, this could impact the reports that the metric is a part of.

8 Prerequisite for Executing the Exachk Tool

The following environment variables should be set up before executing exachk (all three modes: -a, -s, -S).

  1. The environment variable RAT_COPY_EM_XML_FILES should be set. Setting this environment variable will also enable copying of results files on all the nodes in the cluster:

    export RAT_COPY_EM_XML_FILES=1

  2. Specify the exachk output path. The result files will be copied to this location on all the nodes in the cluster:

    export RAT_OUTPUT=[exachk output directory]

    Note:

    The exachk output directory should exist on all nodes of the cluster.

9 Run the Exachk Tool

The Exachk tool should be run in "All" mode, using the -daemon -a options. The exachk daemon must be started and it should be active before running exachk in "All" mode.

The exachk daemon makes it possible to run the -a mode silently.

The following sections are described:

9.1 Run the Exachk Tool Automatically on Engineered Systems

The Exachk tool bundle can be downloaded from "Oracle Exadata Database Machine exachk or HealthCheck" (Doc ID 1070954.1) available in My Oracle Support (https://support.oracle.com):

https://support.oracle.com/epmos/faces/DocumentDisplay?id=1070954.1

To use the exachk daemon:

  1. Start the exachk daemon as oracle user, using -d start option:

    ./exachk -d start.

  2. It will prompt for all the information needed by it to make future runs of exachk silent.

  3. Run the Exachk tool. It should be run in "All" mode while the exachk daemon in active, using the -daemon -a options:

    ./exachk -daemon -a

Refer to Frequency of Running the Exachk Tool to schedule the Exachk execution.

9.2 Run the Exachk Tool Automatically on Exalogic

The Exachk tool bundle can be downloaded from "Exachk Health-Check Tool for Exalogic" (Doc ID 1449226.1) available in My Oracle Support (https://support.oracle.com):

https://support.oracle.com/epmos/faces/DocumentDisplay?id=1449226.1

To use the exachk daemon:

  1. Start the exachk daemon as root user, using -d start option:

    ./exachk -d start.

  2. It will prompt for all the information needed by it to make future runs of exachk silent.

  3. Run the Exachk tool. It should be run in "All" mode while the exachk daemon in active, using the -daemon -a options:

    ./exachk -daemon -a

For limitations and more details on running Exachk on Exalogic, refer to "Exachk Health-Check Tool for Exalogic" (Doc ID 1449226.1) in My Oracle Support:

https://support.oracle.com/epmos/faces/DocumentDisplay?id=1449226.1

9.2.1 Configuring for Exalogic

For configuring Oracle Engineered System Health checks plug-in with the Exalogic virtualized configuration, the Enterprise Manager management agent selected must be running on the Exalogic Enterprise Controller vServer. If an agent is not deployed to the Exalogic Enterprise Controller vServer, refer to the instructions in the Installing Oracle Management Agent chapter of the Oracle Enterprise Manager Cloud Control Basic Installation Guide to deploy an agent:

http://docs.oracle.com/cd/E24628_01/install.121/e22624/install_agent.htm#CACJEFJI

Additionally, make sure that the share with the Exachk tool is mounted with read-only permissions on Exalogic Enterprise Controller vServer. In the event the share is not mounted, mount it and make sure that the mount has a common read-only group permission with the Oracle user of the vServer, thus enabling read-only access (this operation can be performed as root):

mount -t nfs -r <Storage appliance ip>:<path> <path>

For example:

mkdir -p /mnt/common/general/
mount -t nfs -r 192.168.10.15:/export/common/general /mnt/common/general/

Note:

The -r option is for "read only" permissions.

Once configured, network communication is standardized to communicate to the Enterprise Manager OMS server.

Essentially, the Agent running in the Enterprise Controller vServer communicates to the OHS proxy via the IPoIB-virt-admin network, and the OHS proxy in turn communicates to the Enterprise Manager OMS server via the EoIB-external-mgmt network.

10 Frequency of Running the Exachk Tool

The Exachk tool should be run in the following scenarios:

  • Once a month on a regular basis.

  • After taking corrective actions for the failures reported by the Exachk tool.

11 Metrics Collected

The checks executed by the Exachk tool are placed under the metric ExadataResults.

You can also evaluate the metrics any time by running the following command:

./emctl control agent runCollection <targetName>:oracle_exadata_hc ExadataResults

12 Reports

A healthcheck results report, which will be visible in the target's home page, lists all metrics along with the failed checks, irrespective of whether the alert is raised or not.

You can also view the report in a printable view using the menu from the target home page:

  1. On the target home page click Targets, then Information Publisher Reports.

  2. Click Exachk Execution Results.

  3. Click Continue.

You will be directed to the reports screen where you can filter the results on the metric type. Figure 1 shows an example of the report results visible on the target homepage.

This Report is bundled along with the plug-in and will be created in the OMS during deployment.

Figure 1 Report Results Example

Report Results Example

12.1 Downloading the Report as a .csv File

To download the Exachk Execution Results report as a .csv file:

  1. On the target home page click Oracle Engineered System Healthchecks then Information Publisher Reports.

  2. Click the Exachk Execution Results link. Click Continue.

  3. On the Exachk Execution Results page, you can filter the results by metric type.

    In the Exachk Execution Results table region, click the download icon Download CSV file in the top-right corner to download the report as a comma-delimited .csv file.

13 Alerts

By default the plug-in is designed to raise alerts for the following checks only:

  • Exadata-specific alerts:

    1. Verify Disk Cache Policy on Database Server.

    2. Verify Database Server disk controllers use writeback cache.

    3. Verify RAID Controller Battery Condition (Database Server).

    4. Verify RAID Controller Battery Temperature (Database Server).

    5. Verify Database Server Virtual Drive Configuration.

    6. Verify Database Server Physical Drive Configuration.

    7. Imageinfo version comparison across database and storage servers.

    8. Verify Hardware and Firmware on Database and Storage Servers (CheckHWnFWProfile) [Database Server].

    9. Verify Software on Storage Servers (CheckSWProfile.sh).

  • Exalogic-specific alerts:

    1. NFS Mount Point - Attribute Caching.

    2. /conf/configvalid File.

  • Generic alerts:

    1. Exachk not running.

    2. Results and Exception file(s) missing.

    3. Metric Parsing Failed.

Note:

None of the above checks are specific to Exalytics.

13.1 Disable Alerts Using the Monitoring Template

To change the behavior of alerts, you can apply the Oracle provided Engineered System Healthchecks No Alert template to disable alerts for all checks executed by the Exachk tool.

Follow the steps below to apply the Oracle-provided templates:

  1. Log in to Enterprise Manager Cloud Control.

  2. Click Enterprise from the upper-left-corner of the Cloud Control home page. Click Monitoring from the drop-down menu, then Monitoring Templates.

  3. To search for the template:

    1. In the Template Name field, enter Oracle provided Engineered System Healthchecks No Alert template.

    2. Click the check box to enable search to "Display Oracle Certified templates."

    3. Click Go.

  4. Select the radio button next to the template name and click Apply.

  5. In the Apply Monitoring Template screen, click Add and select the targets on which you wish to apply this template from the pop-up window. Click OK.

    Note:

    You can enable the alerts by applying the Oracle provided Engineered System Healthchecks template, following the same steps as above. Applying this template will enable alerts for the check names specified in Alerts section.

13.2 Define a Monitoring Template

To define your own monitoring template and enable alerts for checks, follow the steps below:

  1. Identify the Check ID of the check you want to be alerted of. The Check ID is used to uniquely identify each check. For the check name you want to set up an alert, get the corresponding Check ID from the Exachk Execution Results report.

  2. Click Enterprise from the upper-left corner of the Enterprise Manager Cloud Control home page. Select Monitoring and finally Monitoring Templates as shown below:

    Surrounding text describes templates_menu.gif.

  3. Click the Create button.

  4. On the Create Monitoring Template page, select Target Type. In the Target Type drop-down menu, select Oracle Engineered System Healthchecks as shown in Figure 2:

    Figure 2 Create Monitoring Templates

    Surrounding text describes Figure 2 .

    Click Continue.

  5. In the General tab of the Create Monitoring Template screen, enter a template name and description.

  6. In the Metric Threshold tab of the Create Monitoring Template screen, click edit (pencil icon) next to the status column of Engineered System Healthchecks metric in the tree structure displayed. This will redirect to the Edit Advance Settings: Status page.

  7. Click the Add button on the top right corner of the Create Monitoring Template screen. In the text box that appears, enter the Check ID that you want to be alerted for Check ID. Enter FAIL as the critical threshold.

    Repeat for all Check IDs that you want alerts to be enabled.

  8. Click Continue and then OK to close the Edit Advance settings: Status page.

  9. In the Monitoring Templates screen, select the radio button next to the template you created and click Apply.

  10. In the Apply Monitoring Template screen, click Add and select the targets on which you wish to apply this template. Click OK.

13.3 Disable a Specific Alert

To disable a specific alert, follow the steps below:

  1. Select the Metric and Policy Settings under the Monitoring menu in the target home page.

  2. In this page, update the value for the Critical Threshold column to blank for the alert that you wish to disable and click OK.

14 Upgrading the Plug-in

The Self Update feature allows you to expand Enterprise Manager's capabilities by updating Enterprise Manager components whenever new or updated features become available. Updated plug-ins are made available via the Enterprise Manager Store, an external site that is periodically checked by Enterprise Manager Cloud Control to obtain information about updates ready for download. See the Updating Cloud Control chapter in the Oracle Enterprise Manager Cloud Control Administrator's Guide for steps to update the plug-in:

http://docs.oracle.com/cd/E24628_01/doc.121/e24473/self_update.htm

15 Undeploying the Plug-in

See the Managing Plug-ins chapter in the Oracle Enterprise Manager Cloud Control Administrator's Guide for steps to undeploy the plug-in:

http://docs.oracle.com/cd/E24628_01/doc.121/e24473/plugin_mngr.htm#CJGEFADI

16 Documentation Accessibility

For information about Oracle's commitment to accessibility, visit the Oracle Accessibility Program website at http://www.oracle.com/pls/topic/lookup?ctx=acc&id=docacc.

Access to Oracle Support

Oracle customers that have purchased support have access to electronic support through My Oracle Support. For information, visit http://www.oracle.com/pls/topic/lookup?ctx=acc&id=info or visit http://www.oracle.com/pls/topic/lookup?ctx=acc&id=trs if you are hearing impaired.

System Monitoring Plug-In Installation Guide for Oracle Engineered System Healthchecks, Release 12.1.0.5.0

E27420-05

Copyright © 2015, Oracle and/or its affiliates. All rights reserved.

This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited.

The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us in writing.

If this is software or related documentation that is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, then the following notice is applicable:

U.S. GOVERNMENT END USERS: Oracle programs, including any operating system, integrated software, any programs installed on the hardware, and/or documentation, delivered to U.S. Government end users are "commercial computer software" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such, use, duplication, disclosure, modification, and adaptation of the programs, including any operating system, integrated software, any programs installed on the hardware, and/or documentation, shall be subject to license terms and license restrictions applicable to the programs. No other rights are granted to the U.S. Government.

This software or hardware is developed for general use in a variety of information management applications. It is not developed or intended for use in any inherently dangerous applications, including applications that may create a risk of personal injury. If you use this software or hardware in dangerous applications, then you shall be responsible to take all appropriate fail-safe, backup, redundancy, and other measures to ensure its safe use. Oracle Corporation and its affiliates disclaim any liability for any damages caused by use of this software or hardware in dangerous applications.

Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.

Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices. UNIX is a registered trademark of The Open Group.

This software or hardware and documentation may provide access to or information about content, products, and services from third parties. Oracle Corporation and its affiliates are not responsible for and expressly disclaim all warranties of any kind with respect to third-party content, products, and services unless otherwise set forth in an applicable agreement between you and Oracle. Oracle Corporation and its affiliates will not be responsible for any loss, costs, or damages incurred due to your access to or use of third-party content, products, or services, except as set forth in an applicable agreement between you and Oracle.