The Oracle Enterprise Manager Connector Framework requires a web service interface for exchanging event information with OMU. Since OMU does not come with a web services front end, an Oracle-provided web service front end must be installed before Oracle Enterprise Manager can exchange event information with OMU. Additionally, an Oracle back-end Agent must also be installed on the same system as the OMU server.
This chapter provides information to assist in troubleshooting integration issues with HP OMU. The chapter focuses on troubleshooting issues in the web service front-end and the back-end Agent.
This chapter discusses the following topics:
Before you start the troubleshooting process, ensure that you have done the following:
Installed the OMU Connector as specified in Installing the Connector in Enterprise Manager.
Installed and started the Oracle OMU Agent as specified in Installing and Running the Oracle Agent for HP Operations Manager.
Installed, started, and tested the OMU Web service as specified in Installing the Oracle Web Service for HP Operations Manager.
Created a connector instance as specified in Creating a Connector Instance.
Configured the connector instance as specified in Configuring the Connector.
Set up one or more rules as specified in Setting Events Rules to forward events to the connector instance.
If all the actions above have been completed and the connector is not working, perform the steps in Diagnosing the Problem.
To diagnose the problem:
Verify that the OMU Web service has been successfully started and the WSDL for the OMU Web service can be accessed from the system where the OMU Web service is installed. Perform the following steps to do this:
Perform the steps in Using the Correct URL for OMU Web Service Operations to determine the URL used by the OMU Web Service.
Open a browser and paste the URL from the previous step in the address window. Append ?WSDL to the end of the URL and attempt to load the URL.
The WSDL should be loaded if the Adapter is operational.
Note:
A WSDL is an XML file that describes the web service.
If the WSDL cannot be loaded, this indicates the Web service had startup issues. See Troubleshooting Web Service Startup Errors to diagnose the problem.
Verify that the WSDL for the OMU Web service can be accessed from the system where the Enterprise Manager server is installed.
Open a browser at the Enterprise Manager server and copy the URL from step 1-b above to the address window. The host name for the URL will be localhost. Change localhost to the actual host name or IP address of the system where the OMU Web service is installed. If you specify a host name, make sure that the host name is recognized at the Enterprise Manager server system. This can be done using the ping command.
For example, if the OMU Web service is installed on the server with a host name of SDServer01 and the URL listed in framework.log is:
http://localhost:8080/services/hpovou/EventService
The URL used at the Enterprise Manager server system would be:
http://OMServer01:8080/services/hpovou/EventService
Attempt to load the WSDL by appending ?WSDL at the end of the URL. If the WSDL cannot be loaded, either the host name is not recognized at the Enterprise Manager system, or there is a connectivity issue between the two systems. If you specified a host name, try using the IP address instead of the host name in the URL. If it still does not load, you have a connectivity problem. You will need to consult with your IT department to resolve this issue.
Verify that the OMU Connector specifies the correct URL for the createEvent and updateEvent operations.
Log into the Oracle Enterprise Manager console with an account that has Super Administrator permissions.
From the Setup menu of the Enterprise Manager console, select Extensibility, then Management Connectors.
The Management Connectors page appears, which shows the installed connectors.
Click on the Configure icon associated with the OMU Connector. This invokes edit mode, enabling you to configure the connector.
Verify that the URL identified in step 2-b is specified for both operations (createEvent and updateEvent). If any of the operations are incorrect, change to the correct URL.
Note:
Do not append the WSDL to the end of the URL that is specified here.
Click OK.
If no errors were found in the previous steps, this rules out connectivity issues between the connector and the OMU Web Service. The problem must originate in the OMU Web Service or the Oracle OMU Agent. See Troubleshooting Web Service Operations Errors for information on diagnosing errors.
To identify the cause of a startup failure, navigate to the adapters/log directory in the OMU Web Service installation directory and open the framework.log file in a text editor. Search for Exception to find any errors in the file. If the file does not exist, it indicates that there is a problem locating or executing the JVM. See JVM Errors for information about resolving JVM issues.
Listed below are some possible Exceptions, an explanation of the root cause, and a description of the solution.
java.net.BindException: Address already in use: bind
This error indicates that the web service could not start because of a port conflict. There are two possible causes for this error:
Another application is using a port that the Web service is configured to use. If the web service is configured to use SSL, the port number is 8443. If it is not configured to use SSL, the port number is 8080.
There are two possible solutions to this. You can change the other application to use a different port or you can change the OMU Web Service to use a different port. To change the OMU Web Service to use a different port, see Changing Default Port Numbers.
An instance of the Web service is already running. If this is the case, no change is required. You should only run one instance of the Web service at a time.
org.springframework.beans.factory.BeanInitializationException: Could not load properties; nested exception is java.io.FileNotFoundException: … framework.properties (Permission denied)
This error indicates that the web service could not start because the permissions on the framework.properties file in the conf directory were not set correctly.
To solve the problem, change the permissions to give the account or group under which the OMU Web Service runs read and execute permissions.
For any other startup errors, consult with Oracle Support.
The OMU Web Service requires version 1.6 of the JVM. If multiple versions of the JVM are installed on the system, it is possible that an older version of the JVM is being executed whenever the web service starts.
On UNIX systems, the JAVA_HOME environment variable must be set to the directory where JDK 1.6 is installed in the shell where the web service is started. To properly start the web service on a UNIX platform, perform the following:
Set the JAVA_HOME environment variable to the JDK 1.6 installation directory.
Navigate to the adapters/bin sub-directory in the Web service installation directory.
Execute the ./service.sh start command.
On Windows systems, perform the following to insure that JDK 1.6 is used when starting the Web service.
Navigate to the adapters/bin sub-directory in the Web service installation directory.
Run the iWaveAdaptersw.exe executable.
Click on the Java tab.
Make sure that the Use Default check box is not checked.
In the Java Virtual system box, specify the path to the jvm.dll file in the JDK 1.6 installation directory.
Click OK.
Perform the following steps to identify and configure the connector to use the correct URL for OMU Web Service operations.
Open a command terminal on the system where the OMU web service is installed.
Change the working directory to the adapters/log directory in the OMU web service installation directory.
Open the framework.log file in a text editor.
Go to the bottom of the file and search backwards for the string “Setting the server's publish address to be." Continue searching backwards until you find the URL that contains EventService. The URL listed here is the URL that should be specified for the createEvent and updateEvent operations.
Log in to the Oracle Enterprise Manager console with an account that has 'Super Administrator' permissions.
From the Setup menu of the Enterprise Manager console, select Extensibility, then Management Connectors.
The General tab of the Configure Management Connector page appears.
Click on the name of the OMU connector.
Verify that the URL identified in step 4 is specified for the createEvent and updateEvent operations.
If any of the operations are incorrect, change to the correct URL and click OK.
Perform the following diagnostic steps if messages are not being generated or updated as expected in Operations Manager.
Verify that the event that was triggered is referenced in a rule that forwards events to the OMU connector.
Determine the error that Oracle Enterprise Manager has reported by examining the log file for errors. Perform the following steps to locate errors in the log file:
Open the emoms.trc file in a text editor. The file is located in the log directory at the following location:
<EM_INSTANCE_BASE>/em/<OMS_NAME>/sysman/log/
... where <EM_INSTANCE_BASE> is the OMS Instance Base directory. By default, the OMS Instance Base directory is gc_inst, which is present under the parent directory of the Oracle Middleware Home.
For example, if the Oracle Middleware Home is /u01/app/Oracle/Middleware, then the instance base directory is /u01/app/Oracle/Middleware/gc_inst, and the log and trace files are available in the /u01/app/Oracle/Middleware/gc_inst/em/EMGC_OMS1/sysman/log/ directory path.
Go to the bottom of the file and search backwards for "Caused by."
Lines that start with "Caused by" contain error information. The error information appears after the text in the line that reads "oracle.sysman.emSDK.webservices.outbound.WSInvokeException: caught WebServiceException :".
Diagnose the problem based on the error information. See Resolving Errors from Oracle Enterprise Manager for information on troubleshooting common error events.
This section provides cause and solution information on troubleshooting common error messages. Find the error message in Table 7-1 that matches your error message, then refer to the corresponding section(s) indicated under Possible Cause for instructions on diagnosing and correcting the problem.
Table 7-1 Enterprise Manager Error Messages
| Error Message | Possible Cause |
|---|---|
|
javax.xml.soap.SOAPException: javax.xml.soap.SOAPException: Bad response: 403 Forbidden from url |
Invalid Web Service Credentials |
|
jjavax.xml.soap.SOAPException: javax.xml.soap.SOAPException: Message send failed: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target |
SSL Not Configured in Enterprise Manager |
|
javax.xml.soap.SOAPException: javax.xml.soap.SOAPException: Message send failed: Connection refused |
OMU Web Service is Down |
|
javax.xml.soap.SOAPException: javax.xml.soap.SOAPException: Message send failed: No route to host |
Invalid IP Address |
|
javax.xml.soap.SOAPException: javax.xml.soap.SOAPException: Bad response: 404 Not Found from url … |
Invalid Port Number or Invalid URL Path |
|
javax.xml.soap.SOAPException: javax.xml.soap.SOAPException: Message send failed: Connection timed out |
Firewall Blocking Access |
|
java.net.ConnectException: Connection refused |
Unable to Connect to Oracle OMU Agent |
|
javax.xml.soap.SOAPException: javax.xml.soap.SOAPException: Message send failed: hostname |
Unknown Host |
|
javax.xml.transform.TransformerConfigurationException: Could not compile stylesheet |
Invalid XML Format |
|
Error creating event: opcmsg_get_msgids failed to return a result after 10 attempts |
Timeout |
|
Error creating event: opc_connect failed: -50: Login failed. Either the userid is not a OVO operator or the password is invalid |
Invalid Agent User Name or Password |
|
Error creating event: keyword 'object' is missing |
Request Missing Required Field |
Invalid Web Service Credentials
Cause
The user name or password for accessing the OMU web service is incorrect.
Solution
Log in to the Oracle Enterprise Manager console with an account that has Super Administrator privileges.
From the Setup menu of the Enterprise Manager console, select Extensibility, then Management Connectors.
The Management Connectors page appears.
Click on the name of the OMU Connector.
This invokes Edit mode, enabling you to configure the connector.
Correct the OMU Web Service Username and OMU Web Service Password fields and click OK.
If the credentials supplied appear to have been entered correctly but still do not work, a typographical error might have been made during the setup of the web service. The recommended option is to reset the web service credentials and attempt this again. Refer to Changing Web Service Credentials for the steps required to reset the credentials.
SSL Not Configured in Enterprise Manager
Cause
The SSL handshake between the Oracle Enterprise Manager Connector Framework and the OMU web service failed. This failure occurs because Oracle Enterprise Manager is not configured correctly with the SSL certificate for the OMU web service. The SSL certificate the OMU web service uses must be imported into the Enterprise Manager key store. The certificate is either missing from the key store or does not match the SSL certificate provided by the OMU web service.
Solution
Import the SSL certificate from the OMU web service into the Enterprise Manager key store. See Configuring Enterprise Manager to Use SSL for details on setting up Oracle Enterprise Manager with the OMU SSL certificate.
OMU Web Service Is Down
Cause
The OMU web service is down.
Solution
Perform the following steps to check the status of the web service and start it if necessary.
If the OMU web service is installed on a Unix system:
Open a command terminal on the system where the OMU web service is installed.
Change the working directory to the adapters/bin directory in the OMU web service installation directory.
Enter the following command:
./service.sh status
If the command indicates that the service is not running, restart the web service as specified in Installing the Web Service on Unix.
If the OMU web service is installed on a Windows system:
Open a command terminal on the system where the OMU web service is installed.
Change the working directory to the adapters/log directory in the OMU web service installation directory.
Open the framework.log file in a text editor.
Go to the bottom of the file and search backwards for the string iWave Adapter Framework. If the last occurrence found is iWave Adapter Framework Started, this indicates that the web service is started.
If the web service is not started, restart the web service as specified in Running the Web Service on Windows.
Invalid XML Format
Cause
The connector framework could not process the request because the XSL file was formatted incorrectly. This problem should not occur unless the connector has been customized.
Solution
Examine any changes made to the XSL template files for mistakes that could have caused the problem. If you cannot find the problem manually, load the XSL in a utility that performs XML validation.
Unable to Connect to Oracle OMU Agent
Cause
The OMU web service could not connect to the Oracle OMU Agent. Some of the causes could be:
The Oracle OMU Agent is down.
A configuration error in the Oracle OMU Agent or web service is preventing communication.
Solution
Perform the following steps to verify that the Oracle OMU Agent is operational:
Open a command prompt at the system where the OMU server is installed.
Change the working directory to ...
<OVAG_INST>/ovo-agent/scripts
... where <OVAG_INST> is the Oracle OMU Agent installation directory.
Enter the following command to attempt to start the Agent:
./start.sh
If the Agent is already started, the script displays information indicating that the send/receive Agent is already running.
Perform the following steps to verify that the configuration is correct:
Open a command prompt window at the OMU server system and change the working directory to ...
<OMUA_INSTALL>/ovo-agent/conf
... where <OMUA_INSTALL> is the directory where the Oracle OMU Agent is installed.
Open the ovooper.txt file in a text editor.
Search for the line containing the string ListenHost=.
This parameter defines the hostname/IP address that the Oracle OMU Agent uses when listening for requests from the OMU web service. Make a note of this value.
Search for the line containing the string ListenPort=.
This parameter defines the port that the Oracle OMU Agent uses when listening for requests from the OMU web service. Make a note of this value.
Open a command prompt window at the system where the OMU web service is installed, and change the working directory to:
<OMUWS_INSTALL>/adapters/conf
... where <OMUWS_INSTALL> is the directory where the OMU web service is installed
Open the framework.properties file in a text editor.
Search for the hpovou.xmlagent parameter. The format of the parameter is:
http\://<hostname>\:<port>
Verify that the hostname/IP address and the port number specified in this file match the information in steps 3 and 4.
If the information does not match, correct the information and save the file.
If a change was required in the previous step, stop and then start the web service as instructed in Running the Web Service on Unix and Running the Web Service on Windows.
Request Missing Required Field
Cause
The Oracle OMU Agent could not process the request because key information was missing. This error should not occur if you are using the default configuration. It only occurs if the default mappings were customized and a required field was omitted.
XSolution
You need to modify the XSL file to generate the XML node that maps to the missing OMU field. See Customizing Microsoft HP Operations Manager for information on customizing the default mappings.
Unknown Host
Cause
The system does not recognize the host name specified in the URL.
Solution
You have the following options for addressing this issue.
Coordinate with the system administrator to change the system configuration to recognize the host name.
Specify the IP address in the URL instead of the host name. To do this, perform the following steps:
Determine the IP address of the system where the OMU web service is installed.
Log in to the Oracle Enterprise Manager console with an account that has Super Administrator privileges.
From the Setup menu of the Enterprise Manager console, select Extensibility, then Management Connectors.
The Management Connectors page appears, which shows the installed connectors.
Click on the name of the OMU Connector. This invokes edit mode, enabling you to configure the connector.
Change the host name to the IP address in the URL specified for the createEvent and updateEvent operations.
Click OK.
Make sure that the host name resolves to the correct IP address. To determine the IP address of the host name, issue the ping <hostname> command, where <hostname> is the actual host name. This lists the IP address that was resolved for the host name. If this is incorrect, the system administrator needs to investigate why it is incorrect.
If the hostname/IP address appears to be correct, try to ping the system where the OMU web service is installed using the hostname/IP address. If the ping fails, the system administrator needs to investigate why there is no connectivity.
Invalid IP Address
Cause
The IP address specified in the URL is invalid, or the network is down.
Solution
Verify that the hostname/IP address configured for the connector is correct:
Log in to the Oracle Enterprise Manager console with an account that has Super Administrator privileges.
From the Setup menu of the Enterprise Manager console, select Extensibility, then Management Connectors.
The Management Connectors page appears, which shows the installed connectors.
Click on the name of the OMU Connector. This invokes edit mode, enabling you to configure the connector.
Verify that the hostname/IP address specified in the URL for the createEvent and updateEvent operations are correct.
If the hostname/IP address is incorrect, provide the correct values and click OK.
If the URLs specify a host name, make sure that the host name resolves to the correct IP address. To determine the IP address of the host name, issue the ping <hostname> command, where <hostname> is the actual host name. This lists the IP address that was resolved for the host name. If this is incorrect, the system administrator needs to investigate why it is incorrect.
If the hostname/IP address appears to be correct, try to ping the system where the OMU web service is installed using the hostname/IP address. If the ping fails, the system administrator needs to investigate why there is no connectivity.
Invalid Port Number
Cause
The port number specified in the URL is invalid.
Solution
Verify that the hostname/IP address configured for the connector is correct:
Log in to the Oracle Enterprise Manager console with an account that has Super Administrator privileges.
From the Setup menu of the Enterprise Manager console, select Extensibility, then Management Connectors.
The Management Connectors page appears, which shows the installed connectors.
Click on the name of the OMU Connector. This invokes edit mode, enabling you to configure the connector.
Verify that the port number specified in the URL for the createEvent and updateEvent operations are correct.
If the port number is incorrect, provide the correct value and click OK.
Invalid URL Path
Cause
The web service received the request and rejected it because an invalid path was specified in the URL.
Solution
Perform the following steps to test the URL the connector is using.
Log in to the Oracle Enterprise Manager console with an account that has Super Administrator privileges, entering the appropriate password, then clicking Login.
From the Setup menu of the Enterprise Manager console, select Extensibility, then Management Connectors.
The Management Connectors page appears, which shows the installed connectors.
Click on the name of the OMU Connector. This invokes edit mode, enabling you to configure the connector.
Select and copy the URL specified for the createEvent operation.
Open an internet browser on the system where the Oracle Enterprise Manager server is installed.
In the address window, enter the URL that was copied in step 6 above. Add ?wsdl to the end of the URL. The URL should appear similar to the following example:
http://[Hostname]:8080/services/hpovou/EventService?wsdl
[Hostname] is the actual host name or IP address where the OMU web service is installed.
If the WSDL is loaded, this confirms that the URL is correct. If it fails to load, there is a problem with the URL. Perform the steps specified in Using the Correct URL for OMU Web Service Operations to configure the connector to use the correct URL.
Firewall Blocking Access
Cause
A firewall is blocking access to the system where the OMU Web Service is installed.
Solution
Contact your IT department to give Enterprise Manager access to the port the OMU Web Service uses. Perform the steps specified in Using the Correct URL for OMU Web Service Operations to determine the URL used by the OMU Web Service. The port number specified in the URL is the port number the IT department should open in the firewall.
Timeout
Cause
The web service received the request and successfully sent the request to the Agent. The Agent successfully submitted the create/update request to the OMU API. The Agent timed out waiting for OMU to provide an ID of the resulting message that was created. This error occurs because OMU is not properly configured to create messages for Enterprise Manager.
Solution
Verify that the steps specified in Integrating OMU with Enterprise Manager have been performed to properly configure OMU for Enterprise Manager messages.
Invalid Agent Username or Password
Cause
The credentials that the agent is configured to use to access the OMU API are incorrect.
Solution
Perform the following steps to verify that the Oracle OMU Agent is operational:
Open a command prompt at the system where the OMU server is installed.
Change the working directory to ...
<OVAG_INST>/ovo-agent/scripts
... where <OVAG_INST> is the Oracle OMU Agent installation directory.
Enter the following command to reconfigure the Agent and reset the username/password information used by the Agent to access the API. See Installing the Agent for details on the configuration process.
./configure.sh
Stop and then start the Oracle OMU Agent as specified in Running and Stopping the Agent.