The Oracle Enterprise Manager Connector Framework requires a web service interface for exchanging event information with Netcool/OMNIbus. Since Netcool/OMNIbus does not come with a web services front end, an Oracle-provided web service front end must be installed before Oracle Enterprise Manager can exchange event information with Netcool/OMNIbus. Additionally, an Oracle Agent for Netcool/OMNIbus must also be installed.
This chapter provides information to assist in troubleshooting integration issues with IBM Netcool/OMNIbus. The chapter focuses on troubleshooting issues in the web service front-end and the Oracle Agent.
This chapter discusses the following topics:
Before you start the troubleshooting process, ensure that you have done the following:
Installed the IBM Tivoli Netcool/OMNIbus Connector as specified in Installing the Connector in Enterprise Manager.
Installed and started the Oracle Agent for Netcool/OMNIbus as specified in Installing the Oracle Agent for Netcool/OMNIbus.
Installed, started, and tested the web service for Netcool/OMNIbus service as specified in Installing the Web Service for Netcool/OMNIbus Front-End.
Made the necessary modifications to Netcool/OMNIbus as specified in Modifying Netcool/OMNIbus.
Created a connector instance as specified in Creating a Connector Instance.
Configured the connector instance as specified in Configuring the Connector.
Set up one or more rules as specified in Setting Event Rules to forward events to the connector instance.
If all the actions above have been completed and the connector is not working, perform the steps in Diagnosing the Problem.
To diagnose the problem:
Verify that the web service for Netcool/OMNIbus has been successfully started and the WSDL for the web service can be accessed from the system where the web service for Netcool/OMNIbus is installed:
Perform the steps in Determining the URL Used by the Web Service to determine the URL used by the web service for Netcool/OMNIbus.
Open a browser and paste the URL from the previous step in the address window. Append ?WSDL to the end of the URL and attempt to load the URL.
The WSDL should be loaded if the Adapter is operational.
Note:
A WSDL is an XML file that describes the web service.If the WSDL cannot be loaded, this indicates the web service had startup issues. See Troubleshooting Web Service Startup Errors to diagnose the problem.
Verify that the WSDL for the web service for Netcool/OMNIbus can be accessed from the system where the Enterprise Manager server is installed:
Open a browser at the Enterprise Manager server and copy the URL from step 1-b above to the address window. The host name for the URL will be localhost. Change localhost to the actual host name or IP address of the system where the web service for Netcool/OMNIbus is installed. If you specify a host name, make sure that the host name is recognized at the Enterprise Manager server system. This can be done using the ping command.
For example, if the web service for Netcool/OMNIbus is installed on the server with a host name of SDServer01 and the URL listed in framework.log is:
https://localhost:8443/services/omnibus/EventService
The URL used at the Enterprise Manager server system would be:
https://OMServer01:8443/services/omnibus/EventService
Attempt to load the WSDL by appending ?WSDL at the end of the URL. If the WSDL cannot be loaded, either the host name is not recognized at the Enterprise Manager system, or there is a connectivity issue between the two systems. If you specified a host name, try using the IP address instead of the host name in the URL. If it still does not load, you have a connectivity problem. You will need to consult with your IT department to resolve this issue.
Verify that the Netcool/OMNIbus Connector specifies the correct URL for the createEvent and updateEvent operations:
Log in to Oracle Enterprise Manager Cloud Control with an account that has Super Administrator permissions.
From the Setup menu, select Extensibility, then Management Connectors.
The Management Connectors page appears, which shows the installed connectors.
Click the Configure icon associated with the Netcool/OMNIbus Connector. This invokes edit mode, enabling you to configure the connector.
Verify that the modified URL from step 2-a is specified for both operations (createEvent and updateEvent). If any of the operations are incorrect, change to the correct URL.
Note:
Do not append the WSDL to the end of the URL that is specified here.Click OK.
If no errors were found in the previous steps, this rules out connectivity issues between the connector and the web service for Netcool/OMNIbus. The problem must originate in the web service or the Oracle Agent. See Troubleshooting Web Service Operations Errors for information on diagnosing errors.
To identify the cause of a startup failure, navigate to the adapters/log directory in the web service for Netcool/OMNIbus installation directory and open the framework.log file in a text editor. Search for Exception to find any errors in the file. If the file does not exist, it indicates that there is a problem locating or executing the JVM. See JVM Errors for information about resolving JVM issues.
Listed below are some possible Exceptions, an explanation of the root cause, and a description of the solution.
java.net.BindException: Address already in use: bind
This error indicates that the web service could not start because of a port conflict. There are two possible causes for this error:
Another application is using a port that the web service is configured to use. If the web service is configured to use SSL, the port number is 8443. If it is not configured to use SSL, the port number is 8080.
There are two possible solutions to this. You can change the other application to use a different port or you can change the web service for Netcool/OMNIbus to use a different port. To change the web service to use a different port, see Changing Default Port Numbers in Appendix A.
An instance of the web service is already running. If this is the case, no change is required. You should only run one instance of the web service at a time.
org.springframework.beans.factory.BeanInitializationException: Could not load properties; nested exception is java.io.FileNotFoundException: … framework.properties (Permission denied)
This error indicates that the web service could not start because the permissions on the framework.properties file in the conf directory were not set correctly.
To solve the problem, change the permissions to give the account or group under which the web service for Netcool/OMNIbus runs read and execute permissions.
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.iwave.mapping.MappingGroovyAdapter#1' defined in class path resource [com/iwave/adapters/omnibus/event-adapter-synchronous.xml]: Initialization of bean failed; nested exception is org.springframework.beans.TypeMismatchException: Failed to convert property value of type [java.lang.String] to required type [java.lang.Class] for property 'toType'; nested exception is java.lang.IllegalArgumentException: Cannot find class [com.tivoli.tec.event_delivery.TECEvent]
This error indicates that the evd.jar file is missing.
To solve the problem, perform the steps in Copying the Required .jar Files and then attempt to start the web service for Netcool/OMNIbus.
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'omnibusAdapterConfig' defined in class path resource [com/iwave/adapters/omnibus/event-adapter-synchronous.xml]: Invocation of init method failed; nested exception is com.iwave.adapters.omnibus.InvalidAdapterConfigurationException: Error loading the JDBC driver (com.sybase.jdbc2.jdbc.SybDriver) specified in the properties file: com.sybase.jdbc2.jdbc.SybDriver
This error indicates that the jconn2.jar file is missing.
To solve the problem, perform the steps in Copying the Required .jar Files and then attempt to start the web service.
Note:
For any other startup errors, consult with Oracle Support.The web service for Netcool/OMNIbus requires version 1.6 of the JVM. If multiple versions of the JVM are installed on the system, it is possible that an older version of the JVM is being executed whenever the web service starts.
On Unix systems, the JAVA_HOME environment variable must be set to the directory where Java Development Kit (JDK) 1.6 is installed in the shell where the web service is started. To properly start the web service on a Unix platform:
Set the JAVA_HOME environment variable to the JDK 1.6 installation directory.
Navigate to the adapters/bin sub-directory in the web service installation directory.
Execute the ./service.sh start command.
On Windows systems, perform the following to ensure that JDK 1.6 is used when starting the web service:
Navigate to the adapters/bin sub-directory in the web service installation directory.
Run the iWaveAdaptersw.exe executable.
Click the Java tab.
Make sure that the Use Default check box is not checked.
In the Java Virtual system box, specify the path to the jvm.dll file in the JDK 1.6 installation directory.
Click OK.
Perform the following steps to identify and configure the connector to use the correct URL for web service for Netcool/OMNIbus operations:
Perform the steps in section Determining the URL Used by the Web Service to obtain the URL used by the web service.
Replace localhost in the URL with the hostname or IP address of the system where the web service is installed.
Log in to Oracle Enterprise Manager Cloud Control with an account that has Super Administrator permissions.
From the Setup menu, select Extensibility, then Management Connectors.
The General tab of the Configure Management Connector page appears.
Click on the name of the Netcool/OMNIbus connector.
Verify that the URL identified in step 2 is specified for the createEvent and updateEvent operations.
If any of the operations are incorrect, change to the correct URL and click OK.
Perform the following steps to identify the correct URL for web service for Netcool/OMNIbus operations:
Open a command terminal on the system where the web service is installed.
Change the working directory to the adapters/log directory in the web service installation directory.
Open the framework.log file in a text editor.
Go to the bottom of the file and search backwards for the string Setting the server's publish address to be. Continue searching backwards until you find the URL that contains EventService. The URL listed here is the URL that the web service uses to handle create and update operations.
Perform the following diagnostic steps if messages are not being generated or updated as expected in Netcool/OMNIbus:
Verify that the event that was triggered is referenced in a rule that forwards events to the IBM Tivoli Netcool/OMNIbus Connector.
Determine the error that Oracle Enterprise Manager has reported by examining the log file for errors. Perform the following steps to locate errors in the log file:
Open the emoms_pbs.trc file in a text editor. The file is located in the log directory at the following location:
<EM_INSTANCE_BASE>/em/<OMS_NAME>/sysman/log/
... where <EM_INSTANCE_BASE> is the OMS Instance Base directory. By default, the OMS Instance Base directory is gc_inst, which is present under the parent directory of the Oracle Middleware Home.
For example, if the Oracle Middleware Home is /u01/app/Oracle/Middleware, then the instance base directory is /u01/app/Oracle/Middleware/gc_inst, and the log and trace files are available in the /u01/app/Oracle/Middleware/gc_inst/em/EMGC_OMS1/sysman/log/ directory path.
Go to the bottom of the file and search backwards for Caused by.
Lines that start with Caused by contain error information. The error information appears after the text in the line that reads:
oracle.sysman.emSDK.webservices.outbound.WSInvokeException: caught WebServiceException:
Diagnose the problem based on the error information. See Resolving Errors from Oracle Enterprise Manager for information on troubleshooting common error events.
This section provides cause and solution information on troubleshooting common error messages. Find the error message in Table 7-1 that matches your error message, then refer to the corresponding section(s) indicated under Possible Cause for instructions on diagnosing and correcting the problem.
Table 7-1 Oracle Enterprise Manager Error Messages
| Error Message | Possible Cause |
|---|---|
javax.xml.soap.SOAPException: javax.xml.soap.SOAPException: Bad response: 403 Forbidden from url |
|
jjavax.xml.soap.SOAPException: javax.xml.soap.SOAPException: Message send failed: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target |
|
javax.xml.soap.SOAPException: javax.xml.soap.SOAPException: Message send failed: Connection refused |
|
javax.xml.soap.SOAPException: javax.xml.soap.SOAPException: Message send failed: No route to host |
|
javax.xml.soap.SOAPException: javax.xml.soap.SOAPException: Bad response: 404 Not Found from url … |
|
javax.xml.soap.SOAPException: javax.xml.soap.SOAPException: Message send failed: Connection timed out |
|
javax.xml.soap.SOAPException: javax.xml.soap.SOAPException: Message send failed: <hostname> |
|
javax.xml.transform.TransformerConfigurationException: Could not compile stylesheet |
|
com/ibm/logging/ILogger |
|
Timeout occurred waiting for synchronous response from OMNIbus after inserting an event. |
|
Attempt to insert the event into the master probe failed. |
The user name or password for accessing the web service for Netcool/OMNIbus is incorrect.
Log in to Oracle Enterprise Manager Cloud Control with an account that has Super Administrator privileges.
From the Setup menu, select Extensibility, then Management Connectors.
The Management Connectors page appears.
Click the name of the Netcool/OMNIbus Connector.
This action invokes Edit mode, enabling you to configure the connector.
Correct the Netcool/OMNIbus Web Service Username and Netcool/OMNIbus Web Service Password fields and click OK.
If the credentials supplied appear to have been entered correctly but still do not work, a typographical error might have been made during the setup of the web service. The recommended option is to reset the web service credentials and attempt this again. Refer to Changing Web Service Credentials in Appendix A for the steps required to reset the credentials.
The SSL handshake between the Oracle Enterprise Manager Connector Framework and the web service for Netcool/OMNIbus has failed. This failure occurs because Oracle Enterprise Manager is not configured correctly with the SSL certificate for the web service. The SSL certificate that the web service uses must be imported into the Enterprise Manager key store. The certificate is either missing from the key store or does not match the SSL certificate provided by the web service for Netcool/OMNIbus.
Import the SSL certificate from the web service into the Enterprise Manager key store. See Configure Oracle Enterprise Manager to Use SSL for details on setting up Oracle Enterprise Manager with the Netcool/OMNIbus SSL certificate.
The web service for Netcool/OMNIbus is down.
Perform the following steps to check the status of the web service and start it if necessary.
If the web service is installed on a Unix system:
Open a command terminal on the system where the web service is installed.
Change the working directory to the adapters/bin directory in the web service installation directory.
Enter the following command:
./service.sh status
If the command indicates that the service is not running, restart the web service as specified in Running the Web Service on Unix.
If the web service is installed on a Windows system:
Open a command terminal on the system where the web service is installed.
Change the working directory to the adapters\log directory in the web service installation directory.
Open the framework.log file in a text editor.
Go to the bottom of the file and search backwards for the string iWave Adapter Framework. If the last occurrence found is iWave Adapter Framework Started, this indicates that the web service is started.
If the web service is not started, restart the web service as specified in Running the Web Service on Windows.
The IP address specified in the URL is invalid, or the network is down.
Verify that the hostname/IP address configured for the connector is correct:
Log in to Oracle Enterprise Manager Cloud Control with an account that has Super Administrator privileges.
From the Setup menu, select Extensibility, then Management Connectors.
The Management Connectors page appears, which shows the installed connectors.
Click on the name of the Netcool/OMNIbus Connector. This invokes edit mode, enabling you to configure the connector.
Verify that the hostname/IP address specified in the URL for the createEvent and updateEvent operations are correct.
If the hostname/IP address is incorrect, provide the correct values and click OK.
If the URLs specify a host name, make sure that the host name resolves to the correct IP address. To determine the IP address of the host name, issue the ping <hostname> command, where <hostname> is the actual host name. This lists the IP address that was resolved for the host name. If this is incorrect, the system administrator needs to investigate why it is incorrect.
If the hostname/IP address appears to be correct, try to ping the system where the web service is installed using the hostname/IP address. If the ping fails, the system administrator needs to investigate why there is no connectivity.
The web service received the request and rejected it because an invalid path was specified in the URL.
Perform the following steps to test the URL the connector is using:
Log in to Oracle Enterprise Manager Cloud Control with an account that has Super Administrator privileges.
From the Setup menu, select Extensibility, then Management Connectors.
The Management Connectors page appears, which shows the installed connectors.
Click on the name of the Netcool/OMNIbus Connector. This invokes edit mode, enabling you to configure the connector.
Select and copy the URL specified for the createEvent operation.
Open an internet browser on the system where the Oracle Enterprise Manager server is installed.
In the address window, enter the URL that was copied in step 6 above. Add ?wsdl to the end of the URL. The URL should appear similar to the following example:
http://[Hostname]:8080/services/hpovou/EventService?wsdl
[Hostname] is the actual host name or IP address where the web service is installed.
If the WSDL is loaded, this confirms that the URL is correct. If it fails to load, there is a problem with the URL. Perform the steps specified in Using the Correct URL for Web Service Operations to configure the connector to use the correct URL.
A firewall is blocking access to the system where the web service for Netcool/OMNIbus is installed.
Contact your IT department to give Enterprise Manager access to the port the web service uses. Perform the steps specified in Using the Correct URL for Web Service Operations to determine the URL used by the Web Service. The port number specified in the URL is the port number the IT department should open in the firewall.
The port number specified in the URL is invalid.
Verify that the hostname/IP address configured for the connector is correct:
Log in to Oracle Enterprise Manager Cloud Control with an account that has Super Administrator privileges.
From the Setup menu, select Extensibility, then Management Connectors.
The Management Connectors page appears, which shows the installed connectors.
Click on the name of the Netcool/OMNIbus Connector. This invokes edit mode, enabling you to configure the connector.
Verify that the port number specified in the URL for the createEvent and updateEvent operations are correct.
If the web service is configured to use SSL, the port number should be 8443. Otherwise, the port number should be 8080 unless the port number has been reconfigured as specified in Changing Default Port Numbers in Appendix A.
If the port number is incorrect, provide the correct value and click OK.
The system does not recognize the host name specified in the URL.
You have the following options for addressing this issue:
Coordinate with the system administrator to change the system configuration to recognize the host name.
Change the host name if it is listed incorrectly.
Specify the IP address in the URL instead of the host name. To do this, perform the following steps:
Determine the IP address of the system where the web service is installed.
Log in to Oracle Enterprise Manager Cloud Control with an account that has Super Administrator privileges.
From the Setup menu, select Extensibility, then Management Connectors.
The Management Connectors page appears, which shows the installed connectors.
Click on the name of the Netcool/OMNIbus Connector. This invokes edit mode, enabling you to configure the connector.
Change the host name to the IP address in the URL specified for the createEvent and updateEvent operations.
Click OK.
Make sure that the host name resolves to the correct IP address. To determine the IP address of the host name, issue the ping <hostname> command, where <hostname> is the actual host name. This lists the IP address that was resolved for the host name. If this is incorrect, the system administrator needs to investigate why it is incorrect.
If the hostname/IP address appears to be correct, try to ping the system where the web service is installed using the hostname/IP address. If the ping fails, the system administrator needs to investigate why there is no connectivity.
The connector framework could not process the request because the XSL file was formatted incorrectly. This problem should not occur unless the connector has been customized.
Examine any changes made to the XSL template files for mistakes that could have caused the problem. If you cannot find the problem manually, load the XSL in a utility that performs XML validation.
The error indicates that the web service cannot locate the log.jar file.
To solve the problem, perform the steps in Copying the Required .jar Files and then restart the web service for Netcool/OMNIbus.
The web service received the request and successfully sent the request to the TEC EIF probe. The web service timed out waiting for Netcool/OMNIbus to send a notification that the requested alert that was created/updated. This error occurs because Netcool/OMNIbus is not properly configured to send the response to the web service.
Listed below is a summary of the sequence of events that occur whenever an event is created/updated in Netcool/OMNIbus.
Enterprise Manager makes a web service call for a create/update operation.
The web service makes an API call to the EIF probe to insert an event.
The EIF Probe inserts a row in the custom.oracle_status table for the event.
The oracle_insert_alert trigger fires and creates/updates the appropriate alert in the alerts.status, alerts.details, and alerts.journal database tables.
The oracle_alert_created, oracle_alert_reinserted, or oracle_alert_updated trigger fires calling the oracle_send external procedure.
The oracle_send external procedure calls the oracle_send.cmd (Windows) or oracle_send.sh (Unix) script to send a notification to the web service.
The oracle_send.cmd/oracle_send.sh script calls the universal agent utility (uniagt.exe) to send the notification to the web service.
Upon receipt of the notification, the web service makes a JDBC call to retrieve field information for the alert.
The web service sends a response to Enterprise Manager with the alert information.
A failure in steps 3 through 8 results in a timeout being reported by the web service for Netcool/OMNIbus.
Perform the following steps to identify the location of the failure:
Verify that Netcool/OMNIbus has been customized as specified in Modifying Netcool/OMNIbus and that the Object Server and EIF Probe are operational. Correct any issues that are found and retry.
Open the Netcool/OMNIbus console and determine whether an alert was created/updated.
If the alert was NOT created/updated, the failure occurred in Cause step 3 or 4. Perform the following steps to determine the reason for the failure:
Check the EIF Probe log file for errors. If there are errors, you will need to determine the cause of the error and fix the problem. Most likely the error will be caused by a lack of connectivity between the probe system and the object server system.
If there are no errors in the EIF Probe log file, verify that a row was added to the custom.oracle_status table for the event. If a row was not added, there must be a problem with the configuration of the EIF probe. Enable debug in the EIF Probe and run another test. Look in the EIF Probe log file to see if there is any information about what the probe did with the event.
If a row was added to the custom.oracle_status table, examine the ErrorMessage column for any error information.
Examine the object server log file to verify that the oracle_insert_alert trigger fired. If the trigger did not fire, open the definition for the trigger and verify it is enabled.
If the trigger fired, look for any errors that occurred while running the trigger.
If the alert was created/updated, the failure occurred somewhere in Cause steps 5 through 8. Perform the following steps to determine the reason for the failure:
Examine the object server log file to verify that one of the oracle_alert_created, oracle_alert_reinserted, or oracle_alert_updated triggers fired. If none of the triggers fired, open the definition for the triggers and look for anything that might prevent the trigger from firing. One possibility is that the trigger has been disabled.
If one of the triggers fired, look for any errors that occurred while running the trigger. If there are no errors in the object server log, you will need to enable debug in the script to see if it is being called. Step c below gives instructions for enabling debug in a Windows environment and step d gives instructions for enabling debug in a Unix environment.
For Windows environments, perform the following steps to enable debug in the notification script.
Open the oracle_send.cmd script with a text editor.
Uncomment the following lines by removing the REM at the beginning of the line:
echo %DATE% %TIME% %* >> "%AGENT_DIR%\oracle_send.out" "%AGENT_DIR%\bin\uniagt" -r %OMNIBUS_WS_URL% -v 1 -O "%AGENT_DIR%\agent%1.log"
Comment the following line by prepending REM to the beginning of the line:
"%AGENT_DIR%\bin\uniagt" -r %OMNIBUS_WS_URL%
Save the file and exit.
For Unix environments, perform the following steps to enable debug in the notification script:
Open the oracle_send.sh script with a text editor.
Uncomment the following lines by removing the # at the beginning of the line:
echo `date` $* >> ${AGENT_DIR}/oracle_send.out
${AGENT_DIR}/bin/uniagt -r ${OMNIBUS_WS_URL} -v 1 -O ${AGENT_DIR}/agent/{1}.log
Comment the following line by prepending # to the beginning of the line:
${AGENT_DIR}/bin/uniagt -r ${OMNIBUS_WS_URL}
Save the file and exit.
After enabling debug, run another test and determine whether the oracle_send.out log file was created. If the file was created, open it with a text editor and look at the last line in the file to get the serial number. The line starts with the date/time the script was called followed by the serial number and the name of the trigger that fired.
Verify that an agent log was created for the serial number. The agent log should be named agent<serialno>.log where <serialno> is the serial number from the oracle_send.out log file.
If the file exists, open it with a text editor and look for errors sending to the web service. If errors occur, it will likely be caused by a connectivity issue or an invalid URL specified in the OMNIBUS_WS_URL environment variable. Check the hostname/IP address and port number in the URL. Unless specifically changed, the port number should be 8080. If the notification was sent successfully, you will see something similar to the following in the log.
2012/06/13 11:20:52 response headers: <HTTP/1.1 200 OK Content-Type: text/xml Content-Length: 0 Server: Jetty(6.1.14)
If the agent log indicates the notification was successfully sent to the web service, look in the web service log for errors. The web service log is located in the adapters/log directory of the web service installation directory and is named framework.log. Most likely the error will be caused by a failure to connect to the database to retrieve the alert information.
The web service for Netcool/OMNIbus cannot connect to the TEC EIF probe and send alert information. Listed below are some of the possible issues that could cause a connection failure.
The probe is down.
The hostname/IP address or the port number for the probe is configured incorrectly in the web service.
A firewall is preventing the connection from completing.
Perform the following steps to resolve the issue:
Verify that the TEC EIF probe is up and operational.
Verify that the web service is configured to use the correct hostname/IP address and port number.
To verify:
Open the framework.properties file using a text editor. The file is located in the adapters/conf directory of the web service installation directory.
Verify that the omnibus.probe.master.t_ServerLocation property is set to the hostname or IP address where the master EIF probe is installed.
Verify that the omnibus.probe.master.t_Port property is set to the port number on which the master probe is listening.
If configured to send to a slave probe, verify that the omnibus.probe.slave.t_ServerLocation property is set to the hostname or IP address where the slave EIF probe is installed.
If configured to send to a slave probe, verify that the omnibus.probe.slave.t_Port property is set to the port number on which the slave probe is listening.
Make any required changes and save the file.
If any changes were made, restart the web service as specified in the appropriate section ( Running the Web Service on Unix or Running the Web Service on Windows).
If the problem still is not resolved, contact your IT department to check for a firewall that might be preventing the web service from calling the EIF probe.