This chapter provides information to assist in troubleshooting integration issues with IBM TEC. The chapter focuses on troubleshooting issues in the web service front-end and the back-end Agent.
This chapter discusses the following topics:
Before you start the troubleshooting steps, you must insure that you have done the following:
Install the TEC Connector as specified in Section 2.3, "Installing the Connector in Enterprise Manager".
Install, start, and test the TEC Web service as specified in Section 2.4, "Installing the IBM TEC Web Service".
Install and start the Oracle TEC Agent as specified in Section 2.5, "Installing and Running the IBM TEC Agent".
Create a connector instance as specified in Section 3.1, "Creating a Connector Instance".
Configure the connector instance as specified in Section 3.2, "Configuring the Connector".
Set up one or more rules to forward events to the connector instance.
If all the actions above have been completed and the connector is not working, perform the steps in Section 7.1, "Diagnosing the Problem".
To diagnose why the connector is not working, follow these steps:
Verify that the TEC Web Service has been successfully started and the WSDL for the TEC Web Service can be accessed from the machine where it is installed. Performing the following steps to do this:
Perform the steps in Section 7.4, "Using the Correct URL for TEC Web Service Operations" to determine the URL used by the TEC Web Service.
Open a browser and paste the URL from the previous step in the address window. Append ?WSDL to the end of the URL and attempt to load the URL.
The WSDL should be loaded if the web service is operational.
Note:
A WSDL is an XML file that describes the web service.If the WSDL cannot be loaded, this indicates the web service had startup issues. See Section 7.2, "Troubleshooting Web Service Startup Errors" to diagnose the problem.
Verify that the WSDL for the TEC Web Service can be accessed from the machine where the Enterprise Manager server is installed.
Open a browser at the Enterprise Manager server and copy the URL from Step 1-b above to the address window. The hostname for the URL will be localhost. Change localhost to the actual hostname or IP address of the machine where the TEC Web Service is installed. If you specify a hostname, you must make sure that the hostname is recognized at the Enterprise Manager server machine. This can be done using the ping command.
For example, if the Web Service is installed on the server with a hostname of TECServer01 and the URL listed in framework.log is:
http://localhost:8080/services/tec/EventService
The URL used at the Enterprise Manager server machine would be:
http://TECServer01: 8080/services/tec/EventService
Attempt to load the WSDL by appending ?WSDL at the end of the URL. If the WSDL cannot be loaded, either the hostname is not recognized at the Enterprise Manager machine or there is a connectivity issue between the two machines. If you specified a hostname, try using the IP address instead of the hostname in the URL. If it still will not load, you have a connectivity problem. You will need to consult with your IT department to resolve this issue.
Verify that the TEC Connector specifies the correct URL for the createEvent and updateEvent operations.
Log in to the Oracle Enterprise Manager console with an account that has Super Administrator permissions.
From the Enterprise Manager console, click Setup.
Select Extensibility in the pane.
Select Management Connectors in the left pane.
The Management Connectors page appears, which shows the installed connectors.
Click the Configure icon associated with the TEC Connector. This invokes edit mode, enabling you to configure the connector.
Verify that the URL identified in step 2-b is specified for both operations (createEvent and updateEvent). If any of the operations are incorrect, change to the correct URL.
Note:
Do not append the WSDL to the end of the URL that is specified here.Click OK.
If there were no errors found in the previous steps, this rules out connectivity issues between the connector and the TEC Web Service. The problem must lie in the TEC Web Service or the Oracle TEC Agent. See Section 7.5, "Troubleshooting Web Service Operations Errors" for information on diagnosing errors.
To identify the cause of a startup failure, navigate to the adapters/log directory in the TEC Web Service install directory and open the framework.log file in a text editor. Search for Exception to find any errors in the file. If the file does not exist, it indicates that there is a problem locating or executing the JVM. See Section 7.3, "JVM Errors" for information about resolving JVM issues.
Listed below are some possible Exceptions, an explanation of the root cause, and a description of the solution.
java.net.BindException: Address already in use: bind
This error indicates that the web service could not start because of a port conflict. There are two possible causes for this error.
Another application is using a port that the Web service is configured to use. If the web service is configured to use SSL, the port number is 8443. If it is not configured to use SSL, the port number is 8080.
There are two possible solutions to this. You can change the other application to use a different port or you can change the TEC Web Service to use a different port. To change the TEC Web Service to use a different port, see Section A.2, "Changing Default Port Numbers".
There is an instance of the Web service already running. If this is the case then there is no change required. You should only run one instance of the Web service at a time.
org.springframework.beans.factory.BeanInitializationException: Could not load properties; nested exception is java.io.FileNotFoundException: … framework.properties (Permission denied)
This error indicates that the web service could not start because the permissions on the framework.properties file in the conf directory were not set correctly.
To solve the problem, change the permissions to give the account or group under which the TEC Web Service runs read and execute permissions.
For any other startup errors, consult with Oracle Support.
The TEC Web Service requires version 1.6 of the JVM. If there are multiple versions of the JVM installed on the machine, it is possible that an older version of the JVM is being executed whenever the web service starts.
On UNIX systems, the JAVA_HOME environment variable must be set to the directory where Java 1.6 is installed in the shell where the web service is started. To properly start the web service on a UNIX platform, perform the following:
Set the JAVA_HOME environment variable to the Java 1.6 install directory.
Navigate to the adapters/bin subdirectory in the Web service install directory.
Execute the ./service.sh start command.
On Windows systems, perform the following to insure that Java 1.6 is used when starting the web service.
Navigate to the adapters/bin subdirectory in the Web service install directory.
Run the iWaveAdaptersw.exe executable.
Click the Java tab.
Make sure that the Use Default check box is not checked.
In the Java Virtual Machine box, specify the path to the jvm.dll file in the Java 1.6 install directory.
Click OK.
Perform the following steps to identify and configure the connector to use the correct URL for TEC Web Service operations.
Open a command terminal on the system where the TEC web service is installed.
Change the working directory to the adapters/log directory in the TEC web service installation directory.
Open the framework.log file in a text editor.
Go to the bottom of the file and search backwards for the string ”Setting the server's publish address to be." Continue searching backwards until the URL that contains EventService is found. The URL listed here is the URL that should be specified for the createEvent and updateEvent operations.
Log in to the Oracle Enterprise Manager console with an account that has Super Administrator permissions.
From the Enterprise Manager console, click Setup.
Select Extensibility in the pane.
Select Management Connectors in the left pane.
Click the name of the TEC Connector.
The General tab of the Configure Management Connector page appears.
Verify that the URL identified in step 4 is specified for the createEvent and updateEvent operations.
If any of the operations are incorrect, change to the correct URL, then click OK.
Perform the following diagnostic steps if events are not being generated or updated as expected in TEC.
Verify that the event that was triggered is referenced in a rule that forwards events to the TEC connector.
Determine the error that Oracle Enterprise Manager has reported.
To do this you need to examine the log file for errors. Perform the following steps to locate errors in the log file:
1.) Open the emoms_pbs.trc file in a text editor. The file is located in the log directory at the following location:
<EM_INSTANCE_BASE>/em/<OMS_NAME>/sysman/log/
Where, <EM_INSTANCE_BASE> is the OMS Instance Base directory. By default, the OMS Instance Base directory is gc_inst, which is present under the parent directory of the Oracle Middleware Home.
For example, if the Oracle Middleware Home is /u01/app/Oracle/Middleware, then the instance base directory is /u01/app/Oracle/Middleware/gc_inst, and the log and trace files are available in /u01/app/Oracle/Middleware/gc_inst/em/EMGC_OMS1/sysman/log/ directory path.
2.) Go to the bottom of the file and search backwards for "Caused by". Lines that start with "Caused by" contain error information. The error information comes after the text in the line that reads "oracle.sysman.emSDK.webservices.outbound.WSInvokeException: caught WebServiceException:".
Diagnose the problem based on the error information. See Section 7.6, "Troubleshooting Errors from Oracle Enterprise Manager" for information on troubleshooting common error events.
This section provides cause and solution information on troubleshooting common error messages. Find the error message in Table 7-1 that matches your error message, then refer to the corresponding section(s) indicated under Possible Cause for instructions to diagnose and correct the problem.
Table 7-1 Enterprise Manager Error Messages
| Error Message | Possible Cause |
|---|---|
|
javax.xml.soap.SOAPException: javax.xml.soap.SOAPException: Bad response: 403 Forbidden from url … |
Invalid Web Service Credentials |
|
javax.xml.soap.SOAPException: javax.xml.soap.SOAPException: Message send failed: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target |
SSL Not Configured in Enterprise Manager |
|
javax.xml.soap.SOAPException: javax.xml.soap.SOAPException: Message send failed: Connection refused |
TEC Web Service Down |
|
javax.xml.soap.SOAPException: javax.xml.soap.SOAPException: Message send failed: No route to host |
Invalid IP Address |
|
javax.xml.soap.SOAPException: javax.xml.soap.SOAPException: Bad response: 404 Not Found from url … |
Invalid Port NumberorInvalid URL Path |
|
javax.xml.soap.SOAPException: javax.xml.soap.SOAPException: Message send failed: Connection timed out |
Firewall Blocking Access |
|
javax.xml.soap.SOAPException: javax.xml.soap.SOAPException: Message send failed: hostname; |
Unknown Host |
|
javax.xml.transform.TransformerConfigurationException: Could not compile stylesheet |
Invalid XML Format |
|
Attempt to insert the event into TEC failed |
TEC Server Not Operational |
|
Timeout occurred waiting for synchronous response from TEC after inserting an event |
TEC Server Timeout |
Invalid Web Service Credentials
Cause
The user name or password for accessing the TEC web service is incorrect.
Solution
Log in to the Oracle Enterprise Manager console with an account that has Super Administrator privileges.
From the Enterprise Manager console, click Setup.
Select Extensibility in the pane.
Select Management Connectors from the menu. This displays the Management Connectors page.
Click the name of the TEC Connector.
This invokes edit mode, enabling you to configure the connector.
Correct the TEC Web Service Username and TEC Web Service Password fields, then click OK.
SSL Not Configured in Enterprise Manager
Cause
The SSL handshake between the Oracle Enterprise Manager Connector Framework and the TEC web service failed. This failure occurs because Oracle Enterprise Manager is not configured correctly with the SSL certificate for the TEC web service. The SSL certificate the TEC web service uses must be imported into the certificate store. The certificate is either missing from the certificate store or does not match the SSL certificate provided by the TEC web service.
Solution
Import the SSL certificate from the TEC web service into the certificate store. See Section 6.1, "Configuring Enterprise Manager to use SSL" for details on setting up Oracle Enterprise Manager with the TEC SSL certificate.
Cause
The TEC web service is down.
Solution
Perform the following steps to check the status of the web service and start it if necessary.
If the TEC web service is installed on a Unix system:
Open a command terminal on the system where the TEC web service is installed.
Change the working directory to the adapters/bin directory in the TEC web service installation directory.
Enter the following command:
./service.sh status
If the command indicates that the service is not running, enter the following command:
./service.sh start
If the TEC web service is installed on a Windows system:
Open a command terminal on the system where the TEC web service is installed.
Change the working directory to the adapters/log directory in the TEC web service installation directory.
Open the framework.log file in a text editor.
Go to the bottom of the file and search backwards for the string iWave Adapter Framework. If the last occurrence found is iWave Adapter Framework Started, this indicates that the web service is started.
If the web service is not started, start the web service as specified in Section 2.4.1.3, "Running the Web Service on Unix" or Section 2.4.2.4, "Running the Web Service on Windows".
Cause
The IP address specified in the URL is invalid or the network is down.
Solution
Verify that the hostname/IP address configured for the connector is correct:
Log in to the Oracle Enterprise Manager console with an account that has Super Administrator privileges.
From the Enterprise Manager console, click Setup.
Select Extensibility in the pane.
Select Management Connectors from the menu. This displays the Management Connectors page.
Click the name of the TEC Connector.
This invokes edit mode, enabling you to configure the connector.
Verify that the hostname/IP address specified in the URL for the createEvent and updateEvent operations are correct.
If the hostname/IP address is incorrect, provide the correct value, then click OK.
If the URLs specify a host name, make sure that the host name resolves to the correct IP address. To determine the IP address of the host name, issue the ping <hostname> command, where <hostname> is the actual host name. This lists the IP address that was resolved for the host name. If this is incorrect, the system administrator needs to investigate why it is incorrect.
If the hostname/IP address appears to be correct, try to ping the system where the TEC web service is installed using the hostname/IP address. If the ping fails, the system administrator needs to investigate why there is no connectivity.
Cause
The port number specified in the URL is invalid.
Solution
Verify that the port number configured for the connector is correct:
Log in to the Oracle Enterprise Manager console with an account that has Super Administrator privileges.
From the Enterprise Manager console, click Setup.
Select Extensibility in the pane.
Select Management Connectors from the menu. This displays the Management Connectors page
Click the name of the TEC Connector.
This invokes edit mode, enabling you to configure the connector.
Verify that the port number specified in the URL for the createEvent and updateEvent operations are correct.
If the port number is incorrect, provide the correct value, then click OK.
Cause
The web service received the request and rejected it because an invalid path was specified in the URL.
Solution
Perform the following steps to test the URL the connector is using.
Log in to the Oracle Enterprise Manager console with an account that has Super Administrator privileges.
From the Enterprise Manager console, click Setup.
Select Extensibility in the pane.
Select Management Connectors from the menu. This displays the Management Connectors page.
Click the name of the TEC Connector.
This invokes edit mode, enabling you to configure the connector.
Select and copy the URL specified for the createEvent operation.
Open an internet browser on the system where the Oracle Enterprise Manager server is installed.
In the address window, enter the URL that was copied in step 6 above. Add ?wsdl to the end of the URL. The URL should appear similar to the following example:
http://[Hostname]:8080/services/tec/EventService?wsdl
[Hostname] is the actual host name or IP address where the TEC web service is installed.
If the WSDL is loaded, this confirms that the URL is correct. If it fails to load, there is a problem with the URL. Perform the steps specified in Section 7.4, "Using the Correct URL for TEC Web Service Operations" to configure the connector to use the correct URL.
Cause
A firewall is blocking access to the system where the TEC Web Service is installed.
Solution
Contact your IT department to give Enterprise Manager access to the port used by the TEC Web Service. Perform the steps specified in Section 7.4, "Using the Correct URL for TEC Web Service Operations" to determine the URL used by the TEC Web Service. The port number specified in the URL is the port number the IT department should open in the firewall.
Cause
The system does not recognize the host name specified in the URL.
Solution
You can use the following options to address this issue:
Coordinate with the system administrator to change the system configuration to recognize the host name.
Specify the IP address in the URL instead of the host name. To do this, perform the following steps:
Determine the IP address of the system where the TEC web service is installed.
Log in to the Oracle Enterprise Manager console with an account that has Super Administrator privileges.
From the Enterprise Manager console, click Setup.
Select Extensibility in the pane.
Select Management Connectors from the menu. This displays the Management Connectors page.
Click the name of the TEC Connector.
This invokes edit mode, enabling you to configure the connector.
Change the host name to the IP address in the URL specified for the createEvent and updateEvent operations.
Click OK.
Cause
The connector framework could not process the request because the XSL file was formatted incorrectly. This problem should not occur unless the connector templates have been customized.
Solution
Examine any changes made to the XSL template files for mistakes that could have caused the problem. If you can't find the problem manually, load the XSL in a utility that performs XML validation.
Cause
The web service could not insert the event into TEC because it could not connect to the TEC server. This problem could be caused for one of the following reasons:
The TEC server is down.
The TEC server hostname/IP address or port number is not configured correctly for the TEC web service.
A network outage is preventing the TEC web service from connecting to the TEC server.
Solution
Perform the following steps to determine and correct the root cause of the problem:
Open a command terminal on the system where the TEC web service is installed.
Change the working directory to the adapters/conf directory in the TEC web service installation directory.
Edit the framework.properties file with a text editor.
Search for the tec.server1.config.t_ServerLocation parameter. This parameter specifies the hostname/IP address to use when connecting to the TEC API server.
Verify that the host name or IP address is correct. If the hostname/IP address appears to be correct, try to ping the system where the TEC web service is installed using the hostname/IP address. If the ping fails, the system administrator needs to investigate why there is no connectivity.
Search for the tec.server1.config.t_Port parameter. This parameter specifies the port number to use when connecting to the TEC API server.
Verify that the port number is correct. If the TEC server is running on a Unix machine with a port mapper, the port number should be set to 0. If the TEC server is running on a Windows system or Unix system without a port mapper, the port number must be set to the TEC reception port.
Note:
The TEC reception port is defined by the tec_recv_agent_port parameter in the .tec_config file in the $BINDIR/TME/TEC directory.If the hostname/IP address is valid and reachable, and the port number is correct, the TEC server must be down.
To check the status of the TEC server, open a command terminal on the system where the TEC server is installed.
Set up the TEC environment.
For Windows systems, enter the following commands to start the bash shell:
%SystemRoot%\system32\drivers\etc\Tivoli\setup_env.cmd
bash
For Unix systems, enter the following command to set the TEC environment variables:
/etc/Tivoli/setup_env/sh
Enter the following command to display the status of the TEC server:
wstatesvr
If the TEC server is down, contact your TEC system administrator to start the server.
Cause
The web service successfully sent the event to the TEC server, but did not receive a response in the time-out period. This problem could be caused for one of the following reasons:
The TEC server ignored the request because it contained invalid data.
The TEC rule for the class is incorrectly configured.
Solution
Perform the following steps to determine and correct the root cause of the problem.
Open a command terminal on the system where the TEC server is installed.
Set up the TEC environment.
For Windows systems, enter the following commands to start the bash shell:
%SystemRoot%\system32\drivers\etc\Tivoli\setup_env.cmd
bash
For Unix systems, enter the following command to set the TEC environment variables:
/etc/Tivoli/setup_env.sh
Enter the following command to display the most recent entries in the TEC server log:
wtdumprl –o DESC | more
Locate the entry for the event that timed out, and verify that the log indicates that it was processed. If an error appears, it explains what is incorrect with the data. You will need to modify the mapping to fix the problem. See Section 5.2.3, "Changing a Mapping" for additional information.
If the log indicates that TEC successfully processed the data, you need to verify that the TEC rule base invoked the tec2uniagt script located in the TEC Agent installation directory. Consult with your TEC administrator for information on setting up and viewing the rule logs.