This chapter provides information to assist in troubleshooting integration issues with Microsoft SCOM. The chapter focuses on troubleshooting issues in the web service front-end and the back-end Agent.
Note:
Unless otherwise noted, these instructions apply to the SCOM 2012 connector and to the SCOM 2007 connectors. Instructions specific to the SCOM 2007 connectors are available in Appendix E, "SCOM 2007 Connector".This chapter discusses the following topics:
Before you start the troubleshooting steps, you must insure that you have done the following:
Install the SCOM connector as specified in Installing the Connector in Enterprise Manager.
Install and start the Oracle SCOM Agent as specified in Installing and Running the Oracle SCOM Agent.
Install, start, and test the SCOM Web service as specified in Installing the Microsoft SCOM Web Service.
Create a connector instance as specified in Creating a Connector Instance.
Configure the connector instance as specified in Configuring the Connector.
Set up one or more rules to forward events to the connector instance.
If all the actions above have been completed and the connector is not working, perform the steps in Diagnosing the Problem.
Follow these steps to diagnose a problem.
Verify that the SCOM Web Service has been successfully started and the WSDL for the SCOM Web Service can be accessed from the machine where it is installed. Performing the following steps to do this:
Perform the steps in Using the Correct URL for SCOM Web Service Operations to determine the URL used by the SCOM Web Service.
Open a browser and paste the URL from the previous step in the address window. Append ?WSDL to the end of the URL and attempt to load the URL.
The WSDL should be loaded if the web service is operational.
Note:
A WSDL is an XML file that describes the web service.If the WSDL cannot be loaded, this indicates the web service had startup issues. See Troubleshooting Web Service Startup Errors to diagnose the problem.
Verify that the WSDL for the SCOM Web Service can be accessed from the machine where the Enterprise Manager server is installed.
Open a browser at the Enterprise Manager server and copy the URL from step 1-b above to the address window. The hostname for the URL will be localhost. Change localhost to the actual hostname or IP address of the machine where the SCOM Web Service is installed. If you specify a hostname, you must make sure that the hostname is recognized at the Enterprise Manager server machine. This can be done using the ping command.
For example, if the Web Service is installed on the server with a hostname of SCOMServer01 and the URL listed in framework.log is:
http://localhost:8080/services/SCOM/EventService
The URL used at the Enterprise Manager server machine would be:
http://SCOMServer01:8080/services/SCOM/EventService
Attempt to load the WSDL by appending ?WSDL at the end of the URL. If the WSDL cannot be loaded, either the hostname is not recognized at the Enterprise Manager machine or there is a connectivity issue between the two machines. If you specified a hostname, try using the IP address instead of the hostname in the URL. If it still will not load, you have a connectivity problem. You will need to consult with your IT department to resolve this issue.
Verify that the SCOM connector specifies the correct URL for the createEvent and updateEvent operations.
Log in to the Oracle Enterprise Manager console with an account that has Super Administrator permissions.
From the Enterprise Manager console, click Setup.
Select Extensibility in the pane.
Select Management Connectors in the left pane.
The Management Connectors page appears, which shows the installed connectors.
Click the Configure icon associated with the SCOM connector. This invokes edit mode, enabling you to configure the connector.
Verify that the URL identified in step 2-b is specified for the createEvent and updateEvent operations. Also verify the URLs listed for the setup, initialize, and uninitialize operations. The URL is basically the same as the URL listed above except it is for the SCOMService instead of the EventService. Listed below is the URL that would be used based on the previous example.
http://SCOMServer01:8080/services/SCOM/SCOMService
If any of the operations are incorrect, change to the correct URL.
Note:
Do not append the WSDL to the end of the URL that is specified here.Click OK.
If there were no errors found in the previous steps, this rules out connectivity issues between the connector and the SCOM Web Service. The problem must lie in the SCOM Web Service or the Oracle SCOM Agent. See Troubleshooting Web Service Operations Errors for information on diagnosing errors.
To identify the cause of a startup failure, navigate to the adapters/log directory in the SCOM Web Service install directory and open the framework.log file in a text editor. Search for Exception to find any errors in the file. If the file does not exist, it indicates that there is a problem locating or executing the JVM. See JVM Errors for information about resolving JVM issues.
Listed below are some possible Exceptions, an explanation of the root cause, and a description of the solution.
java.net.BindException: Address already in use: bind
This error indicates that the web service could not start because of a port conflict. There are two possible causes for this error.
Another application is using a port that the Web service is configured to use. If the web service is configured to use SSL, the port number is 8443. If it is not configured to use SSL, the port number is 8080.
There are two possible solutions to this. You can change the other application to use a different port or you can change the SCOM Web Service to use a different port. To change the SCOM Web Service to use a different port, see Changing Default Port Numbers in Appendix A.
There is an instance of the Web service already running. If this is the case then there is no change required. You should only run one instance of the Web service at a time.
org.springframework.beans.factory.BeanInitializationException: Could not load properties; nested exception is java.io.FileNotFoundException: … framework.properties (Permission denied)
This error indicates that the web service could not start because the permissions on the framework.properties file in the conf directory were not set correctly.
To solve the problem, change the permissions to give the account or group under which the SCOM Web Service runs read and execute permissions.
For any other startup errors, consult Oracle Support.
The SCOM Web Service requires version 1.6 of the JVM. If there are multiple versions of the JVM installed on the machine, it is possible that an older version of the JVM is being executed whenever the web service starts.
On UNIX systems, the JAVA_HOME environment variable must be set to the directory where JDK 1.6 is installed in the shell where the web service is started. To properly start the web service on a UNIX platform, perform the following:
Set the JAVA_HOME environment variable to the JDK 1.6 install directory.
Navigate to the adapters/bin subdirectory in the Web service install directory.
Execute the ./service.sh start command.
On Windows systems, perform the following to insure that JDK 1.6 is used when starting the web service.
Navigate to the adapters/bin subdirectory in the Web service install directory.
Run the iWaveAdaptersw.exe executable.
Click the Java tab.
Be sure the Use Default check box is not checked.
In the Java Virtual Machine box, specify the path to the jvm.dll file in the JDK 1.6 install directory.
Click OK.
Perform the following steps to identify and configure the connector to use the correct URL for SCOM Web Service operations.
Open a command terminal on the system where the SCOM web service is installed.
Change the working directory to the adapters/log directory in the SCOM web service installation directory.
Open the framework.log file in a text editor.
Go to the bottom of the file and search backwards for the string Setting the server's publish address to be. Continue searching backwards until you find the URL that contains EventService.
Note:
If you cannot find the URL information in the log file, it means the log file has rolled over. To get the URL information back in the file, stop and start the web service.You should specify the URL listed here for the createEvent and updateEvent operations, with one exception. You must replace the host name of localhost in the URL with the actual host name or IP address of the system where the SCOM Web Service is installed.
Go to the bottom of the file and search backwards for the string Setting the server's publish address to be. Continue searching backwards until you find the URL that contains SCOMService.
You should specify the URL listed here for the setup, initialize and uninitialize operations, with one exception. You must replace the host name of localhost in the URL with the actual host name or IP address of the system where the SCOM web service is installed.
Log in to the Oracle Enterprise Manager console with an account that has Super Administrator permissions.
From the Enterprise Manager console, click Setup.
Select Extensibility in the pane.
Select Management Connectors in the left pane.
Click the name of the SCOM connector.
The General tab of the Configure Management Connector page appears.
Verify that the URL identified in step 4 is specified for the creatEvent and updateEvent operations.
Verify that the URL identified in step 5 is specified for the setup, initialize and uninitialize operations.
If any of the operations are incorrect, change to the correct URL and click OK.
Perform the following diagnostic steps if messages are not being generated or updated as expected in SCOM.
Verify that the event that was triggered is referenced in a rule that forwards events to the SCOM connector.
Verify that the OracleEnterpriseManager.Alert.Creator Management Pack has been imported into the SCOM server:
Open the System Center Operations Manager console window and log in.
In the Administration pane of the Operations Manager console, select Administration, then Management Packs.
Verify that the "OracleEnterpriseManager Alert Creator" Management Pack is listed.
If the pack is not listed, import it as specified in Installing the Alert Creator Management Pack.
Determine the error that Oracle Enterprise Manager has reported.
To do this you need to examine the log file for errors. Perform the following steps to locate errors in the log file:
1.) Open the emoms_pbs.trc file in a text editor. The file is located in the log directory at the following location:
<EM_INSTANCE_BASE>/em/<OMS_NAME>/sysman/log/
Where, <EM_INSTANCE_BASE> is the OMS Instance Base directory. By default, the
OMS Instance Base directory is gc_inst, which is present under the parent directory of the Oracle Middleware Home.
For example, if the Oracle Middleware Home is /u01/app/Oracle/Middleware,
then the instance base directory is /u01/app/Oracle/Middleware/gc_inst, and the log and trace files are available in /u01/app/Oracle/Middleware/gc_inst/em/EMGC_OMS1/sysman/log/ directory path.
2.) Go to the bottom of the file and search backwards for Caused by. Lines that start with Caused by contain error information. The error information comes after the text in the line that reads oracle.sysman.emSDK.webservices.outbound.WSInvokeException: caught WebServiceException:.
Diagnose the problem based on the error information. See Errors from Oracle Enterprise Manager for information on troubleshooting common error events.
This section provides cause and solution information on troubleshooting common error messages. Find the error message in Table 7-1 that matches your error message, then refer to the corresponding section(s) indicated under Possible Cause for instructions to diagnose and correct the problem.
Table 7-1 Enterprise Manager Error Messages
| Error Message | Possible Cause |
|---|---|
javax.xml.soap.SOAPException: javax.xml.soap.SOAPException: Bad response: 403 Forbidden from url … |
|
javax.xml.soap.SOAPException: javax.xml.soap.SOAPException: Message send failed: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target |
|
javax.xml.soap.SOAPException: javax.xml.soap.SOAPException: Message send failed: Connection refused |
|
javax.xml.soap.SOAPException: javax.xml.soap.SOAPException: Message send failed: No route to host |
|
javax.xml.soap.SOAPException: javax.xml.soap.SOAPException: Bad response: 404 Not Found from url … |
or |
javax.xml.soap.SOAPException: javax.xml.soap.SOAPException: Message send failed: Connection timed out |
|
javax.xml.soap.SOAPException: javax.xml.soap.SOAPException: Message send failed: hostname |
|
javax.xml.transform.TransformerConfigurationException: Could not compile stylesheet |
|
Unable to reconnect to server after being disconnected |
|
ERROR - Could not connect to the server <hostname> because it is not operational |
|
ERROR - Could not login to the server because the account was invalid or has insufficient permissions |
or |
ERROR occurred invoking SCOM connector to insert event for null |
or |
javax.xml.ws.WebServiceException: org.apache.cxf.service.factory.ServiceConstructionException: Failed to create service |
|
Request failed because the specified management pack could not be found |
|
Successfully inserted the event but timed out waiting for the alert to be created |
The following errors are described:
The user name or password for accessing the SCOM web service is incorrect.
Log in to the Oracle Enterprise Manager console with an account that has Super Administrator privileges.
From the Enterprise Manager console, click Setup.
Select Extensibility in the pane.
Select Management Connectors from the menu. This will display the Management Connectors page.
Click the name of the appropriate SCOM connector.
This invokes edit mode, enabling you to configure the connector.
Correct the SCOM Web Service Username and SCOM Web Service Password fields, then click OK.
The SSL handshake between the Oracle Enterprise Manager Connector Framework and the SCOM web service failed. This failure occurs because Oracle Enterprise Manager is not configured correctly with the SSL certificate for the SCOM web service. The SSL certificate the SCOM web service uses must be imported into the Enterprise Manager key store. The certificate is either missing from the key store or does not match the SSL certificate provided by the SCOM web service.
Import the SSL certificate from the SCOM web service into the Enterprise Manager key store. See Configuring Enterprise Manager to use SSL for details on setting up Oracle Enterprise Manager with the SCOM SSL certificate.
The SCOM web service is down.
Perform the following steps to check the status of the web service and start it if necessary.
If the SCOM web service is installed on a Unix system:
Open a command terminal on the system where the SCOM web service is installed.
Change the working directory to the adapters/bin directory in the SCOM web service installation directory.
Enter the following command:
./service.sh status
If the command indicates that the service is not running, enter the following command:
./service.sh start
If the SCOM web service is installed on a Windows system:
Open a command terminal on the system where the SCOM web service is installed.
Change the working directory to the adapters/log directory in the SCOM web service installation directory.
Open the framework.log file in a text editor.
Go to the bottom of the file and search backwards for the string iWave Adapter Framework. If the last occurrence found is iWave Adapter Framework Started, this indicates that the web service is started.
If the web service is not started, start the web service as specified in Running the Web Service on Windows.
The IP address specified in the URL is invalid or the network is down.
Verify that the hostname/IP address configured for the connector is correct:
Log in to the Oracle Enterprise Manager console with an account that has Super Administrator privileges.
From the Enterprise Manager console, click Setup.
Select Extensibility in the pane.
Select Management Connectors from the menu. This will display the Management Connectors page.
Click the name of the appropriate SCOM connector.
This invokes edit mode, enabling you to configure the connector.
Verify that the hostname/IP address specified in the URL for the createEvent and updateEvent operations are correct.
If the hostname/IP address is incorrect, provide the correct value, then click OK.
If the URLs specify a host name, make sure that the host name resolves to the correct IP address. To determine the IP address of the host name, issue the ping <hostname> command, where <hostname> is the actual host name. This lists the IP address that was resolved for the host name. If this is incorrect, the system administrator needs to investigate why it is incorrect.
If the hostname/IP address appears to be correct, try to ping the system where the SCOM web service is installed using the hostname/IP address. If the ping fails, the system administrator needs to investigate why there is no connectivity.
The port number specified in the URL is invalid.
Verify that the port number configured for the connector is correct:
Log in to the Oracle Enterprise Manager console with an account that has Super Administrator privileges.
From the Enterprise Manager console, click Setup.
Select Extensibility in the pane.
Select Management Connectors from the menu. This will display the Management Connectors page.
Click the name of the appropriate SCOM connector.
This invokes edit mode, enabling you to configure the connector.
Verify that the port number specified in the URL for the createEvent, updateEvent, setup, initialize, and uninitialize operations are correct.
If the port number is incorrect, provide the correct value and click OK.
The web service received the request and rejected it because an invalid path was specified in the URL.
Perform the following steps to test the URL the connector is using.
Log in to the Oracle Enterprise Manager console with an account that has Super Administrator privileges.
From the Enterprise Manager console, click Setup.
Select Extensibility in the pane.
Select Management Connectors from the menu. This will display the Management Connectors page.
Click the name of the appropriate SCOM connector.
This invokes edit mode, enabling you to configure the connector.
Select and copy the URL specified for the createEvent operation.
Open an internet browser on the system where the Oracle Enterprise Manager server is installed.
In the address window, enter the URL that was copied in step 6 above. Add ?wsdl to the end of the URL. The URL should appear similar to the following example:
http://[Hostname]:8080/services/SCOM/EventService?wsdl
[Hostname] is the actual host name or IP address where the SCOM web service is installed.
If the WSDL is loaded, this confirms that the URL is correct. If it fails to load, there is a problem with the URL. Perform the steps specified in Using the Correct URL for SCOM Web Service Operations to configure the connector to use the correct URL.
A firewall is blocking access to the system where the SCOM Web Service is installed.
Contact your IT department to give Enterprise Manager access to the port used by the SCOM Web Service. Perform the steps specified in Using the Correct URL for SCOM Web Service Operations to determine the URL used by the SCOM Web Service. The port number specified in the URL is the port number the IT department should open in the firewall.
The system does not recognize the host name specified in the URL.
You can use the following options to address this issue:
Coordinate with the system administrator to change the system configuration to recognize the host name.
Specify the IP address in the URL instead of the host name. To do this, perform the following steps:
Determine the IP address of the system where the SCOM web service is installed.
Log in to the Oracle Enterprise Manager console by entering a user name with a Super Administrator role, entering the appropriate password, then click Login.
From the Enterprise Manager console, click Setup, then Extensibility, and finally Management Connectors. The Management Connectors page appears, which shows the installed connectors.
Click on the Configure icon associated with the Microsoft SCOM connector. This invokes edit mode, enabling you to configure the connector.
Change the host name to the IP address in the URL specified for the createEvent, initialize, setup, uninitialize, and updateEvent operations.
Click OK.
The connector framework could not process the request because the XSL file was formatted incorrectly. This problem should not occur unless the connector has been customized.
Examine any changes made to the XSL template files for mistakes that could have caused the problem. If you can't find the problem manually, load the XSL in a utility that performs XML validation.
The SCOM Agent could not insert the alert into SCOM because the wrong host name is configured for SCOM or the SCOM server is down.
Perform the following steps to determine and correct the root cause of the problem:
Verify that the host name or IP address listed in the error message is correct for the SCOM server. If the host name or IP address are incorrect, perform the following steps to correct the configuration:
Open Windows Explorer on the system where the SCOM Agent is located.
Navigate to the bin directory in the SCOM Agent installation directory.
Run the SCOMAgentConfig.exe utility to start the SCOM Agent Configuration Tool.
Click Load to open a directory navigation window.
Navigate to the SCOM Agent installation directory and open the SCOMAgent.cfg file.
Click the Management Groups tab, then click Edit to display the Edit Management Group window.
Correct the hostname/IP address in the Server field, then click Update.
Click Save to save the changes to the configuration file.
Click Exit to exit the utility.
Stop and restart the SCOM Agent in IIS.
Verify that the following OpsMgr services are running:
System Center Data Access
System Center Management
System Center Management Configuration
The SCOM Agent could not send the alert to the SCOM server, because the credentials configured for accessing the SCOM API are invalid.
Perform the following steps to change the credentials for accessing the SCOM API:
Open Windows Explorer on the system where the SCOM Agent is located.
Navigate to the bin directory in the SCOM Agent installation directory.
Run the SCOMAgentConfig.exe utility to start the SCOM Agent Configuration Tool.
Click Load to open a directory navigation window.
Navigate to the SCOM Agent installation directory and open the SCOMAgent.cfg file.
Click the Management Groups tab, then click Edit to display the Edit Management Group window.
Correct the credential information in the Domain, Username, and Password fields, then click Update.
Click Save to save the changes to the configuration file.
Click Exit to exit the utility.
Stop and restart the SCOM Agent in IIS.
The SCOM Agent could not send the alert to the SCOM server, because the credentials configured for accessing the SCOM API do not have sufficient permissions.
Refer to Setting Up the Agent Account. This section provides the steps required to set up the account for accessing the SCOM API.
The web service could not create an alert in SCOM because the SCOM Agent is not operational.
Open IIS Manager on the system where the SCOM Agent was installed, and start the web site for the Agent.
The web service could not connect to the SCOM Agent because the web service has an invalid configuration parameter. Either the URL for the SCOM Agent is incorrect or the credentials for accessing the SCOM Agent are invalid.
Verify that the URL for the SCOM Agent is correct. You should specify the URL that was provided at the end of the SCOM Agent installation. Note that if the host name in the URL is localhost and you are accessing it from another system, you need to replace localhost with the host name or IP address of the SCOM Agent installation machine.
If you do not know the URL, you can determine it as follows:
If the SCOM Agent was installed as a web site, the address is:
http://<IP>:<port>/Service.asmx
... where <IP> is the IP address, and <port> is the port number specified when installing the Agent.
If the SCOM Agent was installed as a virtual directory, the address is:
http://<IP>:<port>/<vdir>/Service.asmx
... where <IP> is the IP address, <port> is the port number for the web service where the agent was installed, and <vdir> is the virtual directory name specified for the Agent.
Select a user name and password that are valid on the system where the SCOM Agent was installed.
Open a command window and change the working directory to adapters\endpoints\SCOM in the SCOM web service installation directory.
Rerun the SCOM Web Service installer using the URL and credentials from the preceding steps. See Installing the Web Service on Unix or Installing the Web Service on Windows, depending on your platform, for the procedure.
Restart the web service as instructed in Installing the Web Service on Unix or Installing the Web Service on Windows.
The web service could not create an alert in SCOM because the OracleEnterpriseManager.Alert.Creator management pack has not been imported into SCOM.
Refer to Installing the Alert Creator Management Pack for the steps required to import the management pack into SCOM.
The web service was able to insert an event in SCOM, but an alert was not created within the timeout period. This likely indicates that an error occurred in the alert generating rule and it was unloaded by SCOM. Whenever this occurs, the System Center Operations Manager Health Service generates an error followed by a warning in the Operations Manager log. The error entry begins with the following message:
A module reported an error 0x80070057 from a callback which was running as part of rule "Create.Default.Alert" running for instance "OracleEnterpriseManager Event Source" with id ...
The warning entry begins with the following message:
Summary: 1 rule(s)/monitor(s) failed and got unloaded, 1 of them reached the failure limit that prevents automatic reload ...
Note:
This situation should not occur if the default SCOM connector configuration files are used. The only known way this can occur is if the SCOM Agent web service is directly accessed and an invalid value is passed for the Priority or Severity fields.Restart the Windows service named "Ops Mgr Health Service" on the SCOM server.