Issue: Tasks fail to start or task submissions fail. The SOA diagnostic log shows this error, indicating that the SOA server cannot connect to the MSAD server:
[soa_server1] [ERROR] [OVD-60143] [oracle.ods.virtualization.engine.backend.jndi.MSAD.BackendJNDI] [tid: [ACTIVE].ExecuteThread: '14' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: cfndmr] [ecid: 0000J5qkW1R4epYVLqESOA1EBZ6^0003dU,1:23453] [APP: soa-infra] [#MSAD] Unable to create connection to ldap://[ldapcml.XXXX.ad]:389 as CN=XXXXX,OU=ServiceAccounts,DC=XXXX,DC=ad.[[ javax.naming.NamingException: No LDAP connection available to process request for DN: CN=XXXXX,OU=ServiceAccounts,DC=XXXXX,DC=ad
Solution: Modify the LibOVD Adapter configuration to increase the AD LDAP Adapter connection pool to 100:
Navigate to the SOA Oracle home directory; for example, MIDDLEWARE_HOME/Oracle_SOA1/common/bin.
Run wlst.sh (UNIX) or wlst.cmd (Windows).
Connect to WebLogic Administration Server using the connect() command.
Enter this command:
modifyLDAPAdapter(adapterName='MSAD', attribute='MaxPoolSize', value=100)
Stop and restart WebLogic Administration Server and the managed server in which SOA application is running to activate the new connection pool setting.
Note: | If the wlst command fails, manually edit this file on Weblogic Administration Server to increase the MaxPoolSize for MSAD adapter to 100: MIDDLEWARE_HOME/user_projects/domains/EPMSystem/config/fmwconfig/ovd/default/adapters.os_xml Stop and restart WebLogic Administration Server and the managed server in which SOA application is running to activate the new connection pool setting. |
All issues addressed in this section indicate that tasks have not started as expected.
Issue: Schedule status does not change from Pending or reverts to Pending after being set to Open.
Solution: A schedule status reverting to Pending indicates that an error occurred when the main orchestration composite to SOA server was created and deployed. Follow these steps to locate and resolve the error:
Check MIDDLEWARE_HOME/user_projects/domains/domain name/servers/SOA managed server name/Logs/soa_server1-diagnostic.log for any exceptions in the SOA server at the time the status reverted. An exception might indicate, for example, that the SOA server ran out of memory.
Check MIDDLEWARE_HOME/user_projects/domains/domain name/servers/managed server name/Logs/FinancialClose.log for errors that occurred in the Financial Close Management managed server.
Note: | You can increase the logging level to get more debugging information by editing the logging.xml in MIDDLEWARE_HOME/user_projects/domains/domain name/config/fmwconfig/servers/managed server name. to change the level to TRACE:32. |
If you see the following NullPointerException in FinancialClose.log, log on to the database and ensure that the TEMPLATE_CONTENT column of FCC_COMPOSITE_TEMPLATES is populated:
Exception NullPointerException has occurred in fcc.model.applicationModule.bpel.CompositeGenerator.generateCompositeArtifacts() [line:120] after the invocation of method: fcc.model.applicationModule.IntegrationTypeManager.handleIntTypeMediator() [line:470]
Ensure that the SOA managed server and the WebLogic Administration Console server are both running.
An error message resembling this one indicates that the SOA managed server is not running:
[2010-07-27T14:14:25.094-04:00] [FinancialClose0] [ERROR] [] [oracle.apps.epm.fcc.model] [tid: 23] [userId: admin] [ecid: 0000IcL7CiR1BhMLUM5Eic1CJPkU0000um,0] [SRC_CLASS: oracle.apps.epm.fcc.model.applicationModule.bpel.CompositeDeployer] [APP: FinancialClose] [SRC_METHOD: m_executeCommand] Can't find resource for bundle java.util.PropertyResourceBundle, key Failed deploying the composite[[ java.net.ConnectException: Connection refused: connect at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
An error message resembling this one indicates that the WebLogic Administration Console server is not running:
[2010-07-23T16:56:47.266-04:00] [FinancialClose0] [ERROR] [] [oracle.apps.epm.fcc.model] [tid: 15] [userId: admin] [ecid: 0000Ic160D^2FSYVLqaQOA1CIS1300006t,0] [SRC_CLASS: oracle.apps.epm.fcc.model.applicationModule.SOAServerManager] [APP: FinancialClose] [SRC_METHOD: _initJMXConnector] [[ java.io.IOException at weblogic.management.remote.common.ClientProviderBase.makeConnection(ClientProviderBase.j ava:195) at weblogic.management.remote.common.ClientProviderBase.newJMXConnector(ClientProviderBase. java:83) at javax.management.remote.JMXConnectorFactory.newJMXConnector(JMXConnectorFactory.java: 338)
Issue: Tasks do not start after the schedule is set to Open status.
Solution: After the schedule is set to Open status, any tasks that have start times in the past and that have no predecessors should change to Open Running status. Ensure that your system is configured correctly.
If tasks do not start when the data sources are configured correctly, follow these steps:
Log on to the Enterprise Manager console (http://WebLogic Admin host:WebLogic Admin port/em).
On the left, expand SOA and then soa-infra (soa_server1).
In the Deployed Composites list in the Dashboard on the right, click MainOrchXXXComposite (at the top of the table), which is the composite for the most recently opened schedule.
If no MainOrchxxComposite has been created, run epmsys_registry.bat view FINANCIAL_CLOSE_PRODUCT/LOGICAL_WEB_APP/FINANCIAL_CLOSE_WEB_APP/APP_SERVER to determine whether adminHost and adminPort properties exist.
If adminHost and adminPort properties do not exist, then Financial Close Management is not linked to the correct APP_SERVER component, which is WebLogic 10 (APP_SERVER). There must be two instances of APP_SERVER components in the registry. Make a note of the both APP_SERVER component ID, and run these commands to resolve the issue:
epmsys_registry.bat removelink # Financial Close Management Product ID # Wrong APP_SEVER Component ID
epmsys_registry.bat createlink # Financial Close Management Product ID # Correct APP_SEVER Component ID
If the number of instances is zero, which indicates that an error may have occurred during event configuration, verify the EDN setup:
If SOA server was configured using MS SQL Server, check EDN settings as follows:
Log on to the Oracle Enterprise Manager console (http://WebLogic Admin host:WebLogic Admin port/em), and ensure that the event is set to EDN-JMS mode.
Log on to WebLogic Administration Console (http://WebLogic Admin host:WebLogic Admin port/console) and ensure that these conditions are true:
EDNDataSource and EDNLocalTxDataSource JDBC data sources are deleted.
The EDN-JMS foreign JNDI provider is set up correctly.
If the SOA server was configured using Oracle Database, log on to WebLogic Administration Console (http://WebLogic Admin host:WebLogic Admin port/console) , and ensure that the EDNDataSource and EDNLocalTxDataSource data sources are targeted to both the Financial Close Management managed server and the SOA server.
Tip: | If you are using Oracle Database, you can check all events published to the SOA server at http://SOA_server_host:8001/soa-infra/events/edn-db-log. |
If the number of instances for the main orchestration composite is 1 or greater and you see no other issues with the main orchestration composite, click FCCTaskExecutionComposite, which is the composite that executes each task in the schedule. Check for any Recent Fault and Rejected Messages for the task execution composite in the Dashboard.
Tip: | Ensure that the RECORDSTR column of the WL_LLR_FINANCIALCLOSE0 table has a width of 4000. |
Check MIDDLEWARE_HOME/user_projects/domains/domain name/servers/SOA managed server name/Logs/soa_server1-diagnostic.log for exceptions in the SOA server at the time the schedule status was set to Open.
An exception in the SOA diagnostic log might indicate, for example, that the business event was not published correctly or that SOA data sources were suspended.
Common errors to look for in the SOA log:
Caused by: java.security.cert.CertificateExpiredException: NotAfter: Thu Aug 26 17:37:01 EDT 2010 at sun.security.x509.CertificateValidity.valid(CertificateValidity.java:256) at sun.security.x509.X509CertImpl.checkValidity(X509CertImpl.java:570) at sun.security.x509.X509CertImpl.checkValidity(X509CertImpl.java:543) at oracle.wsm.security.jps.WsmKeyStore.getJavaCertificate(WsmKeyStore.java:505)
This error indicates that the keystore has expired. Recreate it, and then copy the keystore files to the correct Oracle Fusion Middleware config folder. For instructions, see “Configuring the Keystore for Oracle Web Services Manager” in the Oracle Hyperion Enterprise Performance Management System Installation and Configuration Guide.
[ecid: 0000IgHXWnOCknYVLqNM8A1CZoZd0000DK,0:1:0x5f5e458:3:100000862] [APP: soa-infra] <BaseCubeSessionBean:: log error> Error while invoking bean "cube delivery": Exception not handled by the Collaxa Cube system.[[ an unhandled exception has been thrown in the Collaxa Cube systemr; exception reported is: "ORABPEL-00000 Exception not handled by the Collaxa Cube system. an unhandled exception has been thrown in the Collaxa Cube systemr; exception reported is: "Local Exception Stack: Exception [EclipseLink-4002] (Eclipse Persistence Services - 1.2.0.v20091016-r5565): org.eclipse.persistence.exceptions.DatabaseException Internal Exception: java.sql.SQLException: Internal error: Cannot obtain XAConnection weblogic.common.resourcepool.ResourceDeadException: 0:weblogic.common.ResourceException: Listener refused the connection with the following error: ORA-12516, TNS:listener could not find available handler with matching protocol stack at weblogic.common.resourcepool.ResourcePoolImpl.reserveResourceInternal(ResourcePoolImpl.java:436) at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:332) at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:433) at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:316) at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:93)
This error indicates that the database server cannot handle the load. Increase the PROCESSES parameter value of the database.
FabricInvocationException[[ javax.xml.ws.soap.SOAPFaultException: Transaction timed out after 31 seconds.
This error indicates that the database server cannot handle the load, and the call from SOA timed out. Increase the JTA configuration of this WebLogic Server domain: From WebLogic Admin Server Console, go to the JTA tab, and increase the value of Timeout Seconds.
ORABPEL-10509 User not found. User "#error:noapi#" is not found in configuration "jazn.com"...
The most likely cause of this error is that Financial Close Management failed to retrieve user ID from Shared Services. Check the JDBC data source EPMSystemRegistry in your WebLogic Administration Console to ensure that the connection pool is large enough to handle the number of calls to retrieve user IDs. Connection pool size requirements vary, but the connection pool should exceed the number of close tasks that could start at the same time in the close process. For example, if 50 close tasks could start at the same time, then the connection pool size should be greater than 50.
Caused by: com.oracle.bpel.client.BPELFault: faultName: {{http://schemas.oracle.com/bpel/extension}remoteFault}messageType: {{http://schemas.oracle.com/bpel/extension}RuntimeFaultMessage}
parts: {{
summary=<summary>oracle.fabric.common.FabricInvocationException: Unable to access the following endpoint(s): http://<hostname>:<port>>/FCC-DataModel-context-root/SOAAMService<//summary>
,detail=<detail>Unable to access the following endpoint(s): http://<hostname>:<port>>/FCC-DataModel-context-root/SOAAMService</detail> ,code=<code>null</code>
Ensure that all servers in the WebLogic domain have the correct keystore and credential store files. This error usually results from an incorrect keystore setup.
If the error remains unresolved, enable OWSM logging to see more details about the error. See Enabling OWSM Logging.
These errors:
MDSConfigurationException encountered in parseADFConfigurationMDS-01330: unable to load MDS configuration document.
MDS-01329: Unable to load element "persistence-config"
MDS-01370:MetadataStore configuration for metadata-store-usage "OWSM_TargetRepos" is invalid.
MDS-00922: The ConnectionManager "oracle.mds.internal.persistence.db.JNDIConnectionManagerImpl" cannot be instantiated.
MDS-00929: Unable to look up name "jdbc/mds/owsm" in JNDI context While trying to lookup 'jdbc.mds.owsm' didn't find subcontext 'mds'. Resolved 'jdbc'
In WebLogic Administration Console, verify that wsm-pm is targeted only to the SOA managed server..
Also verify that the JDBC data source mds-owsm is targeted to AdminServer and the SOA managed server.
The policy reference URI is not valid.
In a browser, open http://soa-host:soa-port/wsm-pm/validator (for example, http://localhost:8001/wsm-pm/validator) to verify that your OWSM configuration is correct. If the OWSM configuration is correct, the message Policy Manager Status: Operational is displayed with a list of supported security policies.
If the policy manager status is not operational, check the settings in WebLogic Administration Console. Common OWSM configuration errors include having the application wsm-pm deployed to multiple targets and not targeting the JDBC data source mds-owsm correctly. The application wsm-pm should only be targeted for the SOA managed server.
java.sql.SQLException: Unexpected exception while enlisting XAConnection java.sql.SQLException: XA error: XAResource.XAER_NOTA start() failed on resource 'SOADataSource_EPMSystem': XAER_NOTA : The XID is not valid.
For a JDBC data source that uses XA drivers, use WebLogic Administration Console to verify that XA Transaction Timeout is enabled and XA Transaction Timeout is set to 0.
Issue: A task is changed to Error status.
Solution: Log on to Oracle Hyperion Financial Close Management, and click the History tab. A row on the History tab displays the detailed error message.