General Troubleshooting Tips
The following are some general troubleshooting tips:
This release of Oracle Fusion Applications relies on certified versions of supported platforms documentation for Oracle Fusion Applications for details about hardware and software, minimum disk space and memory requirements, required system libraries, packages, or patches, and minimum database requirements.
If incorrect information is entered on one of the installation screens, return to that screen by clicking Back until the screen is shown.
When an environment is provisioned, provisioning writes debug information to the debug directory (APPLICATIONS_BASE/provisioning/debug).
Do not delete any files from this directory.
Troubleshoot errors using the files in this directory.
The Classloader Analysis Tool (CAT) is a Web-based class analysis tool which simplifies filtering classloader configuration and aids in analyzing classloading issues, such as detecting conflicts, debugging application classpaths and class conflicts, and proposes solutions to help to resolve them. CAT is deployed to every WebLogic domain out of the box. For more information about the Classloader Analysis Tool, see Using the Classloader Analysis Tool (CAT) in the Oracle Fusion Middleware Developing Applications for Oracle Weblogic Server Guide.
Backup Tar Creation
During the postconfigure phase, a "Value too large for defined data type" error message might be displayed while creating the backup tar.
[2016-11-16T00:18:06.695-08:00] [runProvisioning-postconfigure] [WARNING] [] [runProvisioning-postconfigure] [tid: 4881] [ecid: 0000LXgDiJbFW73LVmc9yc1OAcO70000KP,0] [exec] tar: ./domains/<hostname>/CommonDomain/ucm/cs/vault/timesize.test: Value too large for defined data type [2016-11-16T00:18:37.455-08:00] [runProvisioning-postconfigure] [NOTIFICATION] [] [runProvisioning-postconfigure] [tid: 1] [ecid: 0000LXa6sikFW73LVmc9yc1OAcO7000001,0] [logStatus] State=Error!Timestamp=2016-11-16 00:18:37 PST!Target=backup-instance!Category=backup!Domain=None!Host Name=<hostname>!Product Family=orchestration!Product=orchestration!Task=backup!Task ID=orchestration.orchestration.None.backup-instance.None!Message=Unable to create backup of /scratch/mwport/FAREL12/APPTOP/instance at /scratch/mwport/FAREL12/APPTOP/provisioning/restart/backup_postconfigure: exec returned: 1!Detail=!Build File=/scratch/mwport/REL12_ST12/provisioning/provisioning-build/common-restart -build.xml!Line Number=662!
Resolution
Follow these steps to resolve the issue:
Review the error and find out the file causing the "Value too large for defined data type" error
Open that file using the file editor and close it
Run cleanup and restore for postconfigure target
Rerun postconfigure target
Log files contain information about both normal and problematic events. They can help to diagnose and address some problems. For example, log messages that state that a service cannot be reached might indicate a hardware failure.
If a more complex issue is discovered, My Oracle Support personnel may use log files to trace the execution code paths of relevant requests as part of diagnosing the problem. Log files are particularly helpful if the Oracle Fusion Applications environment contains custom code that needs debugging, especially when using a debugger is not feasible (such as on a production system).
During each provisioning phase, the Provisioning Wizard writes the actions of the build processes to a log file created for each domain. Click the Log file icon to see details and error messages. In the log file, search for a specific text string, or move forward and backward through the content. The wrap feature enables text to be easily printed, or even forwarded by email.
In some cases, it might be important to review the log files written by provisioning to find out more about the errors that are encounter and recommended corrective actions. For example, any error during cleanup or restore. Provisioning writes log files to the following location:
UNIX:
APPLICATIONS_BASE/logs/provisioning/host_name
This shared location is accessible from all hosts, and contains the following log files:
runProvisioning-default-main.log: The main log file.
runProvisioning-phase_name.log: The main log file for a given phase, containing the output and error streams. These logs are used by the wizard to keep track of internal information. For example, for runProvisioning-preverify.log, each provisioning thread then writes its own log to runProvisioning-product_family-phase_name.log.
runProvisioning-product_family-phase_name.log: Displayed in the Provisioning Wizard as a Log icon for preconfigure, configure, configure-secondary, postconfigure, and startup phases. The files contain detailed information about the phase. For example, runProvisioning-fin-preverify.log contains information about the preverify actions taken while creating the Oracle Fusion Financials domain.
Because all reference roles must be provided in each product log file, expect to see duplicate reference role entries.
Provisioning also relies on the Oracle Universal Installer (OUI), which writes log files as follows:
UNIX:
Oracle_Inventory_Location/logs
If the location of the Oracle Inventory directory is unknown, it is possible to find it at:
UNIX: /etc/oraInst.loc
Note that APPLICATIONS_BASE is the root directory under which the provisioned environment resides. Except for the web tier host, this physical location must be on a shared drive.
In addition to log file locations discussed in this section, note these log file locations associated only with the preconfigure, configure, configure-secondary, postconfigure, and startup phases:
Oracle WebLogic Server:
UNIX: APPLICATIONS_CONFIG/domains/host_name/domain_name/servers/server_name/logs
Oracle WebLogic Server Node Manager:
UNIX: APPLICATIONS_BASE/fusionapps/wlserver_10.3/common/nodemanager/host_name/
The default log level for provisioning is TRACE:1 and NOTIFICATION:1 for messages written to provisioning log files and console, respectively. The definition of loggers controlling the log level for provisioning log files and console are described as:
<logger name="runProvisioning-default-main" level="TRACE:1">
<handler name="default-main"></handler>
</logger>
<logger name="runProvisioning-console" level="NOTIFICATION:1">
<handler name="prov-cons-handler"></handler></logger>
</loggers>
To change the log level, modify the corresponding logger definition in the framework_location/provisioning/bin/prov-logging-config.xml file.
For more information about log level attributes, see Setting the Level of Information Written to Log Files in the Oracle Fusion Middleware Administering Oracle Fusion Middleware.
The wizard performs automated cleanup and recovery actions. If those processes cannot clean up and restore the session, perform the actions manually.
See Start and Stop Components in the Oracle Fusion Applications Administrator's Guide for complete instructions for stopping and starting components.
Recovery is intended to be systemwide. If a failure occurs on one server, cleanup and recovery steps must be performed on all servers, including the hosts on which the phase completed successfully. During the installation, errors may occur during the running of any of the installation phases. To use the abort feature, it might necessary to perform some cleanup tasks as well.
A Retry button is available on each provisioning phase interview screen to initiate cleanup on the primordial host for that phase, or on a non-primordial host if that is where provisioning was started. Initiating the retry operation affects the full phase, beginning with the primordial host or the host from which provisioning was started. The Retry UI explicitly tells on which hosts execute cleanup, and specifies the command to use to run cleanup. A message displays that tells which host the cleanup target is being run on. If additional cleanup is required on other hosts, those host names are specified after the cleanup target completes. The wizard indicates what cleanup tasks are required, enables the Continue button, and waits until clicking Continue after completing the additional cleanup. Clicking Continue initiates a process to confirm whether the cleanup is complete. If no additional cleanup is required, the Continue button remains disabled, and the wizard starts executing the restore action.
When the restore completes on the current host, a message again displays that tells which hosts execute restore, along with the command to run. The OK button is enabled when the restore is done on the current host. When clicking OK, a process checks again to confirm whether the restore is complete and an error message displays if the restore is not complete. When the restore is done, the phase restarts on the current host if needed. If the phase does not need to run, the retry window closes and the information from the previous run of the phase displays. In the hosts table at the top of the screen, all hosts that were either in a failed or aborted state before the retry is started, are reset when the retry window closes. Then, restart the phase from the command line for all hosts with a reset status.
The wizard detects hosts that require cleanup and displays a message informing of the host names. Perform the cleanup action from the command line on these hosts before initiating any restore action. Command-line syntax for the cleanup action takes the following form:
UNIX: path_to_script/runProvisioning.sh -responseFile provisioning_response_file_location -target cleanup-phase_name
The command-line syntax for the restore action takes the following form:
UNIX: path_to_script/runProvisioning.sh -responseFile provisioning_response_file_location -target restore-phase_name
These actions are available for the preverify, install, preconfigure, configure, configure-secondary, postconfigure, startup, and validate phases. They are also available for shutdown and deinstall actions.
When a failure occurs during one of the provisioning phases, fix the cause of the error and then retry the provisioning phase on the hosts that previously failed using the Provisioning Wizard and the provisioning command-line interface if the failed hosts are primary or secondary hosts.
To retry a provisioning phase, initiate it starting with the primordial host by clicking Retry on the Provisioning Wizard. The wizard starts the cleanup phase on the primordial host.
When prompted, execute a cleanup phase from the command line on each of the hosts that failed the provisioning phase. Do so simultaneously on all failed hosts.
After the cleanup phase completes successfully on all the hosts, continue with the restore phase followed by a retry of the provisioning phase. Start the restore phase first from the primordial host through the Provisioning Wizard, and run the restore phase from the command line on the other hosts from the terminal session. Do so simultaneously on all hosts.
After the restore phase is successful on all hosts, rerun the failed provisioning phase.
When a failure occurs during one of the provisioning phases, do the following:
The automated cleanup and restore actions cannot handle every type of failure. Sometimes manual steps are needed. This is true, for example, when the configure phase fails and any of the following situations exists:
Node Manager is not yet configured
Node Manager is configured with an invalid trust key
Administration Server is not yet registered with the Node Manager
Administration Server is not running
Under any of these circumstances, the Node Manager will not be able to shut down the Administration Server and the Managed Servers during cleanup-phase_name. Shut down manually all servers before continuing with the restore-phase_name.
Shut down Web Tier processes, if any, with this command:
UNIX: APPLICATIONS_CONFIG/CommonDomain_webtier/bin/opmnctl shutdown.
This applies to cleanup-configure, cleanup-configure-secondary, and cleanup-postconfigure.
Shut down Oracle Business Intelligence processes, if any, by running:
UNIX: BI_CONFIG_HOME/bin/opmnctl shutdown.
This applies to cleanup-configure, cleanup-configure-secondary, and cleanup-postconfigure.
Shut down Global Order Promising (GOP) (if provisioned) with this command:
UNIX: gop_instance_base/bin/opmnctl shutdown.
This applies to cleanup-postconfigure.
Shut down Java EE applications using the method recommended for the Oracle WebLogic Server.
This applies to cleanup-configure, cleanup-configure-secondary, and cleanup-postconfigure.
Shut down Node Manager only if it was not configured correctly. See Restart Node Manager on PROVISIONED_HOST.
Errors during cleanup of a target produce messages that inform of the error and display the contents of the associated log file. When scrolling through a message, additional messages can be viewed, including the manual steps that should be taken to fix the problem.
Note that failures during cleanup-install require the following specific cleanup tasks:
oraInventory that provisioning uses.APPLICATIONS_CONFIG/provisioning/phaseguards/.Provisioning cannot reliably stop all grandchild processes associated with failures during the running of a phase. Occasionally, remnant processes are present after a failure. Oracle recommends to manually check and stop all remnant processes that start from the APPLICATIONS_BASE and the framework_location/provisioning folder, except for Node Manager and the Provisioning Wizard.
Take this action after completing the cleanup action for any phase, and before running a restore action for that phase. Without this additional cleanup action, unwanted interference may be experienced with the restore action and subsequent rerun logic. This is especially true for long-running child processes such as LDAP policy migration.
Identify remnant processes in the APPLICATIONS_BASE as follows:
Linux: ps -ef | grep APPLICATIONS_BASE/folder_name
Identify remnant processes in the framework_location/provisioning folder as follows:
Linux: ps -ef | grep framework_location/provisioning/folder_name
Remember, do not stop the Node Manager and UI processes.
If the automated restore operation fails, complete these manual steps for all phases, except as noted:
Delete the restart phase guard (phase_name.grd) file associated with the failure. It is located under APPLICATIONS_BASE/restart/.
Restore the instance directory from the backup, located as follows:
UNIX: APPLICATIONS_BASE/restart/backup_phase_name/instance.tar
Execute the following commands to restore the instance directory:
UNIX:
rm -rf CONFIG_HOME
mkdir CONFIG_HOME
tar -xfv path_to_instance.tar/instance.tar -C CONFIG_HOME
When a local application configuration has been enabled, manually restore the localdomains and localapplications configuration directories on every local_application_config_host:
UNIX: APPLICATIONS_BASE/restart/backup_phase_name/local_application_config_host/localdomains.tar and...
UNIX: APPLICATIONS_BASE/restart/backup_phase_name/local_application_config_host/localapplications.tar
In addition, restore the following file related to Oracle Business Intelligence:
UNIX: APPLICATIONS_BASE/restart/backup_phase_name/local application_config_host/biinst/BIInstance.tar
For restore-configure-secondary and restore-postconfigure only, start the CommonDomain Administration Server.
For restore-postconfigure only, start Oracle HTTP Server by running WT_CONFIG_HOME/bin/opmnctl startall. Oracle HTTP Server must be started from the host where it is installed. It cannot be started from any other host.
For restore-configure and restore-postconfigure, check the restore logs to see if the Oracle Business Intelligence schema restore operation is complete. Perform the restore operation for the database contents first.
Some errors might be encountered during the preverify phase. This section details troubleshooting information for the preverify phase errors.
The following error message is displayed during the preverify phase if Oracle Fusion Applications is installed on the Red Hat Enterprise 6 platform:
[2013-03-22T12:13:01.241+05:30] [runProvisioning-preverify] [NOTIFICATION] [] [runProvisioning-preverify] [tid: 10] [ecid:0000JqGpNU12JRbptGc9yX1HIzpD000000,0] [validateInstallersPrerequisite] Check Name:CertifiedVersions
[2013-03-22T12:13:01.243+05:30] [runProvisioning-preverify] [NOTIFICATION] [] [runProvisioning-preverify] [tid: 10] [ecid:0000JqGpNU12JRbptGc9yX1HIzpD000000,0] [validateInstallersPrerequisite] Check Description:This is a prerequisite condition to test whether the Oracle software is certified on the current O/S or not.
[2013-03-22T12:13:03.175+05:30] [runProvisioning-preverify] [NOTIFICATION] [] [runProvisioning-preverify] [tid: 10] [ecid:0000JqGpNU12JRbptGc9yX1HIzpD000000,0] [validateInstallersPrerequisite] Expected result: One of oracle-6,oracle-5.6,enterprise-5.4,enterprise-4,enterprise-5,redhat-5.4,redhat-4,redhat-5,SuSE-10,SuSE-11
[2013-03-22T12:13:03.178+05:30] [runProvisioning-preverify] [NOTIFICATION] [] [runProvisioning-preverify] [tid: 10] [ecid:0000JqGpNU12JRbptGc9yX1HIzpD000000,0] [validateInstallersPrerequisite] Actual Result: redhat-Red
[2013-03-22T12:13:03.180+05:30] [runProvisioning-preverify] [NOTIFICATION] [] [runProvisioning-preverify] [tid: 10] [ecid:0000JqGpNU12JRbptGc9yX1HIzpD000000,0] [validateInstallersPrerequisite] Check complete. The overall result of this check is: Failed <<<<
[2013-03-22T12:13:16.718+05:30] [runProvisioning-preverify] [NOTIFICATION] [] [runProvisioning-preverify] [tid: 10] [ecid:0000JqGpNU12JRbptGc9yX1HIzpD000000,0] [logStatus] STATE=BUILD_ERROR!TIMESTAMP=2013-03-22 12:13:16 IST!TARGET=private-preverify-installers-prerequisite!CATEGORY=BUILD_ERROR!DOMAIN=NONE!HOSTNAME=in-mum-orafus02.corp.capgemini.com!PRODUCTFAMILY=orchestration!PRODUCT=orchestration!TASK=validateInstallersPrerequisite!TASKID=orchestration.orchestration.BUILD_ERROR.private-preverify-installers-prerequisite.validateInstallersPrerequisite!MESSAGE=The OUI installer prerequisite check failed:: Product : webtier Product : wc Product : soa Product : ecm_bucket2 Product: atgpf Product : odi Product : fusionapps Product : gop !DETAIL=The OUI installer prerequisite check failed :: Product : webtier|Product : wc|Product: soa|Product : ecm_bucket2|Product : atgpf|Product : odi|Product : fusionapps|Product : gop|!BUILDFILE=/u04/prov/provisioning/provisioning-build/common-preverify-build.xml!LINENUMBER=1354!
[2013-03-22T12:13:16.793+05:30] [runProvisioning-preverify] [ERROR] [FAPROV-01045] [runProvisioning-preverify] [tid: 10] [ecid:0000JqGpNU12JRbptGc9yX1HIzpD000000,0] *** Validation Error! ***[[]]
[2013-03-22T12:13:16.797+05:30] [runProvisioning-preverify] [ERROR] [] [runProvisioning-preverify] [tid: 10] [ecid:0000JqGpNU12JRbptGc9yX1HIzpD000000,0] The OUI installer prerequisite check failed :: Product : webtier[[Product : wc
Product : soa
Product : ecm_bucket2
Product : atgpf
Product : odi
Product : fusionapps
Product : gop
]]
This is caused by an error in the operating system validation. Apply the following workaround to resolve the issue:
When there is a build error in the preverify phase in the primordial host, not all validation errors in the non-primordial hosts will be accounted for in the Provisioning Wizard. This is because a build error, which is a much more severe error than validation, occurs in the primordial host before the logic to count validation errors. After fixing the issue that caused the build error and rerunning preverify phase, the validation errors are counted and displayed correctly. This is normal and expected.
Resolve the issue that caused the build error in the primordial host first and rerun preverify phase to find out if there are other validation errors among the hosts. Fix the validation errors where appropriate until validation errors are resolved before proceeding to the next phase.
During the preverify phase, an error might be displayed saying provisioning requires more free space for the local application configuration directory in the current host like the log below:
[2012-02-03T23:11:27.659-08:00] [runProvisioning-preverify] [NOTIFICATION] [] [runProvisioning-preverify] [tid: 12] [ecid: 0000JL7CzeGBDCAnv^n3F11FBDbj000003,0] [logStatus] STATE=BUILD_ERROR!TIMESTAMP=2012-02- 03 23:11:27PST!TARGET=private-preverify-filesystem-Free-space!CATEGORY=BUILD_ERROR!DOMAIN=NONE!HOSTNAME=adcdk16!PRODUCTFAMILY=orchestration!PRODUCT=orchestration!TASK=validateFileSystem!TASKID=orchestration.orchestration.BUILD_ERROR.private-preverify-filesystem-free-space.validateFileSystem!MESSAGE=The file system /scratch/aime/rc4 only has 44271 MB, but 76800 MB is needed! DETAIL=The file system /scratch/aime/rc4 only has 44271 MB, but 76800 MB is needed!BUILDFILE=/net/adcgei13/scratch/aime/rc4/FAINTEG_MAIN_PLATFORMS_120131.1600/provisioning/provisioning-build/common-preverify-build.xml!LINENUMBER=1392! [2012-02-03T23:11:27.698-08:00] [runProvisioning-preverify] [ERROR] [FAPROV-01045] [runProvisioning-preverify] [tid: 12] [ecid: 0000JL7CzeGBDCAnv^n3F11FBDbj000003,0] *** Validation Error! ***[[ ]]
After provisioning completes, it might be possible that only a fraction of the required file system is being used. This is normal. The 77 GB free space is the Oracle recommended value derived from the performance benchmark. It includes projection of disks required for storing diagnostic logs and other information in the local domain over a long period.
During the preverify phase in Solaris 5.11 environments, an error might be displayed saying that PS version 0.5.11 is not available.
[2016-11-15T14:00:45.485+00:00] [runProvisioning-preverify] [ERROR] [] [runProvisioning-preverify] [tid: 22] [ecid: 0000LXcIVwd6ATI5Irg8yf1OAlGw000003,0] PS version 0.5.11 required for Provisioning on Sun Solaris 5.11 is not available. Please install the correct UCB PS package
Execute the below commands to find out the version of the PS package in the environment:
PS_UCB_PKG=`pkg search -l -r /usr/ucb/ps|grep pkg|awk '{print $4}'`
PS_UCB_VERSION=`echo $PS_UCB_PKG | cut -f2 -d"@" | cut -f1 -d"-"`
echo PS_UCB_VERSION=${PS_UCB_VERSION}
If the PS_UCB_VERSION value is 0.5.11, then the above error does not have any functional impact and can be ignored.
To ignore the error, run the preverify target with the command line argument '-ignoreSysPrereqs true' in the runProvisioning command.
If the Human Capital Management (HCM) application offerings is being provisioned, namely Workforce Development and Workforce Deployment, and the /mnt/hwrrepo directory does not exist, a warning message is displayed.
Continue with the provisioning process if the Workforce Reputation (HWR) application is not needed.
If the HWR application is required, follow the instructions in Create the hwrepo directory to prepare the shared mount. Complete provisioning the environment first then prepare the shared mount before starting using the HWR application.
This section includes troubleshooting information for install phase errors.
Interrupt the installation process while it is in progress by clicking Cancel, or by clicking the X icon next to a build that has failed. If Cancel is clicked, all processes running in the background are terminated and put in a Failed status.
Start the wizard again after initiating a Cancel action. The wizard detects that the installation has been partially completed, and presents two options:
Resume from where it was left off. The wizard asks to resume. Click Yes.
The wizard takes to the screen where Cancel was clicked and created the failure. Restart the installation at that point by clicking the Retry button. The wizard performs cleanup and recovery actions.
Clean up the errors manually and rerun the Provision a New Applications Environment option for the response file from the beginning.
When a provisioning response file is created using the Oracle Fusion Applications Provisioning Wizard, if only one RAC node information is provided for the Oracle Fusion Applications database, then the following error is shown during the install phase in the runProvisioning-install.log. The error is generated due to a limitation in the Oracle Data Integrator (ODI) installer and at least two RAC nodes must be provided.
[2012-08-21T10:40:21.365-04:00] [runProvisioning-install] [NOTIFICATION] [] [runProvisioning-install] [tid: 31] [ecid: 0000J_9c8_p7U8lCgvq2So1GCtIP00000M,0] Starting Oracle Universal Installer...[[
Checking if CPU speed is above 300 MHz. Actual 1600 MHz Passed
Checking Temp space: must be greater than 300 MB. Actual 7107 MB Passed
Checking swap space: must be greater than 512 MB. Actual 14527 MB Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2012-08-21_10-40-14AM. Please wait ...Log: /u01/app/oraInventory/logs/install2012-08-21_10-40-14AM.log
Copyright (c) 1999, 2011, Oracle and/or its affiliates. All rights reserved.
Reading response file..
Verifying data......
[VALIDATION] [ERROR]:INST-07221: Specified connect string is not in a valid format
[VALIDATION] [SUGGESTION]:Provide the value in the following format. Format: hostname:port:servicename , For Real Application Cluster Database hostname1:port1^hostname2:port2@servicename
installation Failed. Exiting installation due to data validation failure.]]
[2012-08-21T10:40:21.387-04:00] [runProvisioning-install] [NOTIFICATION] [] [runProvisioning-install] [tid: 10] [ecid: 0000J_9V7RD7U8lCgvq2So1GCtIP000000,0] [logStatus] STATE=BUILD_ERROR!TIMESTAMP=2012-08-21 10:40:21 EDT!TARGET=install!CATEGORY=none!DOMAIN=NONE!HOSTNAME=<host.com>!PRODUCTFAMILY=orchestration!PRODUCT=orchestration!TASK=none!TASKID=orchestration.orchestration.NONE.install.NONE!MESSAGE=An Error Occured: !DETAIL=The following error occurred while executing this line:|<FRAMEWORK_LOCATION>/provisioning/provisioning-build/orchestration-build.xml:3132: The following error occurred while executing this line:|<FRAMEWORK_LOCATION>/provisioning/provisioning-build/common-misc-build.xml:109: An Error Occured: The following error occurred while executing this line:
|<FRAMEWORK_LOCATION>/provisioning/provisioning-build/ base-product-family-build.xml:85: The following error occurred while executing this line:|<FRAMEWORK_LOCATION>/provisioning/provisioning-build/fs-build.xml:281: The following error occurred while executing this line:|<FRAMEWORK_LOCATION>provisioning/provisioning-build/fs-build.xml:1448: The following error occurred while executing this line:|<FRAMEWORK_LOCATION>/provisioning/provisioning-build/base-techstack-build.xml:62: The following error occurred while executing this line:|<FRAMEWORK_LOCATION>/provisioning/provisioning-build/ odi-build.xml:326: OUI installer failed: <REPOSITORY_LOCATION>/installers/odi/Disk1/runInstaller! BUILDFILE=<FRAMEWORK_LOCATION>/provisioning/provisioning-build/ common-misc-build.xml!LINENUMBER=107!
Resolution
Provide the information for at least two RAC nodes on the Database Configuration page of the Provisioning Wizard, save the response file, and then restart the provisioning process to deploy a new Oracle Fusion Applications environment as detailed in the following steps:
APPLICATIONS_CONFIG folder including the folder. This is the location for Oracle Fusion Applications Home displayed on the Installation Location page of the Provisioning Wizard. If Local Applications Configuration is enabled, ensure that the Local Applications Configuration directory and its contents are deleted.The following section describes the steps to troubleshoot postconfigure phase errors.
During the postconfigure phase, some error messages might be encountered in the provisioning log for a domain (for example, in runProvisioning-crm-postconfigure.log).
[2013-12-02T01:51:38.122-08:00] [runProvisioning-crm-postconfigure] [NOTIFICATION] [] [runProvisioning-crm-postconfigure] [tid: 94] [ecid:0000KAmi6CyEGRK5IVd9if1Ib5CC00001L,0] Starting serversoa_server1 .....................................................................................................................................................................................................................................................................Exception while starting server 'soa_server1'[[No stack trace available.
This Exception occurred at Mon Dec 02 01:51:38 PST 2013.
weblogic.management.scripting.ScriptException: Error occured while performing start : Server with name soa_server1 failed to be started
Problem invoking WLST - Traceback (innermost last):
File "/scratch/aime/FINS_IP/apptop/provisioning/debug/<host_name>/postconfigure_02_12_13_01_29_43/nodeManagerStartServer_CRMDomain4661859561054870083.py", line 329, in ?
File "/scratch/aime/FINS_IP/apptop/provisioning/debug/<host_name>/postconfigure_02_12_13_01_29_43/nodeManagerStartServer_CRMDomain4661859561054870083.py", line 187, in startServer
File "<iostream>", line 1279, in start
File "<iostream>", line 1847, in raiseWLSTException
WLSTException: Error occured while performing start : Error starting the server : Error occured while performing start : Server with name soa_server1 failed to be started
Use dumpStack() to view the full stacktrace
This can be because the ODI server and the domain admin server are on different hosts
and
either the ODI server is the last Oracle Fusion Middleware server to be configured
or
the ODI server is isolated on a separate host when the Advanced Topology option is selected on the Domain Topology Configuration screenduring the creation of the Oracle Fusion Applications provisioning response file to specify the placement of each servers on the provisioning hosts.
Resolution
Follow these steps to resolve the issue:
Run cleanup-postconfigure and then restore-postconfigure.
Log in to the respective domain admin console for which server startup failed.
Fix the Node Manager details of the host being used by the ODI server. Change the Listen Address of the host to the ODI server address using the following steps:
Select Environment -> Machines from the Domain Structure menu.
Click the machine name link.
In the Settings for <a machine name> window, select Configuration -> Node Manager.
Update the Listen Address by entering the host name of the ODI server address. Click Lock & Edit located in the top left corner of the Change Center panel.
Enter the host address, click Save and then click Release Configuration.
Continue rerunning the postconfigure phase.
The following section describes the steps to troubleshoot startup phase errors.
Startup Phase Failed with “EQA-15011: The plug-in manager raised an unexpected error”
If you enable local application configuration for a multihost environment, errors may occur during the startup phase with the following messages in the provisioning log:
(<APPLICATIONS_BASE>/logs/provisioning/<host name>/runProvisioning-startup.log)
The errors are due to the timing between the startup of managed servers that happened before the time required for permission grants to propagate from the policy store to the affected managed servers.
Plug-in manager error
[2013-07-18 15:04:53 PDT] Orchestration: private-configure-webcenter-ses-task synchronized - BUILD_ERROR : <FRAMEWORK_LOCATION>/provisioning/provisioning-build/webcenter-build.xml:2570: Unable to create SES source object with name: SecureWebCenterRssSource due to error: EQA-15011: The plug-in manager raised an unexpected error. [2013-07-18 15:04:54 PDT] Orchestration: private-startup synchronized - BUILD_ERROR : <FRAMEWORK_LOCATION>/provisioning/provisioning-build/orchestration-build.xml:2152: The following error occurred while executing this line: <FRAMEWORK_LOCATION>/provisioning/provisioning-build/webcenter-build.xml:2435: Process execution failed with return code: 1. Check the logs for more information.
SES Error
[2013-07-23T23:20:46.182-07:00] [runProvisioning-startup] [NOTIFICATION] [FAPROV-01161] [runProvisioning-startup] [tid: 10] [ecid: 0000K0FLNcF3Z7P_Idt1if1Hvr8A000000,0] [arg: <FRAMEWORK_LOCATION>/provisioning/provlocks/ses.lck] [arg: 23 Jul 2013 23:20:46 PDT] Released lock on file <FRAMEWORK_LOCATION>/provisioning/provlocks/ses.lck at time 23 Jul 2013 23:20:46 PDT [2013-07-23T23:20:46.266-07:00] [runProvisioning-startup] [NOTIFICATION] [] [runProvisioning-startup] [tid: 10] [ecid: 0000K0FLNcF3Z7P_Idt1if1Hvr8A000000,0] [logStatus] STATE=BUILD_ERROR!TIMESTAMP=2013-07-23 23:20:46 PDT!TARGET=private-configure-webcenter-ses-task!CATEGORY=BUILD_ERROR!DOMAIN=NONE!HOSTNAME=adcdav01!PRODUCTFAMILY=orchestration!PRODUCT=orchestration!TASK=synchronized!TASKID=orchestration.orchestration.BUILD_ERRO. private-configure-webcenter-ses-task.synchronized!MESSAGE=<FRAMEWORK_LOCATION>/provisioning/provisioning-build/webcenter-build.xml:2570: Unable to create SES source object with name: SecureWebCenterRssSource due to error: EQA-15011: The plug-in manager raised an unexpected error.!DETAIL=Unable to create SES source object with name: SecureWebCenterRssSource due to error: EQA-15011: The plug-in manager raised an unexpected error.!BUILDFILE=<FRAMEWORK_LOCATION>/provisioning/provisioning-build/webcenter-build.xml!LINENUMBER=2445! [2013-07-23T23:20:46.266-07:00] [runProvisioning-startup] [ERROR] [FAPROV-00298] [runProvisioning-startup] [tid: 10] [ecid: 0000K0FLNcF3Z7P_Idt1if1Hvr8A000000,0] An Error Occured:[[<FRAMEWORK_LOCATION>/provisioning/provisioning-build/webcenter-build.xml:2570: Unable to create SES source object with name: SecureWebCenterRssSource due to error: EQA-15011: The plug-in manager raised an unexpected error.
Resolution
Wait for 10 minutes after the error occurs.
Restart these managed servers in the Common Domain from the WebLogic Admin Console: UCM_server1 (in UCMCluster), WC_Spaces (FS_SPACECluster), and search_server1 (SESCluster) using the Node Manager credentials that you entered in the provisioning response file. You can find the URL for the Command Domain WebLogic Admin Console (http://host:port/console) in the provisioning summary file (the file name is provisioning.summary) located in the same place as the provisioning response file. The following example shows what the entries in the provisioning summary might look like:
Common Domain Admin Server Host: host_name Managed Server Port: port_number Admin Console http://<host>:<port>/console Enterprise Manager Welcome Page http://<host>:<port>/em
After all three affected managed servers are restarted, rerun the provisioning startup phase
The following section describes the steps to troubleshoot Validate phase errors.
During the Validate phase, the following errors might be encountered:
INFO: WSM-09004 Component auditing cannot be initialized error and
SEVERE: Error while invoking endpoint "http://<host>:<7004>/topologyManagerService/ProvisionConfigurationService" from client.
The log messages in the runProvisioning-fs-postconfigure.log files are as follows:
[2012-12-11T01:24:08.130-08:00] [runProvisioning-fs-postconfigure] [TRACE] [] [runProvisioning-fs-postconfigure] [tid: 402] [ecid: 0000Ji9Ho1B3r2GLIyT4if1GljTr000069,0] [SRC_CLASS: oracle.apps.fnd.provisioning.ant.taskdefs.util.logger.ProvisioningLogger] [SRC_METHOD: log] Attempt 1 of 20 for importProvisionDeployedInfo
[2012-12-11T01:24:10.767-08:00] [runProvisioning-fs-postconfigure] [ERROR] [] [runProvisioning-fs-postconfigure] [tid: 403] [ecid: 0000Ji9Hp6F3r2GLIyT4if1GljTr00006A,0] INFO: WSM-09004 Component auditing cannot be initialized.
[2012-12-11T01:24:10.772-08:00] [runProvisioning-fs-postconfigure] [ERROR] [] [runProvisioning-fs-postconfigure] [tid: 403] [ecid: 0000Ji9Hp6F3r2GLIyT4if1GljTr00006A,0] INFO: WSM-09004 Component auditing cannot be initialized.
[2012-12-11T01:24:11.676-08:00] [runProvisioning-fs-postconfigure] [ERROR] [] [runProvisioning-fs-postconfigure] [tid: 403] [ecid: 0000Ji9Hp6F3r2GLIyT4if1GljTr00006A,0] SEVERE: Error while invoking endpoint "http://<host>:<port>/topologyManagerService/ProvisionConfigurationService" from client.
This error message occurs when the validation is initiated by the provisioning framework before the Oracle Web Service Manager (OWSM) is fully initialized. The provisioning framework will retry up to a predetermined number of times, therefore such error messages can be ignored. However, if validation fails after 20 attempts, investigate the issue.
To prepare the Oracle Fusion Applications installation for functional implementation, perform several configuration tasks, some of which are common for all Oracle Fusion Applications or multiple components of the Oracle Fusion Applications suite, and some are specific to the individual components.