14 Troubleshooting Oracle Fusion Applications Using Incidents, Logs, QuickTrace, and Diagnostic Tests

This chapter discusses how to use incidents, log settings, log files, QuickTrace, and diagnostic tests to help diagnose and resolve problems with Oracle Fusion Applications.

This chapter also discusses how incidents are related to log files, QuickTrace output, and diagnostic tests.

This chapter contains the following topics:

• Introduction to Troubleshooting Using Incidents, Logs, QuickTrace, and Diagnostic Tests

• Configuring Log, QuickTrace, and Incident Settings for Troubleshooting

• Configuring the Diagnostic Testing Framework for Troubleshooting

• Investigating, Reporting, and Solving a Problem

• Troubleshooting Using Log Settings

• Troubleshooting Using the Diagnostic Testing Framework

For information about using log settings, log files, and diagnostic tests to monitor normal operations for Oracle Fusion Applications, and for information about how log files and diagnostic tests are related to each other, see Chapter 13. For more information about the individual diagnostic tests that are provided with this release, see the Oracle Fusion Applications Common User Guide in Oracle Fusion Applications Help.

14.1 Introduction to Troubleshooting Using Incidents, Logs, QuickTrace, and Diagnostic Tests

Incidents are collections of information about problematic system events. Incidents can be created automatically or manually. An automatically created incident normally contains information about the state of a particular Oracle WebLogic Server domain at the time when a specific problem occurred. Monitoring for and responding to incidents is a standard activity for help desk staff who support Oracle Fusion Applications in organizations that use Information Technology Infrastructure Library (ITIL) best practices.

This section contains the following topics:

• Relationships Between Incidents, QuickTrace Dumps, Log Files, and Diagnostic Tests

• Introduction to the Oracle Fusion Middleware Diagnostic Framework

• Standard Incident Administration Tasks and Tools

You can use incidents to diagnose and resolve problems, or to supply your help desk personnel or Oracle Support personnel with information about more complex problems. Oracle developers set up error messages in such a way that incidents are created automatically when certain high-priority error conditions occur. You can also create incidents manually.

Automatically created Oracle Fusion Applications incidents normally include diagnostic dump files. Dump files are provided in order to assist you in resolving incidents for your organization and to assist Oracle representatives in providing you with effective support when needed. The types of dump files provided depend on the nature of the incident. Some incidents include Oracle Fusion Middleware dump files such as Oracle WebLogic Server diagnostic images, Java Virtual Machine thread dumps, and Dynamic Monitoring Service metric dumps. For more information about Oracle Fusion Middleware dumps, see the "Diagnosing Problems" chapter in the Oracle Fusion Middleware Administrator's Guide.

Note:

Each Oracle Fusion Applications module is written in one of the following code languages: Java, SOA, PL/SQL, or C. The amount and type of information automatically included with an incident may vary depending on the coding language of the relevant application module.

In general, you can administer Oracle Fusion Applications without knowing which programming language implements particular modules. However, in order to monitor and diagnose all types of incidents successfully, it is important to become familiar with all of the incident configuration settings and the types of diagnostic dump files that are automatically provided for different kinds of incidents.

For more information about incident configuration settings for Oracle Fusion Applications, see Section 14.2.4.

Oracle Fusion Applications incidents may also contain diagnostic log excerpts and diagnostic test results that are specific to Oracle Fusion Applications. Before you transmit incident information to Oracle Support personnel, you can add more information (such as screen shots or more log file information) to incidents that were created either automatically or manually.

14.1.1 Relationships Between Incidents, QuickTrace Dumps, Log Files, and Diagnostic Tests

Incidents are designed to work along with other diagnostic features in your Oracle Fusion applications:

• Error-Handling: The Oracle Fusion Applications code that handles errors is designed so that errors that need quick attention automatically create incidents.

• Diagnostic Tests: Oracle developers create tests that you can use to help diagnose and resolve Oracle Fusion application problems. A diagnostic test may or may not be associated with a particular error message. If Java or SOA code in an Oracle Fusion application handles a particular error in a way that triggers the creation of an incident, then any diagnostic tests that are associated with the error message for the incident run automatically. The test results are associated with the incident and the identity of the user who received the error message is recorded.

• Log files: By default, whenever an Oracle Fusion Applications incident is created automatically, an INCIDENT_ERROR level entry is recorded in the standard Oracle Fusion Applications log file and a copy of that log file is included with the incident. If the incident occurs in Oracle Fusion Middleware code two different kinds of log information are automatically included with the incident: a dump file containing all log entries for the relevant execution context ID (ECID) and either a complete log file or a log file excerpt starting five minutes prior to the incident.

  For some incidents, you may find all the information that you need to resolve the problem in the log or dump files automatically included with the incident. In other cases, you may need to configure your system to log more detailed information, attempt to reproduce the problem, and then use the more detailed information in the log to diagnose the problem. If you work with Oracle Support to resolve a problem, log file excerpts are among the kinds of information that you can add to incidents before sending them to Oracle.

• QuickTrace: If you need to troubleshoot a problem that has generated an incident, and if the information in the relevant Oracle Fusion application log file is insufficient to resolve the problem, then a QuickTrace dump file included with the incident information may supply you with the additional information you need. If so, you can avoid increasing the amount of information to be logged and then attempting to reproduce the problem. (In general, you should increase the level of detail that you collect in your Oracle Fusion Applications logs only when absolutely necessary, as collecting additional information may decrease system performance.)

  By default, QuickTrace continuously records a specified level of log detail in an area of memory, generally at a more granular level of detail than is recorded in Oracle Fusion application log files. The memory that QuickTrace uses is recycled on an ongoing basis, with the oldest information being overwritten first. Because QuickTrace does not format information and because it writes to memory instead of to a log file, it can gather operational information continuously without significantly affecting system performance. The information that QuickTrace stores in memory is written to disk only when an incident occurs or when an administrator manually dumps the contents of a QuickTrace buffer.

  Note:

  The same commands in Oracle Fusion Applications code cause messages to be logged both in Oracle Fusion application diagnostic log files and in QuickTrace dumps. However, your system may gather different amounts of detail for the two kinds of output. At default setting levels, QuickTrace stores much more detailed information than diagnostic log files do.

• Version Information: If you request assistance from Oracle Support to resolve an incident, you may be asked to provide information about the Oracle product versions you are running along with the incident information. In your Oracle Fusion applications, you can obtain some product version information by selecting About Applications from the Help menu. Additional version information can be obtained from Remote Diagnostic Agent (RDA) reports. For more information about RDA, see the "Generating an RDA Report" section in the Oracle Fusion Middleware Administrator's Guide.

You may also find it helpful to view the performance metrics provided by the Dynamic Monitoring Service (DMS). For more information, see the chapter about the Oracle Dynamic Monitoring Service in the Oracle Fusion Middleware Performance and Tuning Guide.

14.1.2 Introduction to the Oracle Fusion Middleware Diagnostic Framework

The Oracle Fusion Middleware Diagnostic Framework provides most of the technologies for creating and handling incidents for Oracle Fusion Applications. For information about the Oracle Fusion Middleware Diagnostic Framework, see the "Understanding the Diagnostic Framework" section in the Oracle Fusion Middleware Administrator's Guide.

14.1.3 Standard Incident Administration Tasks and Tools

The following administrative tasks are part of administering incidents:

• Adjusting configuration settings for incidents

• Using Oracle Enterprise Manager Fusion Applications Control (Fusion Applications Control) to monitor for incidents that are created automatically

• Creating incidents manually when needed

• Using incident information to resolve problems

• Adding additional diagnostic information (such as log file excerpts or screen shots) to existing incidents

• Packaging incident information and sending it to Oracle Support as part of obtaining assistance in resolving problems

The standard tools that you use to complete these tasks include:

• Fusion Applications Control user interface (for configuring incident settings)

• Log Viewer (part of Application Server Control, for viewing log files that you may add to incidents)

• adrci (Automatic Diagnostic Repository Command Interpreter, for viewing and packaging incidents, provided with each instance of the Automatic Diagnostic Repository (ADR) in Oracle Fusion Applications, Oracle Fusion Middleware, and Oracle Database)

  For more information about using adrci, see Section 14.4.2, the "ADCRI Command-Line Utility" section in the Oracle Fusion Middleware Administrator's Guide, and the "ADRCI: ADR Command Interpreter" chapter of Oracle Database Utilities.

• Oracle WebLogic Scripting Tool (WLST) commands related to incidents provided by the Oracle Fusion Middleware Diagnostic Framework.

  For more information about these commands, see the "WLST Commands for Diagnostic Framework" section in the Oracle Fusion Middleware Administrator's Guide.

• SQL Trace (for assessing the efficiency of the SQL statements that Oracle Fusion applications are running)

  For more information about SQL Trace, see Section 12.2.4.6.

14.2 Configuring Log, QuickTrace, and Incident Settings for Troubleshooting

You may find it helpful to change log profile options or QuickTrace properties to gather more information during troubleshooting.

Ordinarily, the default configuration settings for incidents are appropriate for both normal operations and troubleshooting. It is possible to disable incident creation by using configuration settings, but it is recommended that you use the default incident configuration values unless Oracle Support specifically recommends changes to these settings.

This section contains the following topics:

• Assisting Users in Gathering Data Using Troubleshooting Options

• Configuring the Oracle Fusion Middleware Diagnostic Framework

• Precedence and Log Settings for Troubleshooting

• Default System Settings for Incident Creation and QuickTrace

• Adjusting Incident Settings for Troubleshooting

• When Changes to Incident Configuration Settings Take Effect

• Adjusting QuickTrace Configuration Settings for Troubleshooting

• Adjusting Standard Log Levels for Troubleshooting

• Adjusting Log Settings for Oracle Fusion Incentive Compensation Batch Jobs for Troubleshooting

• Adjusting Oracle Fusion General Ledger Log Settings for Troubleshooting

• Adjusting Oracle Fusion Receivables AutoInvoice Log Settings for Troubleshooting

• Disabling Logging of a Particular Message

• Logging More Detailed Information for a Particular Code Module

• Configuring and Using Profile Options for Troubleshooting

14.2.1 Assisting Users in Gathering Data Using Troubleshooting Options

Oracle Fusion Applications provides a Troubleshooting Options dialog box that you can display by choosing the Troubleshooting Options command in the Oracle Fusion Applications Help > Troubleshooting menu. The Troubleshooting Options dialog box provides ways to adjust the following kinds of settings for the current Oracle Fusion Applications user:

• Database trace (optionally capturing bind variables or wait events)

  For more information about Database Trace, see the chapter about using application tracing tools in the Oracle Database Performance Tuning Guide.

• PL/SQL profiler

  For more information about PL/SQL profiler, see the chapter about the PL/SQL hierarchical profiler in the Oracle Database Advanced Application Developer's Guide.

• Applications logging Severity Level

• Applications logging Modules

However, the Help > Troubleshooting menu displays the Troubleshooting Options command only for end users who have a job role that is mapped to the following three duty roles:

• Supportability Level Management Duty (CRM) (FND_SET_SUPPORTABILITY_LEVEL_DUTY_CRM)

• Supportability Level Management Duty (FSCM) (FND_SUPPORTABILITY_LEVEL_MANAGEMENT_DUTY_FSCM)

• Supportability Level Management Duty (HCM) (FND_SUPPORTABILITY_LEVEL_MANAGEMENT_DUTY_HCM)

By default, only the following job roles are matched to the Supportability Level Management Duty duty roles:

• Application Administrator (FND_APPLICATION_ADMINISTRATOR_JOB)

• Application Diagnostics Administrator (FND_DIAG_ADMINISTRATOR_JOB)

These two job roles have broad administrative privileges, so it is recommended that you assign them only to administrative personnel. To give other users access to the Troubleshooting Options dialog box, it is recommended that you find or create a job role that is specifically intended for troubleshooting and that is mapped to the three Supportability Level Management Duty duty roles. Once the troubleshooting job role is available, you can assign users to that job role as needed.

To grant a user access to the Troubleshooting Options command and dialog box and assist the user to gather data for troubleshooting:

1. Use Oracle Identity Manager to find or create a job role (also called an external role) that you can assign to users who need access to the Troubleshooting Options command and dialog box.

   For more information, see the "Creating Roles" section in the Oracle Fusion Middleware User's Guide for Oracle Identity Manager.

2. If you created a job role in Step 1, use Oracle Authorization Policy Manager to map the job role to the following three duty roles:

   • Supportability Level Management Duty (CRM) (FND_SET_SUPPORTABILITY_LEVEL_DUTY_CRM)

   • Supportability Level Management Duty (FSCM) (FND_SUPPORTABILITY_LEVEL_MANAGEMENT_DUTY_FSCM)

   • Supportability Level Management Duty (HCM) (FND_SUPPORTABILITY_LEVEL_MANAGEMENT_DUTY_HCM)

   Note:

   To make sure that the job role you create can be used for troubleshooting any Oracle Fusion application, be sure to map the job role to all three of the Supportability Level Management Duty duty roles. Each of these duty roles provides the needed functionality for a different application stripe, where an application stripe is a subset of policies in the policy store that is used by a particular application or group of applications.

   For more information about mapping job roles to duty roles, see the "Mapping External Roles to an Application Role" section in the Oracle Fusion Middleware Oracle Authorization Policy Manager Administrator's Guide (Oracle Fusion Applications Edition). A job role corresponds to an external role, and a duty role corresponds to an application role.

3. For any user who currently needs access to the Troubleshooting Options dialog box, use Oracle Identity Manager to assign the job role that you found or created in Step 1 to the user.

   For more information, see the "Assigning Members to a Role" section or the "Adding and Removing Roles" section in the Oracle Fusion Middleware User's Guide for Oracle Identity Manager.

4. To make sure that the user has access to the new job role assignment, direct the user to log out from their Oracle Fusion application and log back in.

5. Direct the user to choose Troubleshooting Options from the Help > Troubleshooting menu and to modify Troubleshooting Options settings as needed for the specific problem you are troubleshooting:

   Note:

   If the user does not see Troubleshooting Options in the Help > Troubleshooting menu and if you performed Step 2 of this procedure within the last few minutes, direct the user to wait a few more minutes and then log out and log in again. When you map a job role to one or more duty roles, the change may take effect a few minutes later, after Oracle Authorization Policy Manager refreshes the server cache.

   1. If you want to enable all of the available troubleshooting options, direct the user to select the Enable all checkbox.

   2. If you want to enable Database Trace, direct the user to select the Database trace checkbox.

      For more information about Database Trace, including the options of capturing bind variables or wait events, see the chapter about application tracing tools in the Oracle Database Performance Tuning Guide.

   3. If you want to enable the PL/SQL profiler, direct the user to select PL/SQL profiler.

      For more information, see the section about using the PL/SQL hierarchical profiler in the Oracle Database Advanced Application Developer's Guide.

   4. If you want to log information for the user's actions at a level of detail that is different from the logging done for other users at the same site, direct the user to select the Applications logging checkbox and to select the appropriate level of detail from the Severity Level drop-down list.

      This user action is equivalent to an administrator setting the value of the AFLOG_LEVEL profile option for that user. For more information about the available logging severity levels, see Section 13.7.3.

   5. If you want to limit the logging for the user's actions to operations that use one or more particular code modules, and if you know the name of those code modules, direct the user to enter the module names in the Modules field, separating the names using commas without spaces.

      The user can use the % wildcard character to represent part of a module name or to represent all module names.

   6. Direct the user to click OK.

      Note:

      Users who change the Applications logging Severity Level may need to log out from their Oracle Fusion application and log in again in order to have the changes take effect. It is recommended that you direct the user to do so at this point in the process.

6. Direct the user to try to reproduce the problem.

7. Verify that the additional information you wanted was collected.

8. Unless the user needs ongoing access to the Troubleshooting Options dialog box, use Oracle Identity Manager to revoke the user's assignment to the job role that you found or created in Step 1.

14.2.2 Configuring the Oracle Fusion Middleware Diagnostic Framework

As part of the infrastructure for creating and handling incidents, the Oracle Fusion Middleware Diagnostics Framework includes settings that you can configure to affect functionality areas such as the following:

• Whether multiple incidents that originate from Java or PL/SQL code can be created for the same problem within a particular time period, and, if so, how many (flood control)

• The total amount of space allocated for storage of information about incidents that originate from Java or PL/SQL code (older incidents are purged automatically when the allocated space limit is reached)

For more information about these Oracle Fusion Middleware Diagnostic Framework settings, see the "Configuring the Diagnostic Framework" section in the Oracle Fusion Middleware Administrator's Guide. For information about flood control and disk space management for incidents that originate from PL/SQL code, see Table 14-1.

14.2.3 Precedence and Log Settings for Troubleshooting

During troubleshooting in a production environment, you will normally use profile options to increase or decrease the amount of information gathered in standard Oracle Fusion Applications log files. Where feasible, you will change profile option values at the USER level, rather than at the SITE level. This allows you to log more detailed information for a specific user who attempts to reproduce a problem, without cluttering your log file with unnecessarily detailed information for other users.

For Oracle Fusion applications that are written in Java or SOA, it is also possible for the odlLevel setting for the oracle.apps logger in the logging.xml configuration file to affect the level of detail that is logged. If both the odlLevel setting and the AFLOG_LEVEL profile option have defined values, then the minimum of those two values takes precedence.

Note:

Unless Oracle Support advises you to change the value of the setting, it is strongly recommended that you keep your odlLevel setting for the oracle.apps logger set to a value of All.

14.2.4 Default System Settings for Incident Creation and QuickTrace

The default profile option configuration settings for incident creation and QuickTrace profile options are shown in Table 14-1. For information about additional QuickTrace properties, see Section 14.2.7.

Table 14-1 Profile Options for Oracle Fusion Applications Incidents and QuickTrace

Profile Option Name (and Display Names)	Environment	Description	Possible Values or Example	Applicable Profile Hierarchy Levels	Default Value
AFLOG_ INCIDENT_ ENABLED (Incident Enabled)	Java, PL/SQL, C, and SOA	Enables or disables incident creation for Oracle Fusion Applications. (This profile option does not affect Oracle Fusion Middleware incident creation functionality.) A value of Y allows incident creation at runtime. A value of N prevents incident creation. This profile option is available for SITE but is not available for USER. AFLOG_INCIDENT_ENABLED is a new profile that has been introduced in Fusion to support application incident creation.	Y, N	Site	Y
AFLOG_ PLSQL_FLOOD_ CONTROL_ ENABLED (FND: Incident Flood Control Enabled)	PL/SQL	Determines whether to restrict the collection of diagnostics if multiple PL/SQL incidents occur within a set time period for the same problem. Specify Y to restrict diagnostic collection and N to leave it unrestricted.	Y, N	Site	Y
AFLOG_ PLSQL_FLOOD_ CONTROL_INC_ COUNT (FND: Total number of incidents for problem key)	PL/SQL	When AFLOG_PLSQL_FLOOD_ CONTROL_ENABLED is set to Y, the value of this setting determines the number of PL/SQL incidents with the same problem key for which diagnostics are collected during a set time period. If the number of incidents with the same problem key exceeds the value of this setting during the time period specified by AFLOG_PLSQL_FLOOD_CONTROL_ INC_TIMEPERIOD, an incident is created, but no diagnostics are captured	Positive integers	Site	5
AFLOG_ PLSQL_FLOOD_ CONTROL_INC_ TIMEPERIOD (FND: Flood control time period)	PL/SQL	When AFLOG_PLSQL_FLOOD_ CONTROL_ENABLED is set to Y, the value of this setting specifies the time period in minutes during which diagnostics are not collected if the number of PL/SQL incidents that occur for the same problem exceeds the value of AFLOG_PLSQL_FLOOD_ CONTROL_INC_COUNT.	Positive integers	Site	60
AFLOG_ PLSQL_MAX_ TOTAL_INC_ SIZE (FND: Total size of incident files)	PL/SQL	Sets the amount of disk space in megabytes that is allocated for PL/SQL incident data storage. When incidents use more than this amount of disk space, the oldest incident that is no longer needed is automatically purged. Whether or not an older incident is still needed is determined by checking the incident's status in the Automatic Diagnostic Repository (ADR). If the status of the incident is 3, or KEEP, or tracking (in the Automatic Diagnostic Repository Command Interpreter, adrci), then that incident is not purged. If the allocated disk space is full and if all stored incidents are still needed, a new, empty incident directory called inc_incident_ID_number is created, and log file messages may refer to that location, but diagnostic log excerpts are not saved to that incident directory.	Positive integers	Site	500 (MB)
AFLOG_ QUICKTRACE_ ENABLED (QuickTrace Enabled)	Java and SOA	Enables quick trace to occur at runtime.	Y, N	Site	Y
AFLOG_ QUICKTRACE_ LEVEL (QuickTrace Level)	Java and SOA	Specifies the minimum level for tracing. This setting is similar to the AFLOG_LEVEL profile option for standard Oracle Fusion Applications logging to log files, but the two settings operate independently.	1000 (SEVERE), 900 (WARNING), 800(INFO), 700 (CONFIG), 500 (FINE), 400 (FINER), 300 (FINEST) and 0 (OFF)	Site	500 (FINE)

14.2.5 Adjusting Incident Settings for Troubleshooting

By default, incident creation functionality for Oracle Fusion Applications is enabled, and it should ordinarily remain enabled. However, if Oracle Support directs you to disable incident creation, you can use the AFLOG_INCIDENT_ENABLED profile option to do so. You can use the Manage Administrator Profile Values task in the Setup and Maintenance work area to set the value of this profile option. For more information about this setting, see Section 14.2.4.

Incident creation functionality is also enabled by default for the Oracle Fusion Middleware Diagnostic framework. In addition, various Oracle Fusion Middleware profile options govern the details of how incidents are collected and stored. In general, troubleshooting will not require you to change these settings. For more information, see the "Configuring the Diagnostic Framework" section in the Oracle Fusion Middleware Administrator's Guide.

14.2.6 When Changes to Incident Configuration Settings Take Effect

Changes to incident profile options ordinarily take effect as follows:

• User Sessions: Changes to site level incident profile options take effect for any user session that is started after the setting has been changed. Changes to site level log file profile options do not affect any user sessions that are already in progress when the change is made.

• PL/SQL and C processes such as scheduled jobs: Changes to site level incident profile options take effect for any C or PL/SQL processes that are launched after the setting has been changed. Incident profile option changes do not affect C or PL/SQL processes that are already running.

14.2.7 Adjusting QuickTrace Configuration Settings for Troubleshooting

By default, QuickTrace functionality for Oracle Fusion Applications is enabled at the FINE logging level. QuickTrace should ordinarily remain enabled. However, if Oracle Support directs you to disable QuickTrace, or if you want to change the amount or organization of the information that QuickTrace collects, you can adjust the values of the following profile options either by using Fusion Applications Control or by using the Manage Administrator Profile Values task in the Setup and Maintenance work area:

• QuickTrace Enabled (AFLOG_QUICKTRACE_ENABLED profile option): Governs whether or not QuickTrace gathers any information.

• QuickTrace Level (AFLOG_QUICKTRACE_LEVEL profile option): Specifies the level of detail at which QuickTrace gathers information.

Note:

Because the QuickTrace Enabled and QuickTrace Level profile options apply at the site level, either all of the site's Managed Servers capture QuickTrace information in memory buffers or none of the Managed Servers do so. Similarly, the QuickTrace Level value sets the level of detail that QuickTrace gathers for all of the site's Managed Servers.

For more information about these settings, see Section 14.2.4.

In addition, Table 14-2 shows QuickTrace property settings that you can change using Fusion Applications Control.

Table 14-2 QuickTrace Properties

Property Name	Description	Server Restart Required	Default
Buffer Size (bufferSize)	Approximate size in bytes of each circular QuickTrace buffer. Each buffer's actual memory consumption may be less but not more than the specified value. By default, each WebLogic Server has a single COMMON QuickTrace buffer for storing multiple users' log records in memory. If you want store particular users' log records in individual buffers (one buffer for each user you specify), then you can allocate additional buffers of the same size by setting Enable User Buffer to true, restarting the server, and setting the value of User Names for Reserve Buffer to a comma-separated list of those users.	Yes	5242880
Enable User Buffer (enableUserBuffer)	When this property is set to false, each QuickTrace handler uses a single COMMON buffer to cache log messages for all users. When this property is set to true, each QuickTrace handler maintains an individual buffer for each user who is listed in the User Names for Reserve Buffer (reserveBufferUserIDs) property. Log messages for users who are not listed in that property are cached in the COMMON buffer. To determine in which buffer to place a particular log message, QuickTrace examines the log message for an application User Name value. If the log message does not include a User Name value, QuickTrace examines it for a User ID value provided by the server. If the log message does not include either a User Name or a User ID, then QuickTrace caches the message in the COMMON buffer.	Yes	false
User Names for Reserve Buffer (reserveBufferUserIDs)	Holds a list of application User Name or server User ID values separated by comma. When the Enable User Buffer (enableUserBuffer) property is set to true, for each listed application User Name or server User ID, any log messages associated with that user are cached in an individual QuickTrace user buffer. Log messages that are associated with other users are cached in the COMMON buffer. When the Enable User Buffer (enableUserBuffer) property is set to false, this property has no effect.	No	null

You can also change the properties in Table 14-2 by using the Oracle WebLogic Scripting Tool (WLST) configureLogHandler command. For more information, see the "configureLogHandler" section in the Oracle Fusion Middleware WebLogic Scripting Tool Command Reference.

To change QuickTrace profile option and property settings using Fusion Applications Control:

1. From the navigation pane, select a product family target such as Financials.

2. In the context pane, from the target type dropdown menu, choose Logs > Log Configuration.

3. In the Logging Profile Configuration dialog box, click the Site-Level tab.

4. Expand the Advanced Configuration area of the dialog box.

5. Expand the Java Settings portion of the Advanced Configuration area.

6. Expand the Other QuickTrace Settings portion of the Java Settings area.

7. In the Java Settings portion of the Advanced Configuration area, change the QuickTrace profile settings, as needed:

   1. To disable QuickTrace, clear the QuickTrace Enabled checkbox. To enable QuickTrace, make sure the QuickTrace Enabled checkbox is selected.

   2. To change the amount of information QuickTrace gathers, select a logging level from the QuickTrace Level drop-down list.

8. In the Other QuickTrace Settings portion of the Java Settings area, change the QuickTrace property settings, as needed:

   1. If you intend to use any QuickTrace property value that is not the default value listed in Table 14-2, select the Override Settings checkbox.

   2. In the Buffer Size field, enter the desired QuickTrace buffer size in bytes.

      Note:

      If you increase the value of the Buffer Size setting, QuickTrace uses more memory, which may affect system performance.

   3. If you want QuickTrace to store information in separate buffers for selected individual users, select the Enable User Buffer checkbox.

   4. In the User Names for Reserve Buffer field, enter comma-separated Oracle Fusion Applications user names for all of the users for whom QuickTrace should store information in individual buffers.

      Note:

      The values in the User Names for Reserve Buffer field are used only if the Enable User Buffer checkbox is selected.

9. Click Apply.

   Note:

   Changes to QuickTrace Enabled and QuickTrace Level settings take effect immediately for the user session in which the changes were made, but the changes do not affect other user sessions that are in progress when the changes are made. To use the changed settings for a different user session, log out from your Oracle Fusion application and log back in.

   For each particular Managed Server that runs an Oracle Fusion application, changes to the QuickTrace properties Buffer Size and Enable User Buffer do not take effect until the server is restarted.

   If a value of true is already in effect for the Enable User Buffer property, then changes to the User Names for Reserve Buffer property take effect as soon as you click Apply. Otherwise, you must set the Enable User Buffer property to true and restart the server in order to have changes to the User Names for Reserve Buffer property take effect.

14.2.8 Adjusting Standard Log Levels for Troubleshooting

If you encounter a situation in which an incident does not provide enough information to allow you to resolve the problem, then you may need to increase the standard logging level for the relevant Oracle Fusion application and the application user who will attempt to reproduce the problem. For information about the available log levels, see Section 13.7.3. For information about setting the logging profile options, see Section 13.7.5.

14.2.9 Adjusting Log Settings for Oracle Fusion Incentive Compensation Batch Jobs for Troubleshooting

The logging functionality for certain Oracle Fusion Incentive Compensation batch jobs is separate from the standard logging functionality for Oracle Fusion Applications. By default, the following kinds of Oracle Fusion Incentive Compensation batch jobs do not write log entries:

• Calculation

• Classification

• Collection

• Crediting

• Rollup

For optimum performance and log file sizes, it is recommended that you use the logging functionality for these areas only when troubleshooting an existing problem.

To configure Oracle Fusion Incentive Compensation batch jobs to write nonstandard log entries:

1. Navigate to the Manage Messages task flow in the Setup and Maintenance work area.

2. In the Implementations section of the Tasks pane, click Manage Implementation Projects.

3. In the Manage Implementation Projects pane, use standard Oracle search techniques to locate the implementation project that includes Oracle Fusion Incentive Compensation, then click the project name in the search results.

4. In the Tasks Lists and Tasks table, expand the Define Incentive Compensation entry.

5. Expand the Define Incentive Compensation Shared Configuration entry.

6. Locate the Manage Incentive Compensation Profile Options row of the table and click the Go to Task icon.

7. In the Profiles section of the Tasks pane, click Manage Administrator Profile Values.

8. In the Search area of the Manage Profiles pane, select Incentive Compensation from the Application dropdown list and click Search.

9. In the Results table, locate and select the row for the CN_DEBUG profile option.

10. If the Enable Debug Mode Profile Values table contains a record for the Profile Level and Level Value for which you want to collect log entries, select the record and click Edit; otherwise, click New and fill in the appropriate values, as follows:

    • If you want to collect log entries for a specific user's Oracle Fusion Incentive Compensation batch jobs, you would edit or create a record in which Profile Level is set to User and Level Value is set to the user name of the specific user. Set Profile Value to Y.

    • If you want to collect log entries for all of the Oracle Fusion Incentive Compensation batch jobs at the site, you would edit or create a record in which Profile Level is set to Site and Level Value is blank. Set Profile Value to Y.

11. Click Save.

12. Start a new user session of Oracle Fusion Incentive Compensation and try to replicate the problem.

    For information about how to view the log entries, see Section 14.5.2.

14.2.10 Adjusting Oracle Fusion General Ledger Log Settings for Troubleshooting

In the Oracle Fusion Financials product family, some logging functionality for the Oracle Fusion General Ledger application is separate from the standard logging functionality for Oracle Fusion Applications. By default, the following kinds of Oracle Fusion General Ledger batch jobs do not write log entries:

• OpenPeriod

• Posting

• Translation

• Close Process - Create Income Statement Closing Journals

• Close Process - Create Balance Sheet Closing Journals

For optimum performance and log file sizes, it is recommended that you use the logging functionality for these areas only when troubleshooting an existing problem.

To configure the Oracle Fusion General Ledger application to write nonstandard log entries:

1. From the Setup and Maintenance work area, navigate to the Manage Administrator Profile Values screen.

2. From the Application drop-down list in the Search: Profile Option pane, select General Ledger.

3. In the Profile Display Name field, enter %Debug% and click Search.

4. In the Search Results table, select GL_DEBUG_MODE.

5. If the GL_DEBUG_MODE: Profile Values table contains a record for the Profile Level and Level Value for which you want to collect log entries, select the record and click Edit; otherwise, click New.

6. Fill in the appropriate values, as follows:

   • If you want to collect log entries for a specific user's Oracle Fusion General Ledger batch jobs, edit or create a record in which Profile Level is set to User and Level Value is set to the user name of the specific user. Set Profile Value to Y.

   • If you want to collect log entries for all of the Oracle Fusion General Ledger batch jobs at the site, edit or create a record in which Profile Level is set to Site and Level Value is blank. Set Profile Value to Y.

7. Click Save.

8. Ask an appropriate user to start a new user session of Oracle Fusion General Ledger and try to replicate the problem.

9. Inspect the relevant log file for new entries.

   If the value of the AFLOG_FILENAME profile option is set, then the value of the AFLOG_FILENAME profile option indicates the location of the log file.

   If the value of the AFLOG_FILENAME profile option is not set, then the information is logged to the location designated for Oracle Enterprise Scheduler log files. For information about viewing Oracle Fusion Middleware log files, including log files for Oracle Enterprise Scheduler, see the "Viewing and Searching Log Files" section in the Oracle Fusion Middleware Administrator's Guide.

10. To help manage the amount of disk space used by the logged information, set the value of GL_DEBUG_MODE to N or delete any row of the GL_DEBUG_MODE: Profile Values table that you no longer need.

    For more information about managing the amount of disk space used by Oracle Fusion General Ledger log entries, see Section 13.7.4.3.

14.2.11 Adjusting Oracle Fusion Receivables AutoInvoice Log Settings for Troubleshooting

In the Oracle Fusion Financials product family, logging functionality for the AutoInvoice portion of the Oracle Fusion Receivables application is separate from the standard logging functionality for Oracle Fusion Applications. The amount of information that is logged for AutoInvoice depends on the value of the Log File Message Level system option setting for each business unit. The information is placed in the standard log file for Oracle Enterprise Scheduler.

Note:

For optimum performance and log file sizes, it is recommended that you keep the Log File Message Level system option set to the lowest value that meets your everyday needs, and increase the value of the setting only when troubleshooting an existing problem.

The available values for the Log File Message Level setting are the integers 0 through 5. The value of 0 is recommended for day-to-day business needs. The value of 5 provides the maximum amount of information for troubleshooting.

Specifically, if you set Log File Message Level to a value of 0, the following kinds of information are logged:

• Product Version

• Program Name

• AutoInvoice Start Time

• AutoInvoice Concurrent Request Arguments

• Error and Warning Messages

• AutoInvoice End Time

• AutoInvoice Logical Steps

If you set Log File Message Level to a value of 1, the following additional information is gathered:

• Time-Stamped function labels

If you set Log File Message Level to a value of 2, the following additional information is gathered:

• Sizes of Allocated Arrays

• Dynamic SQL Statements

• Number of Rows Updated, Inserted and Deleted

If you set Log File Message Level to a value of 3, the following additional information is gathered:

• Method IV SQL Array Values

If you set Log File Message Level to a value of 4, the following additional information is gathered:

• Values of all variables that are used to call FND or Tax routines

If you set Log File Message Level to a value of 5, the following additional information is gathered:

• Bad lines

• Rejected lines

To change the amount of information that the Oracle Fusion Receivables application logs for the AutoInvoice functionality area in the current business unit:

1. In the Oracle Fusion Receivables application, select Setup and Maintenance from the Navigator menu.

2. Complete the following substeps to navigate to the Edit System Options screen for the Manage Receivables System Options task.

   1. Expand the Search: Tasks pane.

   2. Enter Manage Receivables System Options in the Name field and click Search.

   3. In the Manage Receivables System Options row of the Search Results table, click Go to Task.

   4. In the Search area of the Manage Receivables System Options screen, select Business Unit from the drop-down list, enter the name of the business unit for which you are troubleshooting, and then click Search.

      Alternately, you can click Search without specifying a business unit name to display a list of the business units you can choose from.

   5. In the Search Results table, click the name of the business unit for which you are troubleshooting.

3. In the Edit System Options screen, scroll down to display the AutoInvoice area of the screen, and then set Log File Message Level to the value that corresponds to the amount of information you want to gather.

   For example, to gather the maximum amount of information for troubleshooting, set the value to 5. To gather the normal amount of information for day-to-day operations, set the value to 0.

4. Click Save to put the change into effect.

   Log entries for the AutoInvoice portion of Oracle Fusion Receivables are placed in the standard log file for Oracle Enterprise Scheduler. For information about viewing Oracle Fusion Middleware log files, including log files for Oracle Enterprise Scheduler, see the "Viewing and Searching Log Files" section in the Oracle Fusion Middleware Administrator's Guide.

   Note:

   After you have gathered the detailed information that you need, remember to repeat this procedure and return the Log File Message Level setting to the value that meets your day-to-day information needs.

14.2.12 Disabling Logging of a Particular Message

You may occasionally encounter a situation in which one of your Oracle Fusion applications repeatedly logs the same error message. In this situation, it is best to discover and correct the cause of the error. However, on rare occasions, Oracle Support may recommend that you use the FND_MESSAGES.LOGGABLE_ALERTABLE attribute to prevent the repeating message from being logged while efforts are underway to resolve the issue.

Caution:

Do not change the FND_MESSAGES.LOGGABLE_ALERTABLE attribute unless Oracle specifically advises you to do so, as this involves permanent loss of data.

Do not change the FND_MESSAGES.LOGGABLE_ALERTABLE attribute directly in the database. Instead, use the Manage Messages task flow.

To disable logging for a particular message:

1. Navigate to the Manage Messages task flow in the Setup and Maintenance work area.

2. In the Message Name field in the Search pane, enter the name of the error message that you want to temporarily stop logging.

3. Select the relevant Oracle Fusion application from the Application drop-down list.

4. If you know it, select the relevant code module name from the Module drop-down list.

5. Click Search.

6. Complete the following substeps to set the value of the FND_MESSAGES.LOGGABLE_ALERTABLE attribute to N for the error message that you want to temporarily stop logging:

   1. In the search results table, select the error message that you want to temporarily stop logging.

   2. Click the edit icon in the table header.

   3. In the Edit Message screen, clear the Logging Enabled checkbox.

   4. Click Save, Save and Create Another, or Save and Close to put the change into effect immediately:

14.2.13 Logging More Detailed Information for a Particular Code Module

If you contact Oracle Support for assistance in resolving a problem, the support representative may ask you to gather more detailed log information on the operations of one or more specific code modules in your Oracle Fusion applications. You can do this by setting the value of the AFLOG_MODULE profile option to one or more module names that Oracle Support specifies.

Caution:

Unless Oracle Support specifically advises you to change the value of the AFLOG_MODULE profile option at the Site level, be sure to adjust the value only for a particular user who will attempt to reproduce the problem you are investigating. Specifying one or more particular modules at the Site level prevents logging on any modules that are not specified.

For more information about using administrative screens to set the value of this profile option either for a specific user or at the Site level, see Section 13.7.5. For more information about having a specific user set the user's own profile option using Troubleshooting Options, see Section 14.2.1.

14.2.14 Configuring and Using Profile Options for Troubleshooting

You may find it useful to adjust one or more of the following profile options during troubleshooting:

• To log more detailed information about a particular user's operations, you can create or change the value of the AFLOG_LEVEL profile option for that user. For more information about setting this profile option, see Section 13.7.

• To log more detailed information for an entire site, you can create or change the value of the AFLOG_LEVEL profile option for the site. For more information about setting this profile option, see Section 13.7.

• To specify the Oracle Fusion Applications code modules for which standard log entries will be written at the level of detail specified by the AFLOG_LEVEL profile option for a particular user, you can set the AFLOG_MODULE profile option for the same user. (To minimize the number of unrelated log entries that will be placed in the log file during troubleshooting, it is recommended that you specify a particular module whenever feasible.) For more information about setting the AFLOG_MODULE profile option, see Section 13.7.

• To turn on nonstandard logging for troubleshooting Oracle Fusion Incentive Compensation, you can set the value of the application-specific CN_DEBUG profile option to Y, either for a single user or for an entire site. For more information about setting this profile option, see Section 14.2.9.

14.3 Configuring the Diagnostic Testing Framework for Troubleshooting

When you investigate an Oracle Fusion Applications problem, you may want to assign job roles to some users, temporarily, that will allow those users to run diagnostic tests. For information about how to do this, see Section 13.9.1.

There is generally no need to change other configuration settings in order to use the Oracle Fusion Applications Diagnostic Testing Framework for troubleshooting. When Java or SOA error-handling code in an Oracle Fusion application creates an incident, the Diagnostic Testing Framework automatically runs any Oracle Fusion Applications diagnostic tests that are associated with the particular error message that is logged. There is no configuration setting in the Diagnostic Testing Framework for turning off this automatic execution of diagnostic tests.

Note:

(Oracle Fusion Applications developers use the APPS_MSG_ID diagnostic test tag to associate particular error messages with appropriate diagnostic tests. You cannot delete or modify the values of diagnostic test tags that are supplied by Oracle.

14.4 Investigating, Reporting, and Solving a Problem

From time to time, you may receive a user complaint or an automatically generated incident notification about a problem with a particular Oracle Fusion application. For any problem that does not have an immediate and obvious solution, you can diagnose and resolve the problem more effectively by becoming familiar with incident functionality and by using a systematic process for your troubleshooting process.

This section contains the following topics:

• Process for Investigating, Reporting, and Solving a Problem

• Managing Oracle Fusion Applications Problems and Incidents

• Working with Automatically Created QuickTrace Dumps

• Working with Manually Created QuickTrace Dumps

• Working with Other Diagnostic Dumps

14.4.1 Process for Investigating, Reporting, and Solving a Problem

When you become aware of a problem with a particular Oracle Fusion application, you can use a systematic process to help diagnose and address the problem. While any problem may have unique features, the information that you gather systematically is likely to be useful during the troubleshooting process. The recommended process varies somewhat depending on whether or not your Oracle Fusion Applications environment includes Oracle Enterprise Manager Cloud Control.

This section contains the following topics:

• Process for Investigating, Reporting, and Solving a Problem Using Cloud Control

• General Process for Investigating, Reporting, and Solving a Problem

14.4.1.1 Process for Investigating, Reporting, and Solving a Problem Using Cloud Control

If your Oracle Fusion Applications environment includes Cloud Control, you can use the Problem Analysis and Analyze Logs options in Cloud Control to help you focus your investigation on the log messages and metrics that are most likely to help you determine the cause of the problem. You can also use Problem Analysis to make quick assessments of the health of your system, when no problems have been reported. For more information about using Problem Analysis for general monitoring, see Chapter 10.

To help diagnose and address a problem in an Oracle Fusion Applications environment that includes Cloud Control, use the following steps:

1. If an Oracle Fusion application user reports an error message to you, make a note of any specific information that the error message may have provided, such as:

   • Incident ID

   • Server Domain

   • Server Instance

   • Application Name

2. If there is an alert for a particular metric, then use the Problem Analysis option in Cloud Control to scan performance metric charts, target status tables, and relevant log portions for indications of problem causes.

   For more information about using the Problem Analysis option, see Section 14.4.6.

3. If no incident was generated, create an incident manually.

   Using incidents for all problems helps you organize the information that you gather during the troubleshooting process. For information about creating an incident manually, see Section 14.4.2.4.

4. If the cause of the problem is still uncertain, manually perform any relevant diagnostic dumps, assigning the dump results to the incident you created.

5. Examine any incident information that has been collected, such as any QuickTrace dumps or diagnostic test results that are associated with the incident, whether those test results and dumps were collected automatically or manually.

   For more information about working with QuickTrace dumps, see Section 14.4.3.

   You can view the results of diagnostic tests that are associated with an incident either from the Incident Detail page in Support Workbench or from the Diagnostic Dashboard. For more information about viewing incident details from Support Workbench, see Section 14.4.2.1. For more information about using the Diagnostic Dashboard to view diagnostic test results, see Section 13.10.9.1.

6. If the cause of the problem is still uncertain, view recent information in the standard log files for any of the following that may be relevant:

   • Oracle Fusion Applications

   • Oracle Fusion Middleware

   • Oracle Business Intelligence

   • Oracle Enterprise Scheduler Service

   • Oracle Service-Oriented Architecture (SOA) Suite

   For more information about locating log files for Oracle Fusion Applications, see Section 13.6.4. For more information about locating log files for Oracle Fusion Middleware components, see the "Log File Location" table in the Oracle Fusion Middleware Administrator's Guide. For more information about locating log files for Oracle Enterprise Scheduler, see Section 5.17, Section 13.7.4.3, or Section 14.5.3.2.

7. If you want to run more diagnostic tests for the relevant Oracle Fusion application or attach the results of such manually-run diagnostic tests to the incident, navigate from Cloud Control to the Diagnostic Dashboard to perform those tasks.

   For information about navigating from Cloud Control to the Diagnostic Dashboard, see Section 13.9.2.2.

   For information about using the Diagnostic Dashboard to run diagnostic tests, see Section 13.10.

   For information about the individual diagnostic tests that are provided with this release, see the Oracle Fusion Applications Common User Guide in Oracle Fusion Applications Help.

   For information about adding diagnostic test results to an incident, see Section 14.6.3.

8. If necessary, change the setting that governs the amount of information to be logged, try to replicate the problem, and inspect the newly logged information.

   The steps for increasing the amount of information to be logged may vary depending on the application involved:

   • For most Oracle Fusion applications, standardized logging mechanisms place information in log files that you can view from Fusion Applications Control. You can increase the amount of information to be gathered for a particular user or for a whole site. For more information, see Section 14.2.1 and Section 13.7.5.

   • If the problem occurred in an application that uses nonstandard mechanisms for logging, such as certain Oracle Fusion Incentive Compensation batch jobs, certain Oracle Fusion General Ledger batch jobs, or the AutoInvoice portion of the Oracle Fusion Receivables application, then you may need to turn on a logging facility that is normally disabled and specify the kind of information you want to gather. For more information, see Section 13.7.6.

9. If the cause of the problem is still uncertain, click My Oracle Support Knowledge in the Problem Details or Incident Details page and log in to the My Oracle Support knowledge base to search for related information and solutions.

10. If you have not succeeded in resolving the problem, use Support Workbench to package the problem or incident along with all related dump files and diagnostic reports and then send the package to Oracle Support as a Service Request.

    For more information, see Section 14.4.2.5 and the topic "Problem Details Page" in the Cloud Control online help.

14.4.1.2 General Process for Investigating, Reporting, and Solving a Problem

To help diagnose and address a problem in an Oracle Fusion Applications environment that does not include Cloud Control, use the following steps:

1. If an Oracle Fusion application user reports an error message to you, make a note of any specific information that the error message may have provided, such as:

   • Incident ID

   • Server Domain

   • Server Instance

   • Application Name

2. If no incident was generated, create an incident manually and then manually perform any relevant diagnostic dumps, assigning the dump results to the incident you created.

   Using incidents for all problems helps you organize the information that you gather during the troubleshooting process. For information about creating an incident manually, see Section 14.4.2.6.

3. Examine any information that was collected and stored in the incident directory, such as QuickTrace dumps and the results of any diagnostic tests that may have been run, whether the tests and dumps were run automatically or were created manually.

   For more information about incident directory locations, see Section 14.4.2. For more information about how to work with QuickTrace dumps, see Section 14.4.3.

4. If the cause of the problem is still uncertain, view recent information in the standard log files for any of the following that may be relevant:

   • Oracle Fusion Applications

   • Oracle Fusion Middleware components

   • Oracle Business Intelligence

   • Oracle Enterprise Scheduler Service

   • Oracle Service-Oriented Architecture (SOA) Suite

   For more information about locating log files for Oracle Fusion Applications, see Section 13.6.4. For more information about locating log files for Oracle Fusion Middleware components, see the "Log File Location" table in the Oracle Fusion Middleware Administrator's Guide. For more information about locating log files for Oracle Enterprise Scheduler, see Section 5.17, Section 13.7.4.3, or Section 14.5.3.2.

5. Locate diagnostic tests that are related to the feature in which the problem occurred, run the diagnostic tests that seem most likely to be relevant, and inspect the results.

   For information about starting and using the Diagnostic Dashboard, see Section 13.9.2 and Section 13.10. For information about the individual diagnostic tests that are provided with this release, see the Oracle Fusion Applications Common User Guide in Oracle Fusion Applications Help.

6. If necessary, change the setting that governs the amount of information to be logged, try to replicate the problem, and inspect the newly logged information.

   The steps for increasing the amount of information to be logged may vary depending on the application involved:

   • For most Oracle Fusion applications, standardized logging mechanisms place information in log files that you can view from Fusion Applications Control. You can increase the amount of information to be gathered for a particular user or for a whole site. For more information, see Section 14.2.1 and Section 13.7.5.

   • If the problem occurred in an application that uses nonstandard mechanisms for logging, such as certain Oracle Fusion Incentive Compensation batch jobs, certain Oracle Fusion General Ledger batch jobs, or the AutoInvoice portion of the Oracle Fusion Receivables application, then you may need to turn on a logging facility that is normally disabled and specify the kind of information you want to gather. For more information, see Section 13.7.6.

7. If the cause of the problem is still uncertain, search the My Oracle Support knowledge base for related information and solutions.

8. If you have not succeeded in resolving the problem, use adrci to package the problem or incident along with all related dump files and diagnostic reports and then send the package to Oracle Support as a Service Request.

   For more information about adding files to an incident and packaging the incident into a zip file, see the "Packaging an Incident" section in the Oracle Fusion Middleware Administrator's Guide.

14.4.2 Managing Oracle Fusion Applications Problems and Incidents

All Oracle Fusion Applications customers can use the Automatic Diagnostic Repository Command Interpreter (adrci) to work with incidents. However, if your Oracle Fusion Applications environment includes Cloud Control, it is recommended that you use Support Workbench to work with incidents, as it provides more convenient ways to perform operations that otherwise require knowledge of specific (adrci) commands.

This section contains the following topics:

• Viewing Incidents Using Support Workbench

• Viewing Incidents Using the Automatic Diagnostic Repository Command Interpreter (adrci)

• Recovering from Incidents Generated During SOA Operations

• Creating Incidents Manually Using Support Workbench

• Packaging Incidents Using Support Workbench

• Creating and Packaging Incidents Using the Automatic Diagnostic Repository Command Interpreter (adrci)

Regardless of which tool you use to work with incidents, for Oracle Fusion Applications, Oracle Fusion Middleware, and Oracle Database, incidents are stored in the incident subdirectories of Automatic Diagnostic Repository (ADR) home directories. However, the location of the appropriate ADR home directory for a given incident depends on the environment from which the incident was created. In all cases, incidents are stored separately from log files.

Paths to directories that contain incident information begin with the path to the ADR base directory, where domain is the full path to an Oracle Fusion Applications domain, including the name of the physical Oracle WebLogic Server domain:

(UNIX) domain/servers/managed_server_name/adr
(Windows) domain\servers\managed_server_name\adr

The ADR base directory path is followed by the path to one of several possible incident directories:

For an incident that involves Oracle Fusion Middleware code, the incident information is stored under the following location, where, domain_name is the name of the Oracle Fusion Applications domain and server_name is the name of the server where the incident occurred:

(UNIX) ADR_base_directory/diag/ofm/domain_name/server_name/incident
(Windows) ADR_base_directory\diag\ofm\domain_name\server_name\incident

For an incident that involves Java code or a SOA composite in an Oracle Fusion application, the incident information is stored under the following location, where application_name is the name of the Oracle Fusion application that was executing when the incident occurred:

(UNIX) ADR_base_directory/diag/ofm/fusionapps/application_name/incident
(Windows) ADR_base_directory\diag\ofm\fusionapps\application_name\incident

For an incident that involves PL/SQL code in an Oracle Fusion application, the incident information is stored under the following location:

(UNIX) ADR_base_directory/diag/ofm/db_server_name/database_instance/incident
(Windows) ADR_base_directory\diag\ofm\database_server\database_instance\incident

For an incident that involves Oracle Database code, the incident information is stored under the following location:

(UNIX) ADR_base_directory/diag/rdbms/db_server_name/database_instance/incident
(Windows) ADR_base_directory\diag\rdbms\database_server\database_instance\incident

For an incident that involves C code in an Oracle Fusion application, the incident information is stored on the Java application server that runs C batch jobs using Oracle Enterprise Scheduler. On that server, the incidents are stored under the following location, where ess_hosted_app_name is the name of the Oracle Fusion application that launches batch jobs:

(UNIX) ADR_base_directory/diag/ofm/fusionapps/ess_hosted_app_name/incident
(Windows) ADR_base_directory\diag\ofm\fusionapps\ess_hosted_app_name\incident

The exact name of the application that launches batch jobs depends upon which product family is in use. Typical values include FinancialsEssApp, HcmEssApp, CrmEssApp, ProjectFinancialsEssApp, ProcurementEssApp, and ScmEssApp. You can determine the correct ess_hosted_app_name name for your Oracle Fusion application by using Fusion Applications Control for the application. In the navigation pane, if you expand the folder that is labeled with the product family name and then expand the Fusion Applications folder, the ess_hosted_app_name is the name under Fusion Applications that contains EssApp.

14.4.2.1 Viewing Incidents Using Support Workbench

If your Oracle Fusion Applications environment includes Cloud Control, it is recommended that you use Support Workbench to work with incidents, as it is simpler to use than adrci.

Note:

To use Support Workbench with Oracle Fusion Applications, you must obtain and install Cloud Control.

As part of viewing problem or incident details using Support Workbench, you must supply credentials both to connect to the Oracle WebLogic Server on which the relevant Oracle Fusion Application is running, and to connect to the host machine on which that Oracle WebLogic Server is running.

To view Oracle Fusion Applications incidents using Support Workbench:

1. In Cloud Control, select one of the following Oracle Fusion Applications target types:

   • Oracle Fusion Applications instance

   • Product family

   • Cluster application

   • J2EE application

   Note:

   You can find cluster application and J2EE targets in navigation hierarchical trees under the product family name and the Fusion Applications folder. If the tree you are viewing includes Oracle Fusion Applications instance names, you must click a product family name to navigate to a page that displays cluster application and J2EE targets in the navigation tree.

2. From the dynamic drop-down menu, choose Diagnostics > Support Workbench.

   If you selected a J2EE application as your target, the Login to Support Workbench page appears. Skip to Step 6.

   If you selected a Fusion Instance, Fusion Product Family, or Fusion Cluster Application as your target, the rollup page for Support Workbench appears, summarizing the Support Workbench problems that are related to your target.

   Note:

   If you want to return to this Support Workbench rollup page at a later time, from another Support Workbench page (such as Incident Details or the page that summarizes problems for a selected Fusion J2EE Application target), you can do so by clicking the Diagnostics link in the Related Links section of the page that corresponds to your selected target type:

   • Fusion Instance Diagnostics

   • Fusion Product Family Diagnostics

   • Fusion Cluster Application Diagnostics

3. From the View By drop-down list, select Deployed Application.

   Note:

   In order for this page to display links for the number of problems and incidents for Oracle Fusion Applications targets, you must view the data by deployed application, rather than by product. The links for number of problems and incidents that appear in the Support Workbench Infrastructure Problems and Incidents table at the bottom of this page are for Oracle WebLogic Server targets, rather than Oracle Fusion Applications targets.

4. From the View Data drop-down list, select the timeframe that contains the problem or incident that interests you.

5. In the Problems and Incidents table, click a link for the number of problems or incidents that interests you.

   The Login to Support Workbench page appears.

6. In the Host Credentials area of the page, use one of the following methods to select credentials for connecting to the machine that hosts the Oracle WebLogic Server on which the Oracle Fusion application was running when the problem or incident occurred.

   • If appropriate credentials have previously been saved as preferred credentials, select Preferred in the Credential field.

   • If appropriate credentials have previously been saved as named credentials, select Named in the Credential field and then select the specific credentials from the Credential Name drop-down list.

   • If appropriate credentials have not previously been set, select New in the Credential field and complete the following sub-steps:

     1. In the UserName and Password fields, enter credential values for connecting to the host machine.

     2. Inspect the default credential name supplied by Support Workbench in the Save As field, and modify the credential name if you prefer.

     3. If you want the credentials you are entering to be the preferred credentials, select the Set as Preferred Credentials checkbox.

        The Preferred Host Credential is checked against the host machine's usernames and passwords, but the preference is only set for the currently selected WebLogic server, even if multiple WebLogic servers run on that host machine.

     4. If you want to verify that the credentials you entered work correctly, click Test.

7. In the WebLogic Server Credentials area of the page, use one of the following methods to select credentials for connecting to the Oracle WebLogic Server where the Oracle Fusion application was running when the problem or incident occurred.

   Note:

   If you are not prompted for WebLogic Server Credentials, that indicates that the relevant Oracle WebLogic Server is currently unavailable. Some operations cannot be done when the server is unavailable, such as creating user-reported problems and performing additional diagnostic dumps. For information about restarting an Oracle WebLogic Server, see the "Starting and Stopping Servers" chapter in Oracle Fusion Middleware Managing Server Startup and Shutdown for Oracle WebLogic Server.

   • If appropriate credentials have previously been saved as preferred credentials, select Preferred in the Credential field.

   • If appropriate credentials have previously been saved as named credentials, select Named in the Credential field and then select the specific credentials from the Credential Name drop-down list.

   • If appropriate credentials have not previously been set, select New in the Credential field and complete the following sub-steps:

     1. In the UserName and Password fields, enter credential values for connecting to the Oracle WebLogic Server.

     2. Inspect the default credential name supplied by Support Workbench in the Save As field, and modify the credential name if you prefer.

     3. If you want the credentials you are entering to be the preferred credentials for the currently selected Oracle WebLogic server, select the Set as Preferred Credentials checkbox.

8. Click Continue.

   A Support Workbench page that summarizes problems related to your target now appears.

9. Click the Problems tab and select the relevant time period from the View drop-down list.

10. In the Details column of the table, click Show for the problem listing that interests you.

11. In the list of incidents for the problem you selected, click the Incidents link for the incident that occurred at the time that interests you.

    The Incident Details page appears.

12. In the Dump Files table, inspect the list of dump files that are associated with this incident, click the View Contents link for any dump file that interests you, and use standard operating system steps to indicate whether to open the file or download it.

13. If you want to create an additional dump file, click the Additional Diagnostics tab, select the kind of dump you want to create, and click Run.

14. In the Run User Action page, enter any required input parameters and choose when to create the dump, and click Submit.

15. Click OK to dismiss the confirmation message that appears when the dump operation completes.

16. To view the new dump file, repeat Step 12.

17. To create additional dump files, repeat Steps 13 through 15.

For information about using Support Workbench to package an incident and send it to Oracle Support, see Section 14.4.2.5.

14.4.2.2 Viewing Incidents Using the Automatic Diagnostic Repository Command Interpreter (adrci)

If your Oracle Fusion Applications environment does not include Cloud Control, then you must use the Automatic Diagnostic Repository Command Interpreter (adrci) to work with incidents.

To view incidents using the Automatic Diagnostic Repository Command Interpreter (adrci):

1. Log in to the server that contains the Automatic Diagnostic Repository (ADR) that was mentioned in any incident notifications you received.

   This may be a repository for either Oracle Database or Oracle Fusion Middleware. The notification may be either a message received by an Oracle Fusion Applications user or an alert email that Oracle Enterprise Manager sent to designated administrators.

2. At an operating system prompt, navigate to the directory that contains a copy of the Automatic Diagnostic Repository Command Interpreter (adrci) that is associated with the ADR home for the incident.

   • If the incident notification referred to an Automatic Diagnostic Repository for Oracle Database, navigate to DB_ORACLE_HOME/bin (for UNIX) or DB_ORACLE_HOME\bin (for Windows), where DB_ORACLE_HOME is the home directory for Oracle Database. For information about home directories for Oracle Database, see the "Overview of Oracle Database Installation" chapter in the Oracle Database Installation Guide for Microsoft Windows.

   • If the incident notification referred to an Automatic Diagnostic Repository for Oracle Fusion Middleware, navigate to WL_HOME/server/adr (for UNIX) or WL_HOME\server\adr (for Windows), where WL_HOME is the home directory for the Oracle WebLogic Server. For more information about the structure of home directories for Oracle Fusion Applications, see Section 1.2.3.

3. At the prompt, enter adrci.

4. Determine where the incident information is stored.

   1. At an adrci prompt, enter show base to display the path to the Automatic Diagnostic Repository (ADR) base directory.

   2. At an adrci prompt, enter show homes to display the path to the ADR home directory, under the ADR base directory.

   3. Make a note of the location of the incident directory, which is a subdirectory of the ADR home directory.

5. At an adrci prompt, enter show incident.

   This command lists all the incidents that have been created in the incident subdirectory of the ADR home directory.

6. If you already know the ID number of the incident you are investigating, skip to Step 8.

   Otherwise, inspect the appropriate diagnostic.log file and make a note of either the incident_ID value or the incidentCustomId value in the relevant log message.

   Note:

   Due to a technical limitation, incidents that are created from PL/SQL code log incidentCustomId values rather than incident_ID values.

   For more information about viewing diagnostic.log files, see Section 13.6.

7. If the diagnostic.log file supplies the relevant incident_ID value, skip to Step 8.

   Otherwise, enter the following command at an adrci prompt, substituting the appropriate incidentCustomId value, and inspect the output for the incident_ID value:

   show incident -mode detail -p "ERROR_ARG1='incidentCustomID'";
   

8. To view detailed information about the incident for which you now know the incident ID number, enter the following command at an adrci prompt, substituting the appropriate incident ID number:

   show incident -mode DETAIL -p "incident_id=incident_id"
   

9. To display a list of the dump files that are associated with the incident_ID incident, enter the following command at an adrci prompt:

   show tracefile -i incident_ID
   

10. To view information in a particular dump file for a particular incident, enter one of the following commands at an adrci prompt:

    • To view a copy of the information in your default editor, enter:

      show trace filename
      

    • To view the information without using an editor:

      show trace filename -term
      

14.4.2.3 Recovering from Incidents Generated During SOA Operations

In some cases in which an Oracle Fusion Applications incident is generated during an operation that involves a SOA composite, the application code may let you manually recover the operation and resume processing at the point where the incident occurred.

To determine whether you can recover or resume processing of an interrupted SOA composite operation:

1. Log on to the server where the incident was logged as a user that has the Oracle Fusion Middleware role of Administrator.

2. Navigate to the directory where the information about the incident is stored.

   For information about typical locations for incidents, see Section 14.4.2.

3. Inspect the readme.txt file in the incident directory and make a note of the composite_instance_id and composite_name values.

   For example, the readme.txt file might include the following lines:

   composite_instance_id: 10009
   composite_name: FinApInvTransactionsInvoiceApprovalComposite
   

4. In the navigation pane of Fusion Applications Control, expand the hierarchical tree as needed to locate and select the SOA composite that was mentioned in the readme.txt file for the incident.

   For example, you might expand the Farm_soa_domain_new listing, and then expand SOA, and then expand soa_infra (soa_server1) and then expand default and finally select FinApInvTransactionsInvoiceApprovalComposite.

5. If it is not already selected, click the Dashboard tab.

6. In the Recent Instances table in the content pane, click the Instance ID value that matches the composite_instance_id value that you saw in the log.

   The Flow Trace page appears.

7. If you have not already done so, use the information on the Flow Trace page to determine how to correct the problem that generated the incident, then repeat steps as needed to return to the Flow Trace page.

   For more information about troubleshooting SOA operations, see the "Troubleshooting Oracle SOA Suite" chapter in the Oracle Fusion Applications Administrator's Troubleshooting Guide.

8. Inspect the Recovery column of the Faults table for a row that contains a Recoverable link.

   • If the Recovery column contains a Recoverable link, then you may be able to resume processing of the SOA operation at the point where it previously failed. Click Recoverable to display more details about the recoverable instance and continue to Step 9.

   • If the Recovery column does not contain a Recoverable link, then the SOA operation is not recoverable. Do not proceed with these instructions. Instead, have an appropriate Oracle Fusion Applications user restart the sequence of operations that led to the incident, beginning as close to the point of failure as is practical.

9. In the list of faults for the instance in question, select the row that contains the word Recoverable in the Recovery column, and then click Recover in the page area below the list.

10. In the Confirmation dialog box, click Yes.

    If the recovery process succeeds, the following message is displayed:

    Action completed successfully.
    

    Click OK to dismiss this message, which marks the completion of the recovery task.

    If the recovery process does not succeed, the list of faults with the word Recoverable in the Recovery column remains visible. This indicates that you have not succeeded in addressing the underlying cause of the original problem. Double-check your fix for the problem or try another approach. When you believe that you have addressed the problem successfully, return to the Flow Trace page and repeat Step 8 through Step 10.

14.4.2.4 Creating Incidents Manually Using Support Workbench

If you discover an issue with your Oracle Fusion applications that you want to gather data about, and if the system has not created an incident for it automatically, then it is recommended that you create an incident for it, yourself. Using incidents for all problems helps you organize the information that you gather during the troubleshooting process.

If your Oracle Fusion Applications environment includes Cloud Control, it is recommended that you use Support Workbench to work with incidents.

Note:

To use Support Workbench with Oracle Fusion Applications, you must obtain and install Cloud Control.

If your Oracle Fusion Applications environment does not include Cloud Control, then you must use the Automatic Diagnostic Repository Command Interpreter (adrci) to work with incidents, instead of Support Workbench. For more information, see Section 14.4.2.6.

When you use Support Workbench to create an incident manually, the process of creating the incident automatically creates a new problem listing that is associated with that incident.

To create a new incident and a new problem listing, manually, using Support Workbench:

1. If you have not already done so, in Cloud Control, select the Fusion J2EE Application target that is associated with the issue for which you are creating an incident.

2. From the dynamic drop-down menu, choose Diagnostics > Support Workbench.

3. In the Host Credentials area of the page, use one of the following methods to supply credentials for connecting to the machine that hosts the Oracle WebLogic Server that is associated with the incident you are creating.

   • If appropriate credentials have previously been saved as preferred credentials, select Preferred in the Credential field.

   • If appropriate credentials have previously been saved as named credentials, select Named in the Credential field and then select the specific credentials from the Credential Name drop-down list.

   • If appropriate credentials have not previously been set, select New in the Credential field and complete the following sub-steps:

     1. In the UserName and Password fields, enter credential values for connecting to the host machine.

     2. Inspect the default credential name supplied by Support Workbench in the Save As field, and modify the credential name if you prefer.

     3. If you want the credentials you are entering to be the preferred credentials for the relevant host, select the Set as Preferred Credentials checkbox.

     4. If you want to verify that the credentials you entered work correctly, click Test.

4. In the WebLogic Server Credentials area of the page, use one of the following methods to supply credentials for connecting to the Oracle WebLogic Server that is associated with the incident you are creating.

   Note:

   If a WebLogic Server Credentials area is not visible on the page, that indicates that the relevant Oracle WebLogic Server is currently unavailable. Some operations cannot be done when the server is unavailable, such as creating user-reported problems and performing additional diagnostic dumps. For information about restarting an Oracle WebLogic Server, see the "Starting and Stopping Servers" chapter in Oracle Fusion Middleware Managing Server Startup and Shutdown for Oracle WebLogic Server.

   • If appropriate credentials have previously been saved as preferred credentials, select Preferred in the Credential field.

   • If appropriate credentials have previously been saved as named credentials, select Named in the Credential field and then select the specific credentials from the Credential Name drop-down list.

   • If appropriate credentials have not previously been set, select New in the Credential field and complete the following sub-steps:

     1. In the UserName and Password fields, enter credential values for connecting to the Oracle WebLogic Server.

     2. Inspect the default credential name supplied by Support Workbench in the Save As field, and modify the credential name if you prefer.

     3. If you want the credentials you are entering to be the preferred credentials, select the Set as Preferred Credentials checkbox.

5. Click Continue.

   A Support Workbench page that summarizes problems related to your target now appears.

6. On the page that summarizes problems for the selected target, scroll down to the Related Links section and click Create User-Reported Problem.

7. On the Create User-Reported Problem page, select the Other issue type and click Continue with Creation of Problem.

8. On the Problem Details page, perform the following sub-steps, as needed:

   1. If you want to associate the new incident with an existing Oracle Service Request, click SR# Edit, enter the Service Request number, and click OK.

   2. If you want to associate the new incident with an existing bug, click Bug#, enter the bug number, and click OK.

   3. On the Incidents tab, if you want to see more information about the incident you are creating, such as user impact, checker findings, and purge date, click Show in the appropriate row of the table, or click Show All Details in the table header.

   4. If you want to change which incidents are displayed in the table, select a new value from the Data Dumped drop-down list and click Go.

   5. If you want to see the list of Oracle Fusion Middleware diagnostic dumps that are created automatically when you manually create an incident, or if you want to run any additional Oracle Fusion Middleware diagnostic dumps, select the listing for your incident in the table and click View to display the Incident Details page.

      For more information about using this page, see the topic "Incident Details Page" in the Cloud Control online help. For more information about Oracle Fusion Middleware dumps, see the "Diagnosing Problems" chapter in the Oracle Fusion Middleware Administrator's Guide.

14.4.2.5 Packaging Incidents Using Support Workbench

If you discover an issue with your Oracle Fusion applications that you want to gather data about, and if the system has not created an incident and problem for it automatically, then it is recommended that you create an incident for it, yourself.

If your Oracle Fusion Applications environment includes Cloud Control, it is recommended that you use Support Workbench to work with incidents. For more information about using Support Workbench to create incidents manually, see Section 14.4.2.4. The process of manually creating an incident using Support Workbench automatically creates an associated problem listing.

Note:

To use Support Workbench with Oracle Fusion Applications, you must obtain and install Cloud Control.

If your Oracle Fusion Applications environment does not include Cloud Control, then you must use the Automatic Diagnostic Repository Command Interpreter (adrci) to work with incidents, instead of Support Workbench. For more information, see Section 14.4.2.6.

As part of creating and packaging incidents using Support Workbench, you must supply credentials both to connect to the Oracle WebLogic Server on which the relevant Oracle Fusion Application is running, and to connect to the host machine on which that Oracle WebLogic Server is running.

Various kinds of incidents can be packaged in Support Workbench. For Oracle Fusion Applications performance incidents, in particular, it is recommended that you use the wizard that is specifically designed for packaging Oracle Fusion Applications incidents, because that wizard greatly simplifies the process of gathering the information that Oracle Support is likely to need when working on your problem. However, you can also use general Support Workbench functionality to package Oracle Fusion Applications incidents.

This section contains the following topics:

• Packaging Oracle Fusion Applications Incidents Using a Wizard

• Packaging Oracle Fusion Applications Incidents Using General Support Workbench Functionality

14.4.2.5.1 Packaging Oracle Fusion Applications Incidents Using a Wizard

If your Oracle Fusion Applications environment includes Cloud Control, it is recommended that you package Oracle Fusion Applications performance incidents using the special Support Workbench wizard for this incident type. The wizard provides a convenient way to include various file types that Oracle Support may need to resolve your problem.

Note:

To use the wizard for packaging Oracle Fusion Applications performance incidents, the relevant incident must already exist. If you need to create an incident manually, see Section 14.4.2.4.

Note:

When you plan to package an Oracle Fusion Applications incident, it is recommended that you perform the following actions before you begin the process of packaging the incident.

• Use the Diagnostic Dashboard to perform any diagnostic tests that are specific to that Oracle Fusion application and that are relevant for the selected incident. For more information, see Section 13.10.

• Associate the diagnostic test results with the selected incident. For more information, see Section 14.6.3.

The packaging procedure does not automatically include the results of manually run Oracle Fusion Applications diagnostic tests unless those results are already associated with the incident being packaged.

To package incidents using the Support Workbench wizard for Oracle Fusion Applications incidents:

1. If you have not already done so, in Cloud Control, select a target that is associated with the problem or incident that you want to package.

   Make sure that you choose one of the following target types:

   • An Oracle Fusion Applications instance target

   • An Oracle Fusion Applications product family target

   • An Oracle Fusion Applications cluster application target

     Note:

     The Support Workbench wizard for Oracle Fusion Applications incidents is unavailable when an Oracle Fusion application instance (J2EE Fusion Application) target is selected.

2. From the dynamic drop-down menu, choose Diagnostics > Support Workbench.

3. In the Support Workbench rollup page for your target, scroll down until the Problem Summary table is visible and select the problem that is associated with the incident that you want to package.

4. Click Create Package.

5. In the Host Credentials area of the Create Package: Credentials page, use one of the following methods to select credentials for connecting to the machine that hosts the Oracle WebLogic Server on which the Oracle Fusion application was running when the problem or incident occurred.

   • If appropriate credentials have previously been saved as preferred credentials, select Preferred in the Credential field.

   • If appropriate credentials have previously been saved as named credentials, select Named in the Credential field and then select the specific credentials from the Credential Name drop-down list.

   • If appropriate credentials have not previously been set, select New in the Credential field and complete the following sub-steps:

     1. In the UserName and Password fields, enter credential values for connecting to the host machine.

     2. Inspect the default credential name supplied by Support Workbench in the Save As field, and modify the credential name if you prefer.

     3. If you want the credentials you are entering to be the preferred credentials for the Oracle WebLogic Server, select the Set as Preferred Credentials checkbox.

     4. If you want to verify that the credentials you entered work correctly, click Test.

6. In the WebLogic Server Credentials area of the Create Package: Credentials page, use one of the following methods to select credentials for connecting to the Oracle WebLogic Server where the Oracle Fusion application was running when the problem or incident occurred.

   Note:

   If a WebLogic Server Credentials area is not visible on the page, that indicates that the relevant Oracle WebLogic Server is currently unavailable. Some operations cannot be done when the server is unavailable, such as creating user-reported problems and performing additional diagnostic dumps. For information about restarting an Oracle WebLogic Server, see the "Starting and Stopping Servers" chapter in Oracle Fusion Middleware Managing Server Startup and Shutdown for Oracle WebLogic Server.

   • If appropriate credentials have previously been saved as preferred credentials, select Preferred in the Credential field.

   • If appropriate credentials have previously been saved as named credentials, select Named in the Credential field and then select the specific credentials from the Credential Name drop-down list.

   • If appropriate credentials have not previously been set, select New in the Credential field and complete the following sub-steps:

     1. In the UserName and Password fields, enter credential values for connecting to the Oracle WebLogic Server.

     2. Inspect the default credential name supplied by Support Workbench in the Save As field, and modify the credential name if you prefer.

     3. If you want the credentials you are entering to be the preferred credentials for the Oracle WebLogic Server, select the Set as Preferred Credentials checkbox.

7. Click Next.

8. In the Create Package: Package Details page, specify the a name and description for the package you are creating, select the incidents to include in the package, and then click Next.

9. In the Create Package: Dumps page, select various kinds of dump files to add to the package and then click Next.

   Note:

   Unless Oracle Support has given you other instructions, it is recommended that you select all of the types of dump fields that are listed on the page. If you select Heap Dumps, you must supply relevant file names and locations. If you select AWR Reports, you must specify the name of the relevant database instance and credentials for working with that database.

10. In the Create Package: Review page, decide whether or not to customize the package:

    • If you need to customize the package, select the Customize Package checkbox to display the Customize Package page, and proceed to Step 11.

    • If you do not need to customize the package, click Proceed to Send and skip to Step 17.

11. On the Customize Package: PackageName page, review the current contents of the package, decide how you want to customize the package, and use the tabs, buttons, and links on the page to make the desired changes.

    For more information about the settings and controls on this page, see the "Customize Package Page" topic in the Cloud Control online help.

    For more information about customizing actions that you start on this page but complete on other pages, see any of the following Cloud Control online help topics that interest you:

    • Add Incidents Page

    • Add Problems to Package Page

    • Exclude Problems Page

    • Package Manifest Page

    • Additional Dumps and Test Cases Page

    • Add External Files Page

    • Copy Out Files Page

    • Copy In Files Page

12. When you have made the desired changes, click the Finish Contents Preparation link in the Packaging Tasks area of the page.

    After a short wait, the page displays information about any additional relevant files that were added automatically.

13. In the Packaging Tasks area of the page, click the Generate Upload File link.

14. On the Generate Upload File: PackageName page, select whether the upload file should contain the entire contents of the package, or only an incremental addition made since the last upload file was generated.

    For more information about the settings on this page, see the topic "Generate Upload File Page" in the Cloud Control online help.

15. On the same page, specify whether to generate the package immediately or at a specified later time, and then click Submit.

    It may take several minutes to generate the upload file. When the process is complete, a confirmation message is displayed. Click OK to dismiss it.

16. On the Customize Package: PackageName page, click either the Send Upload Files link or the Send to Oracle button in the Packaging Tasks area of the page.

17. On the Send to Oracle page, supply the necessary credentials for logging in to My Oracle Support, make sure that the package you want to upload is selected, and specify whether to send the package immediately or at a specified later time.

    For more information about the settings on this page, see the topic "Send to Oracle Page" in the Cloud Control online help.

18. When you are satisfied with the composition of the package and the details for sending it to Oracle Support, click Submit.

    After a brief wait, Support Workbench displays a message about whether or not the selected package was sent to Oracle successfully.

    If the send attempt was not successful, address any problem that the message describes, then repeat the send attempt.

14.4.2.5.2 Packaging Oracle Fusion Applications Incidents Using General Support Workbench Functionality

If your Oracle Fusion Applications environment includes Cloud Control, it is recommended that you package Oracle Fusion Applications performance incidents using the special Support Workbench wizard for that incident type, as described in Section 14.4.2.5.1. For Oracle Fusion Applications incidents that do not concern performance, you can use general Support Workbench functionality to package Oracle Fusion Applications incidents.

Note:

When you plan to package an Oracle Fusion Applications incident, it is recommended that you perform the following actions before you begin the process of packaging the incident.

• If no incident exists for the issue that you are investigating, create one manually. For more information, see Section 14.4.2.4.

• Use the Diagnostic Dashboard to perform any diagnostic tests that are specific to that Oracle Fusion application and that are relevant for the selected incident. For more information, see Section 13.10.

• Associate the diagnostic test results with the selected incident. For more information, see Section 14.6.3.

The packaging procedure does not automatically include the results of manually run Oracle Fusion Applications diagnostic tests unless those results are already associated with the incident being packaged.

To package incidents using general Support Workbench functionality:

1. In Cloud Control, if you have not yet selected the J2EE Fusion Application target that is associated with the problem or incident that you want to package, select that target and skip to Step 2.

   If you have already selected the J2EE Fusion Application target that is associated with the problem and have already started working with Support Workbench, then complete the following substeps:

   1. Navigate to either the applicable Problem Details page or the Support Workbench rollup page that summarizes the problems that are related to your target.

   2. If you navigated to the Support Workbench rollup page, skip to Step 6; otherwise, continue to the following sub-step.

   3. If you navigated to the applicable Problem Details page, click either Quick Package or the Package the Problem link in the Investigate and Resolve area of the page.

      For more information about the Quick Package button and the Package the Problem link, see the topic "Problem Details Page" in the Cloud Control online help.

   4. If you clicked the Package the Problem link, then skip to Step 9; otherwise, continue to the following sub-step.

   5. If you clicked Quick Package, then skip to Step 10.

2. From the dynamic drop-down menu, choose Diagnostics > Support Workbench.

3. In the Host Credentials area of the page, use one of the following methods to select credentials for connecting to the machine that hosts the Oracle WebLogic Server on which the Oracle Fusion application was running when the incident or issue occurred.

   • If appropriate credentials have previously been saved as preferred credentials, select Preferred in the Credential field.

   • If appropriate credentials have previously been saved as named credentials, select Named in the Credential field and then select the specific credentials from the Credential Name drop-down list.

   • If appropriate credentials have not previously been set, select New in the Credential field and complete the following sub-steps:

     1. In the UserName and Password fields, enter credential values for connecting to the host machine.

     2. Inspect the default credential name supplied by Support Workbench in the Save As field, and modify the credential name if you prefer.

     3. If you want the credentials you are entering to be the preferred credentials for the relevant host, select the Set as Preferred Credentials checkbox.

     4. If you want to verify that the credentials you entered work correctly, click Test.

4. In the WebLogic Server Credentials area of the page, use one of the following methods to select credentials for connecting to the Oracle WebLogic Server where the Oracle Fusion application was running when the problem or incident occurred.

   Note:

   If a WebLogic Server Credentials area is not visible on the page, that indicates that the relevant Oracle WebLogic Server is currently unavailable. Some operations cannot be done when the server is unavailable, such as creating user-reported problems and performing additional diagnostic dumps. For information about restarting an Oracle WebLogic Server, see the "Starting and Stopping Servers" chapter in Oracle Fusion Middleware Managing Server Startup and Shutdown for Oracle WebLogic Server.

   • If appropriate credentials have previously been saved as preferred credentials, select Preferred in the Credential field.

   • If appropriate credentials have previously been saved as named credentials, select Named in the Credential field and then select the specific credentials from the Credential Name drop-down list.

   • If appropriate credentials have not previously been set, select New in the Credential field and complete the following sub-steps:

     1. In the UserName and Password fields, enter credential values for connecting to the Oracle WebLogic Server.

     2. Inspect the default credential name supplied by Support Workbench in the Save As field, and modify the credential name if you prefer.

     3. If you want the credentials you are entering to be the preferred credentials, select the Set as Preferred Credentials checkbox.

5. Click Continue.

   A Support Workbench page that summarizes problems related to your target now appears.

6. Click the Problems tab and select the relevant time period from the View drop-down list.

7. In the table that lists problems, select the checkbox for one or more relevant problems.

8. Click Package.

9. On the Package: Select Packaging Mode page, select the mode of packaging that you want to use, and then click Continue.

   Note:

   If you selected multiple problems in Step 7, you must select the Custom Packaging mode at this point.

   For more information about packaging modes, see the topic "Package Page" in the Cloud Control online help.

   If you select Quick Packaging, continue to Step 10.

   If you select Custom Packaging, skip to Step 15.

10. On the Quick Packaging: Create New Package page, supply information about the name and description for the package, indicate whether you want the package to be sent to Oracle Support automatically, and then click Next.

    For more information about the fields on this page, see the topic "Quick Packaging Wizard" in the Cloud Control online help.

11. On the Quick Packaging: View Contents page, review the information about the incidents that will be included in the package, and click Next when you are satisfied.

12. On the Quick Packaging: View Manifest page, you can review information about the specific files to be included in the package.

    Click Next when you are ready to proceed to the next step in the packaging process.

13. On the Quick Packaging: Schedule page, specify whether to generate the package immediately or at a specified later time.

    If, earlier, you elected to send the package to Oracle Support automatically, the timing you specify for generating the package also applies to sending the package.

    For more information about the fields on this page, see the topic "Quick Packaging Wizard" in the Cloud Control online help.

14. When you are satisfied with the composition of the package, click Submit.

    If you selected Quick Packaging in Step 9, skip the remaining steps of this procedure.

15. If you selected Custom Packaging in Step 9, on the Custom Packaging: Select Package page, either select an existing package to modify, or supply information about the name and description for a new package, and then click OK.

    For more information about the settings on this page, see the topic "Custom Packaging Page" in the Cloud Control online help.

    After a short wait, the page displays a message confirming either that the selected problem was added to an existing package, or that a new package was created.

16. On the Customize Package: PackageName page, review the current contents of the package, decide how you want to customize the package, and use the tabs, buttons, and links on the page to make the desired changes.

    For more information about the settings and controls on this page, see the "Customize Package Page" topic in the Cloud Control online help.

    For more information about customizing actions that you start on this page but complete on other pages, see any of the following Cloud Control online help topics that interest you:

    • Add Incidents Page

    • Add Problems to Package Page

    • Exclude Problems Page

    • Package Manifest Page

    • Additional Dumps and Test Cases Page

    • Add External Files Page

    • Copy Out Files Page

    • Copy In Files Page

17. When you have made the desired changes, click the Finish Contents Preparation link in the Packaging Tasks area of the page.

    After a short wait, the page displays information about any additional relevant files that were added automatically.

18. In the Packaging Tasks area of the page, click the Generate Upload File link.

19. On the Generate Upload File: PackageName page, select whether the upload file should contain the entire contents of the package, or only an incremental addition made since the last upload file was generated.

    For more information about the settings on this page, see the topic "Generate Upload File Page" in the Cloud Control online help.

20. On the same page, specify whether to generate the package immediately or at a specified later time, and then click Submit.

    It may take several minutes to generate the upload file. When the process is complete, a confirmation message is displayed. Click OK to dismiss it.

21. On the Customize Package: PackageName page, click the Send Upload Files link in the Packaging Tasks area of the page.

22. On the Send to Oracle page, supply the necessary credentials for logging in to My Oracle Support, make sure that the package you want to upload is selected, and specify whether to send the package immediately or at a specified later time.

    For more information about the settings on this page, see the topic "Send to Oracle Page" in the Cloud Control online help.

23. When you are satisfied with the composition of the package and the details for sending it to Oracle Support, click Submit.

    After a brief wait, Support Workbench displays a message about whether or not the selected package was sent to Oracle successfully.

    If the send attempt was not successful, address any problem that the message describes, then repeat the send attempt.

14.4.2.6 Creating and Packaging Incidents Using the Automatic Diagnostic Repository Command Interpreter (adrci)

If you discover an issue with your Oracle Fusion applications that you want to gather data about, and if the system has not created an incident for it automatically, then it is recommended that you create an incident for it, yourself.

If your Oracle Fusion Applications environment includes Cloud Control, it is recommended that you use Support Workbench to create and package incidents. For more information, see Section 14.4.2.4 and Section 14.4.2.5.

If your Oracle Fusion Applications environment does not include Cloud Control, you must use WLST to create incidents and adrci to package them. For more information, see the "Creating an Incident Manually" section in the Oracle Fusion Middleware Administrator's Guide.

After creating the incident manually, you can add files to it. For example, you might want to add Oracle Fusion Applications diagnostic test results, Oracle Fusion Middleware diagnostic dumps, excerpts from downloaded log files, and Remote Diagnostic Agent (RDA) reports to the incident. You might also want to add a Readme.txt file containing information for Oracle Support. After you add files to an incident, you can package the incident into a zip file for transmission to Oracle Support. For more information about adding files to an incident and packaging the incident into a zip file, see the "Packaging an Incident" section in the Oracle Fusion Middleware Administrator's Guide.

14.4.3 Working with Automatically Created QuickTrace Dumps

As part of assisting you with resolving incidents, Oracle representatives often examine any dump files that were automatically created when the incident occurred, including QuickTrace dump files.

When a QuickTrace dump file is automatically created as part of an incident, the file is stored in the incident directory along with any other dump files that are related to that incident. QuickTrace dump file names begin with odl.quicktrace.

This section contains the following topics:

• Viewing Automatically Created QuickTrace Dump Files Using Oracle WebLogic Scripting Tool

• Viewing Automatically Created QuickTrace Dump Files Using the Automatic Diagnostic Repository Command Interpreter (adrci)

• Viewing Automatically Created QuickTrace Dump Files Using Support Workbench

14.4.3.1 Viewing Automatically Created QuickTrace Dump Files Using Oracle WebLogic Scripting Tool

You can use the following standard Oracle WebLogic Scripting Tool commands to view information about automatically created QuickTrace dump files:

• Use the listIncidents command to view a list of incidents.

• Use the showIncident command to view details about a particular incident, including a list of dump files associated with that incident.

• Use the getIncidentFile command to view a specified dump file such as a QuickTrace dump file.

For more information about using these commands, see the "Viewing Problems and Incidents" section and the "Working with Diagnostic Dumps" section in the Oracle Fusion Middleware Administrator's Guide.

14.4.3.2 Viewing Automatically Created QuickTrace Dump Files Using the Automatic Diagnostic Repository Command Interpreter (adrci)

You can use the following standard Automatic Diagnostic Repository Command Interpreter (adrci) commands to view information about automatically created QuickTrace dump files:

• Use the show homes command to view a list of Automatic Diagnostic Repository (ADR) home directories. Depending on the configuration of your system, QuickTrace dumps may be located under one or more of these ADR home directories.

• Use the set home directory_path command to indicate which respository you want to work with, replacing directory_path with one of the locations from the output of the show home command.

• Use the show incdir -i incdir_incident_number command to view a list of files associated with the incident that incident_number identifies, including any QuickTrace dump files for that incident. QuickTrace dump file names start with odl_quicktrace and end with .dmp.

• Use the show tracefile -i incident_number command to view a list of just the dump files associated with the incident that incident_number identifies.

• Use the show trace filename command to view the contents of the QuickTrace dump file that you specify.

For more information about using adrci, see Section 14.4.2 and the chapter about the ADR Command Interpreter in Oracle Database Utilities.

14.4.3.3 Viewing Automatically Created QuickTrace Dump Files Using Support Workbench

If your Oracle Fusion Applications environment includes Cloud Control, then you can use Support Workbench to view Oracle Fusion Applications incident information, including automatically created QuickTrace dumps.

1. In Oracle Enterprise Manager, select an Oracle Fusion Applications instance, product family, or cluster application.

2. From the dynamic drop-down menu, choose Diagnostics > Support Workbench.

3. From the View Data drop-down list, select the timeframe that contains the problem for which you want to inspect a QuickTrace dump.

4. Scroll down to the Support Workbench Problems Summary table and click the Incident count for the problem that interests you.

5. Click the Incidents tab in the Problem Details page.

6. Click the Summary of the incident that interests you.

7. In the Guided Resolution area of the Incident Details page, click View Diagnostic Data.

8. In the Dump Files table, locate a row that contains a File Name that starts with odl_quicktrace.

9. Scroll to the right and click the View Contents icon in that row.

10. Inspect the information in the View Log File page for the dump file, and click Download if you want to download the file.

14.4.4 Working with Manually Created QuickTrace Dumps

Oracle Support may occasionally ask you to perform a manual dump of QuickTrace buffers as part of work to resolve a problem that did not generate an incident automatically.

To create a QuickTrace dump manually:

1. Decide whether you want to associate the QuickTrace dump file with an existing incident at the time when you perform the dump:

   • If you want to associate a new QuickTrace dump file with an existing incident at the time when you perform the dump, make a note of the relevant incident ID. This can either be an automatically created incident or a manually created incident. For more information about creating incidents manually, see Section 14.4.2.4 and Section 14.4.2.6.

   • If you do not want to associate a new QuickTrace dump with an incident at the present time, decide where you will store the dump file.

2. Use the Oracle WebLogic Scripting Tool executeDump command to perform an odl.quicktrace dump, choosing appropriate syntax for your needs:

   • If you want to associate the dump file with an existing incident, use the id argument. This allows you to retrieve the dump file, later, using the getIncidentFile argument

   • If would rather associate the dump file with an incident at a later time, use the outputFile argument to specify an output file location at the time when you execute the dump.

   For more information about using the executeDump command and its arguments, see the "Executing Dumps" section in the Oracle Fusion Middleware Administrator's Guide.

14.4.5 Working with Other Diagnostic Dumps

In some troubleshooting situations, you may find it useful to locate or obtain Oracle Fusion Middleware diagnostic dumps such as Oracle WebLogic Server diagnostic images, Java Virtual Machine thread dumps, and Dynamic Monitoring Service metric dumps.

Oracle Fusion Applications incidents that are generated from Java code (including SOA) automatically execute these kinds of dumps and include the results in the incident data. Oracle Fusion Middleware diagnostic dumps are not available for Oracle Fusion Applications incidents that are generated from PL/SQL or C code.

If you troubleshoot an Oracle Fusion Applications problem that did not generate an incident automatically, it is recommended that you create an incident manually, then execute diagnostic dumps manually, and examine the dump files for information that may help you to resolve the problem. If you need to work with Oracle Support to resolve the problem, it is recommended that you include manually executed dump results in the incident data package that you send to Oracle for review. For information about how to list available diagnostic dumps, view dump descriptions, and manually execute selected dumps for Oracle Fusion Middleware, see the "Working with Diagnostic Dumps" section in the Oracle Fusion Middleware Administrator's Guide.

14.4.6 Working with Cloud Control Problem Analysis and the Analyze Logs Options

If your Oracle Fusion Applications environment includes Cloud Control, you can use the Problem Analysis and Analyze Log options in Cloud Control to help you inspect metrics, target status information, and logs during troubleshooting.

To inspect metrics, status information, and logs using Cloud Control:

1. Use one of the following methods to select a target for which you want to use problem analysis or analyze logs:

   • Navigate to Incident Manager and select the incident that interests you.

   • Navigate to the home page of an Oracle Fusion Applications product family and locate a metric chart in the Performance Indicators section.

   • From the home page of any Fusion Instance, Product Family, Product, Fusion Cluster Application, or Fusion J2EE Application, choose Monitoring > Key Performance Indicators or Monitoring > System Performance and then locate a metric chart.

2. Use one of the following methods to display Problem Analysis pages for Oracle Fusion Applications targets:

   • In Incident Manager, click Problem Analysis in the Guided Resolution pane.

   • On a home page or a metrics page such as a Key Performance Indicators or System Performance page, under or adjacent to any metric chart, click a link to display the Additional Information pop-up and then click Problem Analysis.

     The name and location of the link that displays the Additional Information popup varies depending on the page and metric chart you are viewing. For example, some typical link names are:

     • Request Processing Time (for Response and Load metric charts)

     • Payables Invoice Validation Run Time (for an Oracle Fusion Financials Key Performance Indicator metric chart)

   In the Problem Analysis page, the Source Metric chart displays the type of metric for which the incident was automatically created or the type of metric for which you clicked Request Processing Time.

3. Near the top of the Metric Dependency Charts pane, click the time period for which you want the charts to display data, or click and drag within a metric chart to indicate the time period you want to inspect.

4. Inspect the charts for unusual increases in recorded metrics such as request processing time, CPU usage in storage units or percent utilized, number of requests per minute, Java Virtual Machine heap memory usage, or server memory usage.

   If you find that request processing time increased due to a high number of requests per minute, you may need to increase the capacity of your system.

5. If the metric charts do not indicate the cause of the problem, scroll down to the Related Information pane and inspect the List of Related Targets To Analyze table for information about target health (status) and recent configuration changes.

6. If you want to see a reminder of the topology of the components for which data is being displayed, click the Related Targets Topology tab.

7. If the List of Related Targets to Analyze table does not indicate the cause of the problem, scroll up and click a link adjacent to one of the metrics and then click Analyze Logs in the Additional Information pop-up to display log messages for the selected target and its members during the selected time period.

8. To display log messages in the selected time period for all targets related to the relevant incident or starting metric, scroll up to the Metric Dependency Charts pane header and click View Related Log Messages.

9. Inspect any log messages that are displayed for possible causes of problems.

14.5 Troubleshooting Using Log Settings

You can use log settings to increase the amount of information that a particular Oracle Fusion application gathers about its own operations. This is particularly useful with problems that are easily reproducible—you can change log settings, reproduce the problem, then return log settings to their normal levels so that you do not use disk space for unnecessarily detailed information that is not relevant to the problem.

This section contains the following topics:

• Troubleshooting Using Standard Log Files

• Viewing Special Log Output for Oracle Fusion Incentive Compensation

• Correlating Log Messages Across Log Files and Components

• Downloading Log Files

When you are troubleshooting a problem in an Oracle Fusion Applications environment, you may also find it helpful to use the logging capabilities of Oracle Fusion Middleware. At times, Oracle Support may ask you to use certain of these capabilities when attempting to reproduce a problem, particularly if you have a dedicated testing environment available.

14.5.1 Troubleshooting Using Standard Log Files

Most code modules in Oracle Fusion Applications use a standard diagnostic logging framework in which profile options determine how much information is logged for a particular site or a particular user. For information about adjusting site-level profile options or adjusting user-level profile options as an administrator, see Section 13.7.5. For information about allowing selected users to adjust the level of information that is logged for their own sessions, see Section 14.2.1.

14.5.2 Viewing Special Log Output for Oracle Fusion Incentive Compensation

For information about configuring the special logging functionality for certain Oracle Fusion Incentive Compensation batch jobs, see Section 14.2.9. After you have enabled this functionality and attempted to replicate the Oracle Fusion Incentive Compensation problem that you are troubleshooting, you must perform some database queries to view the resulting log entries. For these queries, you must specify the name of the Oracle Fusion Incentive Compensation process that you ran when attempting to replicate the reported problem. The process names that you can use in the query are as follows:

• CALCULATION MAIN PROCESS

• CLASSIFICATION

• COLLECT_STATISTICS

• Crediting

• GEN_RULE_ENGINE

• GET_WINNERS

• PopulateSrpBatches

• PopulateSrpTables

• REVERT TRANSACTIONS

• ROLLUP

• Rollup within Crediting

Note:

These process names are case-sensitive. For more information about how Oracle Fusion Incentive Compensation processes operate, see product-specific documentation in Oracle Fusion Applications Help.

To view special log entries for Oracle Fusion Incentive Compensation batch jobs:

1. Use a SQL client to connect to Oracle Database and log in as a user that has permission to view the Fusion database schema.

2. Execute the following SQL command, replacing process_name with the name of the Oracle Fusion Incentive Compensation process that you ran when attempting to replicate the reported problem:

   select process_audit_id from cn_process_audits_all
   where process_type = process_name
   order by creation_date desc;
   

3. Make a note of the process audit ID that is returned in the first row of the output.

4. Execute the following SQL command, replacing selected_process_audit_ID with the value that you noted in Step 3.

   select message_type, message_text from cn_process_audit_lines_all
   where process_audit_id = selected_process_audit_ID
   order by process_audit_line_id asc;
   

5. If you have not already done so, place the output of Step 4 in a file that you can inspect with a text editor or send to Oracle Support if requested.

6. When you are finished gathering log entries, reset the value of the CN_DEBUG profile option setting to N.

   For information about setting the value of the CN_DEBUG profile option, see Section 14.2.9.

14.5.3 Correlating Log Messages Across Log Files and Components

Sometimes, as part of the process of troubleshooting Oracle Fusion Applications, you may need to find all of the log entries that result from a particular action. If the action involves multiple Oracle Fusion applications or multiple Oracle Fusion Middleware components, then the log entries are placed in multiple log files. You can use certain values related to the particular action to locate the relevant log messages in those log files.

This section contains the following topics:

• Values for Correlating Log Messages

• Correlating Log Messages for Oracle Enterprise Scheduler Jobs

• Correlating Log Messages for SOA Services and BPEL Processes

• Correlating Log Messages for Other Process Types

14.5.3.1 Values for Correlating Log Messages

You can use a particular action's Execution Context Identifier (ECID) and Relationship Identifier (RID) values to locate the log entries that result from that action.

A unique ECID value is assigned to each user request. However, depending on what is involved in the request, a log entry that is related to that request may have the original ECID of the request, or a different ECID of a child process spawned by the original request. For example, if you submit a job to Oracle Enterprise Scheduler, the request to schedule the job and the job, itself, always have different ECIDs.

For a particular ECID, if the request requires processing by multiple components, by multiple modules within a component, or using multiple threads, then a unique RID value may be assigned when the work passes to a new thread or process.

In log entries, the ECID and RID values are typically separated by a comma. ECID values can include alphanumeric characters, colons, periods, and hyphens. RID values can include numerals and colons. The RID value indicates the relationship of the current thread or process to its creator. The first RID created for a request is RID = 0. Each shift in generation is represented by a colon. For example, the RID value for the seventh child process of the third child process of the root context for a request is represented as 0:3:7.

14.5.3.2 Correlating Log Messages for Oracle Enterprise Scheduler Jobs

When a job is submitted to Oracle Enterprise Scheduler, the request to schedule the job and the job, itself, always have different Execution Context Identifiers (ECIDs).

The ECID for the job-scheduling request is displayed in the requesting user session at the time when the job is submitted. However, the ECID of the Oracle Enterprise Scheduler job, itself, is not assigned until the job starts, and it is not displayed in the requesting user session. To locate log entries related to the Oracle Enterprise Scheduler job, itself, you must use the job Request ID value.

To correlate log messages for Oracle Enterprise Scheduler jobs using Fusion Applications Control:

1. In the navigation pane, expand the Farm listing and then the Scheduling Services listing and then determine whether Scheduling Services contains Oracle Enterprise Scheduler clusters that contain individual Oracle Enterprise Scheduler servers, or individual Oracle Enterprise Scheduler servers without clusters.

2. To open the ESSAPP deployment administration page, click the listing for an Oracle Enterprise Scheduler cluster such as ESSAPP (ESSCluster), if one is available, or an individual Oracle Enterprise Scheduler server such as ESSAPP (ess_server1).

   Note:

   If you can select a cluster, that will allow you to search all servers in the cluster simultaneously. This is an advantage because each job request is logged only on the Oracle Enterprise Scheduler server where that request is actually processed.

3. If the job request for the action that you are troubleshooting appears in the list of Top 10 Running Job Requests, click its Request ID link to display the request details, then skip to Step 8.

4. In the context pane, from the dynamic Scheduling Service Group or Scheduling Service target menu, choose Job Requests > Search Job Request.

5. Use standard search techniques to locate the job request for the action that you are troubleshooting.

   For information about searching Oracle Enterprise Scheduler job requests, see Section 5.8.2.

6. In the search results, make a note of the Request ID value for the job that interests you.

7. In the search results, click the Request ID link to display the request details.

8. In the Request Details pane, select Request Log from the Action drop-down list.

   The Log Messages page displays all the log messages for the ECID associated with the selected job request.

9. Click any Log File link to display a list of the related log messages in the View Log Messages page.

   At this point, you can inspect the log messages on your screen or download the log file and search it for all messages containing the relevant ECID.

   Note:

   If the log level for a job request is set to FINER or FINEST when the job runs, then the diagnostic log entries for the job request are written to the log file specified by the AFLOG_FILENAME profile option, as well as to the request log file that is available to the user who submitted the job request.

14.5.3.3 Correlating Log Messages for SOA Services and BPEL Processes

If you need to correlate log messages that are related to SOA services or BPEL processes, you obtain the Execution Context Identifier (ECID) by first locating the relevant SOA composite instance.

To correlate log messages for SOA Services or BPEL processes using Fusion Applications Control:

1. In the navigation pane, expand the Farm listing, and then the SOA listing, and then the soa_infra listing, and then and select default.

2. In either pane, click the SOA composite instance name for which you want to locate log entries.

3. Make sure that your browser is set to allow the display of popup windows.

   For example, in Firefox, choose Options > Content from the Tools menu and either make sure that the Block pop-up windows checkbox is cleared or add any relevant hosts to Exceptions.

   Similarly, in Internet Explorer, choose Pop-up Blocker > Turn Off Pop-up Blocker from the Tools menu.

4. In the Recent Instances table, click the relevant Instance ID link to display the Flow Trace popup window.

   The ECID is displayed in the upper right corner of the window.

5. Make a note of the ECID value and close the Flow Trace popup window.

6. To view the relevant log entries using Fusion Applications Control, complete the following substeps. To view the log entries using the Oracle WebLogic Scripting Tool (WLST), skip to Step 7.

   1. In the navigation pane, select an appropriate target.

   2. From the dynamic target menu, choose Logs > View Log Messages.

   3. On the Log Messages page, in the Search area, click Add Field.

   4. In the popup list, select the ECID checkbox and click Add.

   5. Enter the ECID value in the ECID field.

   6. Adjust other search field values appropriately and click Search.

      Note:

      You can leave any search fields blank; blank fields are not used as search criteria.

      Skip the remainder of this procedure.

7. In the Oracle WebLogic Scripting Tool (WLST), enter a command like the following example, but substitute the ECID value that you previously noted.

   displayLogs(ecid='0000Hl9TwKUCslT6uBi8UH18lkWX000002')
   

14.5.3.4 Correlating Log Messages for Other Process Types

Many operations within Oracle Fusion applications do not use Oracle Enterprise Scheduler or SOA services or BPEL processes. For these operations, you can correlate log entries using the Execution Context Identifier (ECID).

To locate log entries that result from one particular Oracle Fusion application action that does not use Oracle Enterprise Scheduler or SOA services or BPEL processes:

1. Use Fusion Applications Control to search the log file that contains messages for the Oracle Fusion application in which the action was taken.

   Use any search criteria that will allow you to locate a log entry that is associated with the particular action. The associated ECID and RID values are included in the detailed log entry information. For more information about searching for particular information in an Oracle Fusion Applications log file, see Section 13.6.2 or Section 13.6.3.

2. When you have the ECID and RID values for the action, use one of the following methods to view all of the log entries associated with the action.

   • In Fusion Applications Control, on the Log Messages page, in the Search area, click Add Field to make sure that the ECID search field is displayed, then enter the ECID value in that field and click Search.

   • In Fusion Applications Control, on the page of Log Messages search results, with a relevant log message selected, select by ECID from the View Related Messages list.

   • In the Oracle WebLogic Scripting Tool (WLST), enter a command like the following example, but substitute the ECID value that you previously noted.

     displayLogs(ecid='0000Hl9TwKUCslT6uBi8UH18lkWX000002')
     

For more information about using ECIDs and RIDs to correlate log entries, see the "Correlating Messages Across Log Files and Components" section in the Oracle Fusion Middleware Administrator's Guide.

14.5.4 Downloading Log Files

As part of the process of troubleshooting Oracle Fusion Applications problems, an Oracle Support representative may occasionally request a copy of a particular log file. You can add downloaded log files to incidents in order to package those log files with incident information that you send to Oracle Support.

For more information about using Support Workbench to package incidents along with external files such as log files, see Section 14.4.2.5.2. For more information about adding files to an incident and packaging the incident into a zip file using adrci, see the "Packaging an Incident" section in the Oracle Fusion Middleware Administrator's Guide.

To download an Oracle Fusion Applications log file using Fusion Applications Control:

1. From the navigation pane, select the target for which you want to download a log file.

2. In the context pane, from the dynamic target menu, choose Logs > View Log Messages to display the log entries for the target you selected.

3. Use one of the following methods to locate the log file that you want to download:

   • Inspect the Log File column in the table of log entries that appears by default. If the column contains a link for the log file that you want to download, click the link and proceed to Step 4.

   • Perform a standard log file search using criteria that you expect to list the desired log file in the search results. When the Log File column contains a link for the log file that you want to download, click the link and proceed to Step 4.

   • Expand the Selected Targets section of the Search area and click the Target Log Files icon for the target most likely to be associated with the log file you want to download. In the list of log files that appears, select the file that you want to download and proceed to Step 4.

4. Click Download and specify where you want the log file to be saved.

14.6 Troubleshooting Using the Diagnostic Testing Framework

Some Oracle Fusion Applications diagnostic tests are designed to help you to monitor the health of your system. Other diagnostic tests are designed to help you troubleshoot problems or to gather data that Oracle Support may need when helping you to resolve a problem. For basic information about using Diagnostic Testing Framework features that are useful for both troubleshooting and monitoring system health, see Chapter 13. For information about the individual diagnostic tests that are provided with this release, see the Oracle Fusion Applications Common User Guide in Oracle Fusion Applications Help.

This section contains the following topics:

• Working with Automatically Launched Diagnostic Tests

• Registering Diagnostic Tests

• Sending Diagnostic Test Results to Oracle Support

• Purging the Results of Selected Diagnostic Test Runs

Note:

The user name that you use to sign in to an Oracle Fusion application affects which diagnostic operations are available to you. Be sure that you sign in using an account that is assigned to the job roles for the diagnostic operations that you need. For more information, see Section 13.9.1.

14.6.1 Working with Automatically Launched Diagnostic Tests

To make sure that certain diagnostic tests run automatically when certain error conditions occur, Oracle developers set the value of each test's APPS_MSG_ID tag to match the identifier of any error message that should trigger the automatic execution of that test. There is no configuration setting for disabling this automatic execution of diagnostic tests.

If an error has resulted in the automatic creation of an incident, then the results of any automatically run diagnostic tests are automatically associated with the incident. The identity of the user who received the error message is also recorded.

14.6.2 Registering Diagnostic Tests

Oracle supplies you with a number of diagnostic tests for Oracle Fusion Applications in seed data. Diagnostic tests that are supplied in seed data do not require you to register them.

However, if you report an Oracle Fusion application issue to Oracle Support, you may receive additional diagnostic tests to help resolve the issue. Depending on the exact circumstances, those diagnostic tests may or may not require registration. If registration is required for the additional diagnostic tests, Oracle will provide you with detailed registration instructions to follow.

14.6.3 Sending Diagnostic Test Results to Oracle Support

From time to time, you may run a diagnostic test and then want to send the results of the test to Oracle Support. To accomplish this, you attach the results of the test to the relevant incident.

Note:

You can also attach other types of files to incidents, such as log file excerpts or Readme.txt files containing information for Oracle Support. For more information about using Support Workbench to package incidents along with files such as log file excerpts or readme.txt files, see Section 14.4.2.5.2. For more information about using adrci to add such files to incidents, see the "Packaging an Incident" section in the Oracle Fusion Middleware Administrator's Guide.

To attach diagnostic test results to an incident:

1. Sign in to the Oracle Fusion application that generated the incident, using an account that has the Diagnostic Advanced User duty role or the Diagnostic Administrator duty role.

2. In the Oracle Fusion application, choose Troubleshooting > Run Diagnostic Tests from the Help menu to display the Diagnostic Dashboard instance for that application. For more information, see Section 13.9.2.

3. If you are not already displaying the Diagnostic Test Run Status pane, expand the Tasks panel in the Regional area of your screen and click Run Status.

4. In the Diagnostic Test Run Status table, locate and select the test run that has the report that you want to add to an incident.

5. Click Add to Incident to display the Select Incident dialog box.

6. Click the Select ADR Home icon to the right of the ADR Home field.

7. In the Search and Select: ADR Home dialog box, locate and select the table row for the ADR_HOME and server that your Oracle Fusion application is currently using.

   If you wish, you can use any of the following techniques to locate the row that describes your ADR_HOME and server:

   • In the ADR Home field, enter any part of your ADR_HOME name and then click Search.

   • In the Server field, enter any part of the name of the server that is running your Oracle Fusion application and then click Search.

   • In both the ADR Home field and the Server field, enter the values you want to search for, then select either All (to display rows that match both search strings) or Any (to display rows that match either search string) and click Search.

   Note:

   In the Search and Select: ADR Home dialog box, search operations are case-sensitive.

8. With the correct table row highlighted, click OK to return to the Select Incident dialog box.

9. To display a list of known problems for your selected ADR home directory, click the Select Problem icon to the right of the Problem field.

10. In the Search and Select: Problem dialog box, locate and select the problem that describes the incident to which you want to add test results.

11. With the correct problem highlighted, click OK to return to the Select Incident dialog box.

12. From the Incident field, select the incident to which you want to add test results, and then click OK.

    The test report results are added to the incident and a confirmation message is displayed.

14.6.4 Purging the Results of Selected Diagnostic Test Runs

From time to time, you may want to remove diagnostic test run results from your database, to keep the Run Status table from becoming too large. For information about deleting diagnostic test results, see Section 13.10.11.