17 Troubleshooting Oracle Fusion Applications Using Incidents, Logs, QuickTrace, and Diagnostic Tests

This chapter discusses how to use incidents, log settings, log files, QuickTrace, and diagnostic tests to help diagnose and resolve problems with Oracle Fusion Applications.

This chapter also discusses how incidents are related to log files, QuickTrace output, and diagnostic tests.

This chapter contains the following topics:

• Introduction to Troubleshooting Using Incidents, Logs, QuickTrace, and Diagnostic Tests

• Configuring Log, QuickTrace, and Incident Settings for Troubleshooting

• Configuring the Diagnostic Testing Framework for Troubleshooting

• Investigating, Reporting, and Solving a Problem

• Troubleshooting Using Log Settings

• Troubleshooting Using the Diagnostic Testing Framework

For information about using log settings, log files, and diagnostic tests to monitor normal operations for Oracle Fusion Applications, and for information about how log files and diagnostic tests are related to each other, see Chapter 13. For more information about the individual diagnostic tests that are provided with this release, see the Oracle Fusion Applications Common User Guide in Oracle Fusion Applications Help.

17.1 Introduction to Troubleshooting Using Incidents, Logs, QuickTrace, and Diagnostic Tests

Incidents are collections of information about problematic system events. Incidents can be created automatically or manually. Each automatically created incident contains information about the state of a particular Oracle WebLogic Server domain at the time when a specific problem occurred. Monitoring for and responding to incidents is a standard activity for help desk staff who support Oracle Fusion Applications in organizations that use Information Technology Infrastructure Library (ITIL) best practices.

This section contains the following topics:

• Relationships Between Incidents, QuickTrace Dumps, Log Files, and Diagnostic Tests

• Introduction to the Oracle Fusion Middleware Diagnostic Framework

• Standard Incident Administration Tasks and Tools

You can use incidents to diagnose and resolve problems, or to supply your help desk personnel or Oracle Support personnel with information about more complex problems. Oracle developers set up error messages in such a way that incidents are created automatically when certain high-priority error conditions occur. You can also create incidents manually.

All automatically created Oracle Fusion Applications incidents include diagnostic dump files. Dump files are provided in order to assist you in resolving incidents for your organization and to assist Oracle representatives in providing you with effective support when needed. The types of dump files provided depend on the nature of the incident. Some incidents include Oracle Fusion Middleware dump files such as Oracle WebLogic Server diagnostic images, Java Virtual Machine thread dumps, and Dynamic Monitoring Service metric dumps. For more information about Oracle Fusion Middleware dumps, see the "Diagnosing Problems" chapter in the Oracle Fusion Middleware Administrator's Guide.

Note:

Each Oracle Fusion Applications module is written in one of the following code languages: Java, SOA, PL/SQL, or C. The amount and type of information automatically included with an incident may vary depending on the coding language of the relevant application module.

In general, you can administer Oracle Fusion Applications without knowing which programming language implements particular modules. However, in order to monitor and diagnose all types of incidents successfully, it is important to become familiar with all of the incident configuration settings and the types of diagnostic dump files that are automatically provided for different kinds of incidents.

Oracle Fusion Applications incidents may also contain diagnostic log excerpts and diagnostic test results that are specific to Oracle Fusion Applications. Before you transmit incident information to Oracle Support personnel, you can add more information (such as screen shots or more log file information) to incidents that were created either automatically or manually.

17.1.1 Relationships Between Incidents, QuickTrace Dumps, Log Files, and Diagnostic Tests

Incidents are designed to work along with other diagnostic features in your Oracle Fusion applications:

• Error-Handling: The Oracle Fusion Applications code that handles errors is designed so that errors that need quick attention automatically create incidents.

• Diagnostic Tests: Oracle developers create tests that you can use to help diagnose and resolve Oracle Fusion application problems. A diagnostic test may or may not be associated with a particular error message. If an Oracle Fusion application handles a particular error in a way that triggers the creation of an incident, then any diagnostic tests that are associated with the error message for the incident run automatically. The test results are associated with the incident and the identity of the user who received the error message is recorded.

• Log files: By default, whenever an Oracle Fusion Applications incident is created automatically, an INCIDENT_ERROR level entry is recorded in the standard Oracle Fusion Applications log file and a copy of that log file is included with the incident. If the incident occurs in Oracle Fusion Middleware code two different kinds of log information are automatically included with the incident: a dump file containing all log entries for the relevant execution context ID (ECID) and either a complete log file or a log file excerpt starting five minutes prior to the incident.

  For some incidents, you may find all the information that you need to resolve the problem in the log or dump files automatically included with the incident. In other cases, you may need to configure your system to log more detailed information, attempt to reproduce the problem, and then use the more detailed information in the log to diagnose the problem. If you work with Oracle Support to resolve a problem, log file excerpts are among the kinds of information that you can add to incidents before sending them to Oracle.

• QuickTrace: If you need to troubleshoot a problem that has generated an incident, and if the information in the relevant Oracle Fusion application log file is insufficient to resolve the problem, then a QuickTrace dump file included with the incident information may supply you with the additional information you need. If so, you can avoid increasing the amount of information to be logged and then attempting to reproduce the problem. (In general, you should increase the level of detail that you collect in your Oracle Fusion Applications logs only when absolutely necessary, as collecting additional information may decrease system performance.)

  By default, QuickTrace continuously records a specified level of log detail in an area of memory, generally at a more granular level of detail than is recorded in Oracle Fusion application log files. The memory that QuickTrace uses is recycled on an ongoing basis, with the oldest information being overwritten first. Because QuickTrace does not format information and because it writes to memory instead of to a log file, it can gather operational information continuously without significantly affecting system performance. The information that QuickTrace stores in memory is written to disk only when an incident occurs or when an administrator manually dumps the contents of a QuickTrace buffer.

  Note:

  The same commands in Oracle Fusion Applications code cause messages to be logged both in Oracle Fusion application diagnostic log files and in QuickTrace dumps. However, your system may gather different amounts of detail for the two kinds of output. At default setting levels, QuickTrace stores much more detailed information than diagnostic log files do.

• Version Information: If you request assistance from Oracle Support to resolve an incident, you may be asked to provide information about the Oracle product versions you are running along with the incident information. In your Oracle Fusion applications, you can obtain some product version information by selecting About Applications from the Help menu. Additional version information can be obtained from Remote Diagnostic Agent (RDA) reports. For more information about RDA, see the "Generating an RDA Report" section in the Oracle Fusion Middleware Administrator's Guide.

You may also find it helpful to view the performance metrics provided by the Dynamic Monitoring Service (DMS). For more information, see the chapter about the Oracle Dynamic Monitoring Service in the Oracle Fusion Middleware Performance and Tuning Guide.

17.1.2 Introduction to the Oracle Fusion Middleware Diagnostic Framework

The Oracle Fusion Middleware Diagnostic Framework provides most of the technologies for creating and handling incidents for Oracle Fusion Applications. For information about the Oracle Fusion Middleware Diagnostic Framework, see the "Diagnosing Problems" chapter in the Oracle Fusion Middleware Administrator's Guide.

17.1.3 Standard Incident Administration Tasks and Tools

The following administrative tasks are part of administering incidents:

• Adjusting configuration settings for incidents

• Using Oracle Enterprise Manager Fusion Applications Control to monitor for incidents that are created automatically

• Creating incidents manually when needed

• Using incident information to resolve problems

• Adding additional diagnostic information (such as log file excerpts or screen shots) to existing incidents

• Packaging incident information and sending it to Oracle Support as part of obtaining assistance in resolving problems

The standard tools that you use to complete these tasks include:

• Oracle Enterprise Manager Fusion Applications Control user interface (for configuring incident settings)

• Log Viewer (part of Application Server Control, for viewing log files that you may add to incidents)

• adrci (Automatic Diagnostic Repository Command Interpreter, for viewing and packaging incidents, provided with each instance of the Automatic Diagnostic Repository (ADR) in Oracle Fusion Applications, Oracle Fusion Middleware, and Oracle Database)

  For more information about using adrci, see Section 17.4.2, the "ADCRI Command-Line Utility" section in the Oracle Fusion Middleware Administrator's Guide, and the "ADRCI: ADR Command Interpreter" chapter of Oracle Database Utilities.

• Oracle Weblogic Scripting Tool (WLST) commands related to incidents provided by the Oracle Fusion Middleware Diagnostic Framework.

  For more information about these commands, see the "WLST Commands for Diagnostic Framework" section in the Oracle Fusion Middleware Administrator's Guide.

• SQL Trace (for assessing the efficiency of the SQL statements that Oracle Fusion applications are running)

  For more information about SQL Trace, see Section 12.2.4.6.

17.2 Configuring Log, QuickTrace, and Incident Settings for Troubleshooting

You may find it helpful to change log profile options or QuickTrace properties to gather more information during troubleshooting.

Ordinarily, the default configuration settings for incidents are appropriate for both normal operations and troubleshooting. It is possible to disable incident creation by using configuration settings, but it is recommended that you use the default incident configuration values unless Oracle Support specifically recommends changes to these settings.

This section contains the following topics:

• Assisting Users in Gathering Data Using Troubleshooting Options

• Configuring the Oracle Fusion Middleware Diagnostic Framework

• Precedence and Log Settings for Troubleshooting

• Default System Settings for Incident Creation and QuickTrace

• Adjusting Incident Settings for Troubleshooting

• When Changes to Incident Configuration Settings Take Effect

• Adjusting QuickTrace Configuration Settings for Troubleshooting

• Adjusting Standard Log Levels for Troubleshooting

• Adjusting Log Settings for Oracle Fusion Incentive Compensation Batch Jobs for Troubleshooting

• Adjusting Oracle Fusion General Ledger Log Settings for Troubleshooting

• Adjusting Oracle Fusion Receivables AutoInvoice Log Settings for Troubleshooting

• Disabling Logging of a Particular Message

• Logging More Detailed Information for a Particular Code Module

• Configuring and Using Profile Options for Troubleshooting

17.2.1 Assisting Users in Gathering Data Using Troubleshooting Options

Oracle Fusion Applications provides a Troubleshooting Options dialog box that you can display by choosing the Troubleshooting Options command in the Oracle Fusion Applications Help > Troubleshooting menu. The Troubleshooting Options dialog box provides ways to adjust the following kinds of settings for the current Oracle Fusion Applications user:

• Database trace (optionally capturing bind variables or wait events)

  For more information about Database Trace, see the chapter about using application tracing tools in the Oracle Database Performance Tuning Guide.

• PL/SQL profiler

  For more information about PL/SQL profiler, see the chapter about the PL/SQL hierarchical profiler in the Oracle Database Advanced Application Developer's Guide.

• Applications logging Severity Level

• Applications logging Modules

However, the Help > Troubleshooting menu displays the Troubleshooting Options command only for end users who have a job role that is mapped to the following three duty roles:

• Supportability Level Management Duty (CRM) (FND_SET_SUPPORTABILITY_LEVEL_DUTY_CRM)

• Supportability Level Management Duty (FSCM) (FND_SUPPORTABILITY_LEVEL_MANAGEMENT_DUTY_FSCM)

• Supportability Level Management Duty (HCM) (FND_SUPPORTABILITY_LEVEL_MANAGEMENT_DUTY_HCM)

By default, only the following job roles are matched to the Supportability Level Management Duty duty roles:

• Application Administrator (FND_APPLICATION_ADMINISTRATOR_JOB)

• Application Diagnostics Administrator (FND_DIAG_ADMINISTRATOR_JOB)

These two job roles have broad administrative privileges, so it is recommended that you assign them only to administrative personnel. To give other users access to the Troubleshooting Options dialog box, it is recommended that you find or create a job role that is specifically intended for troubleshooting and that is mapped to the three Supportability Level Management Duty duty roles. Once the troubleshooting job role is available, you can assign users to that job role as needed.

To grant a user access to the Troubleshooting Options command and dialog box and assist the user to gather data for troubleshooting:

1. Use Oracle Identity Manager to find or create a job role (also called an external role) that you can assign to users who need access to the Troubleshooting Options command and dialog box.

   For more information, see the "Creating Roles" section in the Oracle Fusion Middleware User's Guide for Oracle Identity Manager.

2. If you created a job role in Step 1, use Oracle Authorization Policy Manager to map the job role to the following three duty roles:

   • Supportability Level Management Duty (CRM) (FND_SET_SUPPORTABILITY_LEVEL_DUTY_CRM)

   • Supportability Level Management Duty (FSCM) (FND_SUPPORTABILITY_LEVEL_MANAGEMENT_DUTY_FSCM)

   • Supportability Level Management Duty (HCM) (FND_SUPPORTABILITY_LEVEL_MANAGEMENT_DUTY_HCM)

   Note:

   To make sure that the job role you create can be used for troubleshooting any Oracle Fusion application, be sure to map the job role to all three of the Supportability Level Management Duty duty roles. Each of these duty roles provides the needed functionality for a different application stripe, where an application stripe is a subset of policies in the policy store that is used by a particular application or group of applications.

   For more information about mapping job roles to duty roles, see the "Mapping External Roles to an Application Role" section in the Oracle Fusion Middleware Oracle Authorization Policy Manager Administrator's Guide (Oracle Fusion Applications Edition). A job role corresponds to an external role, and a duty role corresponds to an application role.

3. For any user who currently needs access to the Troubleshooting Options dialog box, use Oracle Identity Manager to assign the job role that you found or created in Step 1 to the user.

   For more information, see the "Assigning Members to a Role" section in the Oracle Fusion Middleware User's Guide for Oracle Identity Manager.

4. To make sure that the user has access to the new job role assignment, direct the user to log out from their Oracle Fusion application and log back in.

5. Direct the user to choose Troubleshooting Options from the Help > Troubleshooting menu and to modify Troubleshooting Options settings as needed for the specific problem you are troubleshooting:

   1. If you want to enable all of the available troubleshooting options, direct the user to select the Enable all checkbox.

   2. If you want to enable Database Trace, direct the user to select the Database trace checkbox.

      For more information about Database Trace, including the options of capturing bind variables or wait events, see the chapter about application tracing tools in the Oracle Database Performance Tuning Guide.

   3. If you want to enable the PL/SQL profiler, direct the user to select PL/SQL profiler.

      For more information, see the section about using the PL/SQL hierarchical profiler in the Oracle Database Advanced Application Developer's Guide.

   4. If you want to log information for the user's actions at a level of detail that is different from the logging done for other users at the same site, direct the user to select the Applications logging checkbox and to select the appropriate level of detail from the Severity Level drop-down list.

      This user action is equivalent to an administrator setting the value of the AFLOG_LEVEL profile option for that user. For more information about the available logging severity levels, see Section 13.7.3.

   5. If you want to limit the logging for the user's actions to operations that use one or more particular code modules, and if you know the name of those code modules, direct the user to enter the module names in the Modules field, separating the names using commas without spaces.

      The user can use the % wildcard character to represent part of a module name or to represent all module names.

   6. Direct the user to click OK.

      Note:

      Users who change the Applications logging Severity Level may need to log out from their Oracle Fusion application and log in again in order to have the changes take effect. It is recommended that you direct the user to do so at this point in the process.

6. Direct the user to try to reproduce the problem.

7. Verify that the additional information you wanted was collected.

8. Unless the user needs ongoing access to the Troubleshooting Options dialog box, use Oracle Identity Manager to revoke the user's assignment to the job role that you found or created in Step 1.

17.2.2 Configuring the Oracle Fusion Middleware Diagnostic Framework

As part of the infrastructure for creating and handling incidents, the Oracle Fusion Middleware Diagnostics Framework includes settings that you can configure to affect functionality areas such as the following:

• Whether multiple incidents can be created for the same problem within a particular time period, and, if so, how many (flood control)

• The total amount of space allocated for storage of incident information (older incidents are purged automatically when the allocated space limit is reached)

For more information, see the "Configuring the Diagnostic Framework" section in the Oracle Fusion Middleware Administrator's Guide.

17.2.3 Precedence and Log Settings for Troubleshooting

During troubleshooting in a production environment, you will normally use profile options to increase or decrease the amount of information gathered in standard Oracle Fusion Applications log files. Where feasible, you will change profile option values at the USER level, rather than at the SITE level. This allows you to log more detailed information for a specific user who attempts to reproduce a problem, without cluttering your log file with unnecessarily detailed information for other users.

For Oracle Fusion applications that are written in Java or SOA, it is also possible for the odlLevel setting for the oracle.apps logger in the logging.xml configuration file to affect the level of detail that is logged. If both the odlLevel setting and the AFLOG_LEVEL profile option have defined values, then the minimum of those two values takes precedence.

Note:

Unless Oracle Support advises you to change the value of the setting, it is strongly recommended that you keep your odlLevel setting for the oracle.apps logger set to a value of All.

17.2.4 Default System Settings for Incident Creation and QuickTrace

The default profile option configuration settings for incident creation and QuickTrace profile options are shown in Table 17-1. For information about additional QuickTrace properties, see Section 17.2.7.

Table 17-1 Profile Options for Oracle Fusion Applications Incidents and QuickTrace

Profile Option Name (and Display Names)	Environment	Description	Possible Values or Example	Applicable Profile Hierarchy Levels	Default Value
AFLOG_INCIDENT_ENABLED (Incident Enabled)	Java, PL/SQL, C, and SOA	Enables or disables incident creation for Oracle Fusion Applications. (This profile option does not affect Oracle Fusion Middleware incident creation functionality.) A value of Y allows incident creation at runtime. A value of N prevents incident creation. This profile option is available for SITE but is not available for USER. AFLOG_INCIDENT_ENABLED is a new profile that has been introduced in Fusion to support application incident creation.	Y, N	Site	Y
AFLOG_QUICKTRACE_ENABLED (QuickTrace Enabled)	Java and SOA	Enables quick trace to occur at runtime.	Y, N	Site	Y
AFLOG_QUICKTRACE_LEVEL (QuickTrace Level)	Java and SOA	Specifies the minimum level for tracing. This setting is similar to the AFLOG_LEVEL profile option for standard Oracle Fusion Applications logging to log files, but the two settings operate independently.	1000 (SEVERE), 900 (WARNING), 800(INFO), 700 (CONFIG), 500 (FINE), 400 (FINER), 300 (FINEST) and 0 (OFF)	Site	500 (FINE)

17.2.5 Adjusting Incident Settings for Troubleshooting

By default, incident creation functionality for Oracle Fusion Applications is enabled, and it should ordinarily remain enabled. However, if Oracle Support directs you to disable incident creation, you can use the AFLOG_INCIDENT_ENABLED profile option to do so. You can use the Manage Administrator Profile Values task in the Setup and Maintenance work area to set the value of this profile option. For more information about this setting, see Section 17.2.4.

Incident creation functionality is also enabled by default for the Oracle Fusion Middleware Diagnostic framework. In addition, various Oracle Fusion Middleware profile options govern the details of how incidents are collected and stored. In general, troubleshooting will not require you to change these settings. For more information, see the "Configuring the Diagnostic Framework" section in the Oracle Fusion Middleware Administrator's Guide.

17.2.6 When Changes to Incident Configuration Settings Take Effect

Changes to incident profile options ordinarily take effect as follows:

• User Sessions: Changes to site level incident profile options take effect for any user session that is started after the setting has been changed. Changes to site level log file profile options do not affect any user sessions that are already in progress when the change is made.

• PL/SQL and C processes such as scheduled jobs: Changes to site level incident profile options take effect for any C or PL/SQL processes that are launched after the setting has been changed. Incident profile option changes do not affect C or PL/SQL processes that are already running.

17.2.7 Adjusting QuickTrace Configuration Settings for Troubleshooting

By default, QuickTrace functionality for Oracle Fusion Applications is enabled at the FINE logging level. QuickTrace should ordinarily remain enabled. However, if Oracle Support directs you to disable QuickTrace, or if you want to change the amount or organization of the information that QuickTrace collects, you can adjust the values of the following profile options either by using Fusion Applications Control or by using the Manage Administrator Profile Values task in the Setup and Maintenance work area:

• QuickTrace Enabled (AFLOG_QUICKTRACE_ENABLED profile option): Governs whether or not QuickTrace gathers any information.

• QuickTrace Level (AFLOG_QUICKTRACE_LEVEL profile option): Specifies the level of detail at which QuickTrace gathers information.

Note:

Because the QuickTrace Enabled and QuickTrace Level profile options apply at the site level, either all of the site's Managed Servers capture QuickTrace information in memory buffers or none of the Managed Servers do so. Similarly, the QuickTrace Level value sets the level of detail that QuickTrace gathers for all of the site's Managed Servers.

For more information about these settings, see Section 17.2.4.

In addition, Table 17-2 shows QuickTrace property settings that you can change using Fusion Applications Control.

Table 17-2 QuickTrace Properties

Property Name	Description	Server Restart Required	Default
Buffer Size (bufferSize)	Approximate size in bytes of each circular QuickTrace buffer. Each buffer's actual memory consumption may be less but not more than the specified value. By default, each WebLogic Server has a single COMMON QuickTrace buffer for storing multiple users' log records in memory. If you want store particular users' log records in individual buffers (one buffer for each user you specify), then you can allocate additional buffers of the same size by setting Enable User Buffer to true, restarting the server, and setting the value of User Names for Reserve Buffer to a comma-separated list of those users.	Yes	5242880
Enable User Buffer (enableUserBuffer)	When this property is set to false, each QuickTrace handler uses a single COMMON buffer to cache log messages for all users. When this property is set to true, each QuickTrace handler maintains an individual buffer for each user who is listed in the User Names for Reserve Buffer (reserveBufferUserIDs) property. Log messages for users who are not listed in that property are cached in the COMMON buffer. To determine in which buffer to place a particular log message, QuickTrace examines the log message for an application User Name value. If the log message does not include a User Name value, QuickTrace examines it for a User ID value provided by the server. If the log message does not include either a User Name or a User ID, then QuickTrace caches the message in the COMMON buffer.	Yes	false
User Names for Reserve Buffer (reserveBufferUserIDs)	Holds a list of application User Name or server User ID values separated by comma. When the Enable User Buffer (enableUserBuffer) property is set to true, for each listed application User Name or server User ID, any log messages associated with that user are cached in an individual QuickTrace user buffer. Log messages that are associated with other users are cached in the COMMON buffer. When the Enable User Buffer (enableUserBuffer) property is set to false, this property has no effect.	No	null

You can also change the properties in Table 17-2 by using the Oracle Weblogic Scripting Tool (WLST) configureLogHandler command. For more information, see the "configureLogHandler" section in the Oracle Fusion Middleware WebLogic Scripting Tool Command Reference.

To change QuickTrace profile option and property settings using Fusion Applications Control:

1. From the navigation pane, select a product family target such as Financials.

2. In the context pane, from the target type dropdown menu, choose Logs > Log Configuration.

3. In the Logging Profile Configuration dialog box, click the Site-Level tab.

4. Expand the Advanced Configuration area of the dialog box.

5. Expand the Java Settings portion of the Advanced Configuration area.

6. Expand the Other QuickTrace Settings portion of the Java Settings area.

7. In the Java Settings portion of the Advanced Configuration area, change the QuickTrace profile settings, as needed:

   1. To disable QuickTrace, clear the QuickTrace Enabled checkbox. To enable QuickTrace, make sure the QuickTrace Enabled checkbox is selected.

   2. To change the amount of information QuickTrace gathers, select a logging level from the QuickTrace Level drop-down list.

8. In the Other QuickTrace Settings portion of the Java Settings area, change the QuickTrace property settings, as needed:

   1. If you intend to use any QuickTrace property value that is not the default value listed in Table 17-2, select the Override Settings checkbox.

   2. In the Buffer Size field, enter the desired QuickTrace buffer size in bytes.

      Note:

      If you increase the value of the Buffer Size setting, QuickTrace uses more memory, which may affect system performance.

   3. If you want QuickTrace to store information in separate buffers for selected individual users, select the Enable User Buffer checkbox.

   4. In the User Names for Reserve Buffer field, enter comma-separated Oracle Fusion Applications user names for all of the users for whom QuickTrace should store information in individual buffers.

      Note:

      The values in the User Names for Reserve Buffer field are used only if the Enable User Buffer checkbox is selected.

9. Click Apply.

   Note:

   Changes to QuickTrace Enabled and QuickTrace Level settings take effect immediately for the user session in which the changes were made, but the changes do not affect other user sessions that are in progress when the changes are made. To use the changed settings for a different user session, log out from your Oracle Fusion application and log back in.

   For each particular Managed Server that runs an Oracle Fusion application, changes to the QuickTrace properties Buffer Size and Enable User Buffer do not take effect until the server is restarted.

   If a value of true is already in effect for the Enable User Buffer property, then changes to the User Names for Reserve Buffer property take effect as soon as you click Apply. Otherwise, you must set the Enable User Buffer property to true and restart the server in order to have changes to the User Names for Reserve Buffer property take effect.

17.2.8 Adjusting Standard Log Levels for Troubleshooting

If you encounter a situation in which an incident does not provide enough information to allow you to resolve the problem, then you may need to increase the standard logging level for the relevant Oracle Fusion application and the application user who will attempt to reproduce the problem. For information about setting the logging profile options, see Section 13.7.5.

17.2.9 Adjusting Log Settings for Oracle Fusion Incentive Compensation Batch Jobs for Troubleshooting

The logging functionality for certain Oracle Fusion Incentive Compensation batch jobs is separate from the standard logging functionality for Oracle Fusion Applications. By default, the following kinds of Oracle Fusion Incentive Compensation batch jobs do not write log entries:

• Calculation

• Classification

• Collection

• Crediting

• Rollup

For optimum performance and log file sizes, it is recommended that you use the logging functionality for these areas only when troubleshooting an existing problem.

To configure Oracle Fusion Incentive Compensation batch jobs to write nonstandard log entries:

1. Navigate to the Manage Messages task flow in the Setup and Maintenance work area.

2. In the Implementations section of the Tasks pane, click Manage Implementation Projects.

3. In the Manage Implementation Projects pane, use standard Oracle search techniques to locate the implementation project that includes Oracle Fusion Incentive Compensation, then click the project name in the search results.

4. In the Tasks Lists and Tasks table, expand the Define Incentive Compensation entry.

5. Expand the Define Incentive Compensation Shared Configuration entry.

6. Locate the Manage Incentive Compensation Profile Options row of the table and click the Go to Task icon.

7. In the Profiles section of the Tasks pane, click Manage Administrator Profile Values.

8. In the Search area of the Manage Profiles pane, select Incentive Compensation from the Application dropdown list and click Search.

9. In the Results table, locate and select the row for the CN_DEBUG profile option.

10. If the Enable Debug Mode Profile Values table contains a record for the Profile Level and Level Value for which you want to collect log entries, select the record and click Edit; otherwise, click New and fill in the appropriate values, as follows:

    • If you want to collect log entries for a specific user's Oracle Fusion Incentive Compensation batch jobs, you would edit or create a record in which Profile Level is set to User and Level Value is set to the user name of the specific user. Set Profile Value to Y.

    • If you want to collect log entries for all of the Oracle Fusion Incentive Compensation batch jobs at the site, you would edit or create a record in which Profile Level is set to Site and Level Value is blank. Set Profile Value to Y.

11. Click Save.

12. Start a new user session of Oracle Fusion Incentive Compensation and try to replicate the problem.

    For information about how to view the log entries, see Section 17.5.2.

17.2.10 Adjusting Oracle Fusion General Ledger Log Settings for Troubleshooting

In the Oracle Fusion Financials product family, some logging functionality for the Oracle Fusion General Ledger application is separate from the standard logging functionality for Oracle Fusion Applications. By default, the following kinds of Oracle Fusion General Ledger batch jobs do not write log entries:

• OpenPeriod

• Posting

• Translation

• Close Process - Create Income Statement Closing Journals

• Close Process - Create Balance Sheet Closing Journals

For optimum performance and log file sizes, it is recommended that you use the logging functionality for these areas only when troubleshooting an existing problem.

To configure the Oracle Fusion General Ledger application to write nonstandard log entries:

1. From the Setup and Maintenance work area, navigate to the Manage Administrator Profile Values screen.

2. From the Application drop-down list in the Search: Profile Option pane, select General Ledger.

3. In the Profile Display Name field, enter %Debug% and click Search.

4. In the Search Results table, select GL_DEBUG_MODE.

5. If the GL_DEBUG_MODE: Profile Values table contains a record for the Profile Level and Level Value for which you want to collect log entries, select the record and click Edit; otherwise, click New.

6. Fill in the appropriate values, as follows:

   • If you want to collect log entries for a specific user's Oracle Fusion General Ledger batch jobs, edit or create a record in which Profile Level is set to User and Level Value is set to the user name of the specific user. Set Profile Value to Y.

   • If you want to collect log entries for all of the Oracle Fusion General Ledger batch jobs at the site, edit or create a record in which Profile Level is set to Site and Level Value is blank. Set Profile Value to Y.

7. Click Save.

8. Ask an appropriate user to start a new user session of Oracle Fusion General Ledger and try to replicate the problem.

9. Inspect the relevant log file for new entries.

   If the value of the AFLOG_FILENAME profile option is set, then the value of the AFLOG_FILENAME profile option indicates the location of the log file.

   If the value of the AFLOG_FILENAME profile option is not set, then the information is logged to the location designated for Oracle Enterprise Scheduler log files. For information about viewing Oracle Fusion Middleware log files, including log files for Oracle Enterprise Scheduler, see the "Viewing and Searching Log Files" section in the Oracle Fusion Middleware Administrator's Guide.

10. To help manage the amount of disk space used by the logged information, set the value of GL_DEBUG_MODE to N or delete any row of the GL_DEBUG_MODE: Profile Values table that you no longer need.

    For more information about managing the amount of disk space used by Oracle Fusion General Ledger log entries, see Section 13.7.4.3.

17.2.11 Adjusting Oracle Fusion Receivables AutoInvoice Log Settings for Troubleshooting

In the Oracle Fusion Financials product family, logging functionality for the AutoInvoice portion of the Oracle Fusion Receivables application is separate from the standard logging functionality for Oracle Fusion Applications. The amount of information that is logged for AutoInvoice depends on the value of the Log File Message Level system option setting for each business unit. The information is placed in the standard log file for Oracle Enterprise Scheduler.

Note:

For optimum performance and log file sizes, it is recommended that you keep the Log File Message Level system option set to the lowest value that meets your everyday needs, and increase the value of the setting only when troubleshooting an existing problem.

The available values for the Log File Message Level setting are the integers 0 through 5. The value of 0 is recommended for day-to-day business needs. The value of 5 provides the maximum amount of information for troubleshooting.

Specifically, if you set Log File Message Level to a value of 0, the following kinds of information are logged:

• Product Version

• Program Name

• AutoInvoice Start Time

• AutoInvoice Concurrent Request Arguments

• Error and Warning Messages

• AutoInvoice End Time

• AutoInvoice Logical Steps

If you set Log File Message Level to a value of 1, the following additional information is gathered:

• Time-Stamped function labels

If you set Log File Message Level to a value of 2, the following additional information is gathered:

• Sizes of Allocated Arrays

• Dynamic SQL Statements

• Number of Rows Updated, Inserted and Deleted

If you set Log File Message Level to a value of 3, the following additional information is gathered:

• Method IV SQL Array Values

If you set Log File Message Level to a value of 4, the following additional information is gathered:

• Values of all variables that are used to call FND or Tax routines

If you set Log File Message Level to a value of 5, the following additional information is gathered:

• Bad lines

• Rejected lines

To change the amount of information that the Oracle Fusion Receivables application logs for the AutoInvoice functionality area in the current business unit:

1. In the Oracle Fusion Receivables application, select Setup and Maintenance from the Navigator menu.

2. Complete the following substeps to navigate to the Edit System Options screen for the Manage Receivables System Options task.

   1. Expand the Search: Tasks pane.

   2. Enter Manage Receivables System Options in the Name field and click Search.

   3. In the Manage Receivables System Options row of the Search Results table, click Go to Task.

   4. In the Search area of the Manage Receivables System Options screen, select Business Unit from the drop-down list, enter the name of the business unit for which you are troubleshooting, and then click Search.

      Alternately, you can click Search without specifying a business unit name to display a list of the business units you can choose from.

   5. In the Search Results table, click the name of the business unit for which you are troubleshooting.

3. In the Edit System Options screen, scroll down to display the AutoInvoice area of the screen, and then set Log File Message Level to the value that corresponds to the amount of information you want to gather.

   For example, to gather the maximum amount of information for troubleshooting, set the value to 5. To gather the normal amount of information for day-to-day operations, set the value to 0.

4. Click Save to put the change into effect.

   Log entries for the AutoInvoice portion of Oracle Fusion Receivables are placed in the standard log file for Oracle Enterprise Scheduler. For information about viewing Oracle Fusion Middleware log files, including log files for Oracle Enterprise Scheduler, see the "Viewing and Searching Log Files" section in the Oracle Fusion Middleware Administrator's Guide.

   Note:

   After you have gathered the detailed information that you need, remember to repeat this procedure and return the Log File Message Level setting to the value that meets your day-to-day information needs.

17.2.12 Disabling Logging of a Particular Message

You may occasionally encounter a situation in which one of your Oracle Fusion applications repeatedly logs the same error message. In this situation, it is best to discover and correct the cause of the error. However, on rare occasions, Oracle Support may recommend that you use the FND_MESSAGES.LOGGABLE_ALERTABLE attribute to prevent the repeating message from being logged while efforts are underway to resolve the issue.

Caution:

Do not change the FND_MESSAGES.LOGGABLE_ALERTABLE attribute unless Oracle specifically advises you to do so, as this involves permanent loss of data.

Do not change the FND_MESSAGES.LOGGABLE_ALERTABLE attribute directly in the database. Instead, use the Manage Messages task flow.

To disable logging for a particular message:

1. Navigate to the Manage Messages task flow in the Setup and Maintenance work area.

2. In the Message Name field in the Search pane, enter the name of the error message that you want to temporarily stop logging.

3. Select the relevant Oracle Fusion application from the Application drop-down list.

4. If you know it, select the relevant code module name from the Module drop-down list.

5. Click Search.

6. Complete the following substeps to set the value of the FND_MESSAGES.LOGGABLE_ALERTABLE attribute to N for the error message that you want to temporarily stop logging:

   1. In the search results table, select the error message that you want to temporarily stop logging.

   2. Click the edit icon in the table header.

   3. In the Edit Message screen, clear the Logging Enabled checkbox.

   4. Click Save, Save and Create Another, or Save and Close to put the change into effect immediately:

17.2.13 Logging More Detailed Information for a Particular Code Module

If you contact Oracle Support for assistance in resolving a problem, the support representative may ask you to gather more detailed log information on the operations of one or more specific code modules in your Oracle Fusion applications. You can do this by setting the value of the AFLOG_MODULE profile option to one or more module names that Oracle Support specifies.

Caution:

Unless Oracle Support specifically advises you to change the value of the AFLOG_MODULE profile option at the Site level, be sure to adjust the value only for a particular user who will attempt to reproduce the problem you are investigating. Specifying one or more particular modules at the Site level prevents logging on any modules that are not specified.

For more information about using administrative screens to set the value of this profile option either for a specific user or at the Site level, see Section 13.7.5. For more information about having a specific user set the user's own profile option using Troubleshooting Options, see Section 17.2.1.

17.2.14 Configuring and Using Profile Options for Troubleshooting

You may find it useful to adjust one or more of the following profile options during troubleshooting:

• To log more detailed information about a particular user's operations, you can create or change the value of the AFLOG_LEVEL profile option for that user. For more information about setting this profile option, see Section 13.7.5.

• To log more detailed information for an entire site, you can create or change the value of the AFLOG_LEVEL profile option for the site. For more information about setting this profile option, see Section 13.7.5.

• To specify the Oracle Fusion Applications code modules for which standard log entries will be written at the level of detail specified by the AFLOG_LEVEL profile option for a particular user, you can set the AFLOG_MODULE profile option for the same user. (To minimize the number of unrelated log entries that will be placed in the log file during troubleshooting, it is recommended that you specify a particular module whenever feasible.) For more information about setting the AFLOG_MODULE profile option, see Section 13.7.5.

• To turn on nonstandard logging for troubleshooting Oracle Fusion Incentive Compensation, you can set the value of the application-specific CN_DEBUG profile option to Y, either for a single user or for an entire site. For more information about setting this profile option, see Section 17.2.9.

17.3 Configuring the Diagnostic Testing Framework for Troubleshooting

When you investigate an Oracle Fusion Applications problem, you may want to assign job roles to some users, temporarily, that will allow those users to run diagnostic tests. For information about how to do this, see Section 13.9.1.

There is generally no need to change other configuration settings in order to use the Oracle Fusion Applications Diagnostic Testing Framework for troubleshooting. When an incident is created, the Diagnostic Testing Framework automatically runs any Oracle Fusion Applications diagnostic tests that Oracle Fusion Applications developers have associated with the incident's particular error message. There is no configuration setting in the Diagnostic Testing Framework for turning off this automatic execution of diagnostic tests.

Note:

(Oracle Fusion Applications developers use the APPS_MSG_ID diagnostic test tag to associate particular error messages with appropriate diagnostic tests. You cannot delete or modify the values of diagnostic test tags that are supplied by Oracle.

17.4 Investigating, Reporting, and Solving a Problem

From time to time, you may receive a user complaint or an automatically generated incident notification about a problem with a particular Oracle Fusion application. For any problem that does not have an immediate and obvious solution, you can diagnose and resolve the problem more effectively by becoming familiar with incident functionality and by using a systematic process for your troubleshooting process.

This section contains the following topics:

• Process for Investigating, Reporting, and Solving a Problem

• Viewing Problems and Incidents

• Recovering from Incidents Generated During SOA Operations

• Creating and Packaging Incidents Manually

• Working with Automatically Created QuickTrace Dumps

• Working with Other Diagnostic Dumps

17.4.1 Process for Investigating, Reporting, and Solving a Problem

When you become aware of a problem with a particular Oracle Fusion application, you can use the following general strategy to help diagnose and address the problem. While any problem may have unique features, the information that you gather in this way is likely to be useful during the troubleshooting process.

To help diagnose and address the problem, use the following steps:

1. If an Oracle Fusion application user reports an error message to you, make a note of any specific information that the error message may have provided, such as:

   • Incident ID

   • Server Domain

   • Server Instance

   • Application Name

2. If no incident was generated, create an incident manually and then manually perform any relevant diagnostic dumps, assigning the dump results to the incident you created.

   Using incidents for all problems helps you organize the information that you gather during the troubleshooting process. For information about creating an incident manually, see Section 17.4.4.

3. Examine any information that was collected and stored in the incident directory, such as QuickTrace dumps and the results of any diagnostic tests that may have been run, whether the tests and dumps were run automatically or were created manually.

   For more information about incident directory locations and how to work with QuickTrace dumps, see Section 17.4.2 and Section 17.4.5.

4. View recent information in the standard log files for all of the following:

   • Oracle Fusion Applications

   • Oracle Business Intelligence

   • Oracle Enterprise Scheduler Service

   • Service-Oriented Architecture (SOA)

   For more information about locating log files for Oracle Fusion Applications, see Section 13.6.3. For more information about locating log files for Oracle Fusion Middleware components, see the "Log File Location" table in the Oracle Fusion Middleware Administrator's Guide. For more information about locating log files for Oracle Enterprise Scheduler, see Section 5.15, Section 13.7.4.3, or Section 17.5.3.2.

5. Open the Diagnostic Dashboard and search for diagnostic tests that are associated with the application in which the problem occurred. Run the diagnostic tests that seem most likely to be relevant and inspect the results.

   For information about the individual diagnostic tests that are provided with this release, see the Oracle Fusion Applications Common User Guide in Oracle Fusion Applications Help.

6. If necessary, change the setting that governs the amount of information to be logged, try to replicate the problem, and inspect the newly logged information.

   The steps for increasing the amount of information to be logged may vary depending on the application involved:

   • For most Oracle Fusion applications, standardized logging mechanisms place information in log files that you can view from Oracle Enterprise Manager Fusion Applications Control. You can increase the amount of information to be gathered for a particular user or for a whole site.

   • If the problem occurred in an application that uses nonstandard mechanisms for logging, such as certain Oracle Fusion Incentive Compensation batch jobs, certain Oracle Fusion General Ledger batch jobs, or the AutoInvoice portion of the Oracle Fusion Receivables application, then you may need to turn on a logging facility that is normally disabled and specify the kind of information you want to gather.

17.4.2 Viewing Problems and Incidents

For Oracle Fusion Applications, Oracle Fusion Middleware, and Oracle Database, incidents are stored in the incident subdirectories of Automatic Diagnostic Repository (ADR) home directories. However, the location of the appropriate ADR home directory for a given incident depends on the environment from which the incident was created. In all cases, incidents are stored separately from log files, and you can use the Automatic Diagnostic Repository Command Interpreter (adrci) to work with incidents.

Paths to directories that contain incident information begin with the path to the ADR base directory, where domain is the full path to an Oracle Fusion Applications domain, including the name of the physical Oracle WebLogic Server domain:

(UNIX) domain/servers/managed_server_name/adr
(Windows) domain\servers\managed_server_name\adr

The ADR base directory path is followed by the path to one of several possible incident directories:

For an incident that involves Oracle Fusion Middleware code, the incident information is stored under the following location, where, domain_name is the name of the Oracle Fusion Applications domain and server_name is the name of the server where the incident occurred:

(UNIX) ADR_base_directory/diag/ofm/domain_name/server_name/incident
(Windows) ADR_base_directory\diag\ofm\domain_name\server_name\incident

For an incident that involves Java code or a SOA composite in an Oracle Fusion application, the incident information is stored under the following location, where application_name is the name of the Oracle Fusion application that was executing when the incident occurred:

(UNIX) ADR_base_directory/diag/ofm/fusionapps/application_name/incident
(Windows) ADR_base_directory\diag\ofm\fusionapps\application_name\incident

For an incident that involves PL/SQL code in an Oracle Fusion application, the incident information is stored under the following location:

(UNIX) ADR_base_directory/diag/ofm/db_server_name/database_instance/incident
(Windows) ADR_base_directory\diag\ofm\database_server\database_instance\incident

For an incident that involves Oracle Database code, the incident information is stored under the following location:

(UNIX) ADR_base_directory/diag/rdbms/db_server_name/database_instance/incident
(Windows) ADR_base_directory\diag\rdbms\database_server\database_instance\incident

For an incident that involves C code in an Oracle Fusion application, the incident information is stored on the Java application server that runs C batch jobs using Oracle Enterprise Scheduler. On that server, the incidents are stored under the following location, where ess_hosted_app_name is the name of the Oracle Fusion application that launches batch jobs:

(UNIX) ADR_base_directory/diag/ofm/fusionapps/ess_hosted_app_name/incident
(Windows) ADR_base_directory\diag\ofm\fusionapps\ess_hosted_app_name\incident

The exact name of the application that launches batch jobs depends upon which product family is in use. Typical values include FinancialsEssApp, HcmEssApp, CrmEssApp, ProjectFinancialsEssApp, ProcurementEssApp, and ScmEssApp. You can determine the correct ess_hosted_app_name name for your Oracle Fusion application by using Oracle Enterprise Manager Fusion Applications Control for the application. In the navigation pane, if you expand the folder that is labeled with the product family name and then expand the Fusion Applications folder, the ess_hosted_app_name is the name under Fusion Applications that contains EssApp.

To view incidents:

1. Log in to the server that contains the Automatic Diagnostic Repository (ADR) that was mentioned in any incident notifications you received.

   This may be a repository for either Oracle Database or Oracle Fusion Middleware. The notification may be either a message received by an Oracle Fusion Applications user or an alert email that Oracle Enterprise Manager sent to designated administrators.

2. At an operating system prompt, navigate to the directory that contains a copy of the Automatic Diagnostic Repository Command Interpreter (adrci) that is associated with the ADR home for the incident.

   • If the incident notification referred to an Automatic Diagnostic Repository for Oracle Database, navigate to DB_ORACLE_HOME/bin (for UNIX) or DB_ORACLE_HOME\bin (for Windows).

   • If the incident notification referred to an Automatic Diagnostic Repository for Oracle Fusion Middleware, navigate to MW_HOME/wlserver_versionNumber/server/adr (for UNIX) or MW_HOME\wlserver_versionNumber\server\adr (for Windows).

3. At the prompt, enter adrci.

4. Determine where the incident information is stored.

   1. At an adrci prompt, enter show base to display the path to the Automatic Diagnostic Repository (ADR) base directory.

   2. At an adrci prompt, enter show homes to display the path to the ADR home directory, under the ADR base directory.

   3. Make a note of the location of the incident directory, which is a subdirectory of the ADR home directory.

5. At an adrci prompt, enter show incident.

   This command lists all the incidents that have been created in the incident subdirectory of the ADR home directory.

6. If you already know the ID number of the incident you are investigating, skip to Step 8.

   Otherwise, inspect the appropriate diagnostic.log file and make a note of either the incident_ID value or the incidentCustomId value in the relevant log message.

   Note:

   Due to a technical limitation, incidents that are created from PL/SQL code log incidentCustomId values rather than incident_ID values.

   For more information about viewing diagnostic.log files, see Section 13.6.

7. If the diagnostic.log file supplies the relevant incident_ID value, skip to Step 8.

   Otherwise, enter the following command at an adrci prompt, substituting the appropriate incidentCustomId value, and inspect the output for the incident_ID value:

   show incident -mode detail -p "ERROR_ARG1='incidentCustomID'";
   

8. To view detailed information about the incident for which you now know the incident ID number, enter the following command at an adrci prompt, substituting the appropriate incident ID number:

   show incident -mode DETAIL -p "incident_id=incident_id"
   

9. To display a list of the dump files that are associated with the incident_ID incident, enter the following command at an adrci prompt:

   show tracefile -i incident_ID
   

10. To view information in a particular dump file for a particular incident, enter one of the following commands at an adrci prompt:

    • To view a copy of the information in your default editor, enter:

      show trace filename
      

    • To view the information without using an editor:

      show trace filename -term
      

17.4.3 Recovering from Incidents Generated During SOA Operations

In some cases in which an incident is generated during an operation that involves a SOA composite, the application code may let you manually recover the operation and resume processing at the point where the incident occurred.

To determine whether you can recover or resume processing of an interrupted SOA composite operation:

1. Log on to the server where the incident was logged as a user that has the Oracle Fusion Middleware role of Administrator.

2. Navigate to the directory where the information about the incident is stored.

   For information about typical locations for incidents, see Section 17.4.2.

3. Inspect the readme.txt file in the incident directory and make a note of the composite_instance_id and composite_name values.

   For example, the readme.txt file might include the following lines:

   composite_instance_id: 10009
   composite_name: FinApInvTransactionsInvoiceApprovalComposite
   

4. In the navigation pane of Oracle Enterprise Manager Fusion Applications Control, expand the hierarchical tree as needed to locate and select the SOA composite that was mentioned in the readme.txt file for the incident.

   For example, you might expand the Farm_soa_domain_new listing, and then expand SOA, and then expand soa_infra (soa_server1) and then expand default and finally select FinApInvTransactionsInvoiceApprovalComposite.

5. If it is not already selected, click the Dashboard tab.

6. In the Recent Instances table in the content pane, click the Instance ID value that matches the composite_instance_id value that you saw in the log.

   The Flow Trace screen appears.

7. If you have not already done so, use the information on the Flow Trace screen to determine how to correct the problem that generated the incident, then repeat steps as needed to return to the Flow Trace screen.

   For more information about troubleshooting SOA operations, see Chapter 21.

8. Inspect the Recovery column of the Faults table for a row that contains a Recoverable link.

   • If the Recovery column contains a Recoverable link, then you may be able to resume processing of the SOA operation at the point where it previously failed. Click Recoverable to display more details about the recoverable instance and continue to Step 9.

   • If the Recovery column does not contain a Recoverable link, then the SOA operation is not recoverable. Do not proceed with these instructions. Instead, have an appropriate Oracle Fusion Applications user restart the sequence of operations that led to the incident, beginning as close to the point of failure as is practical.

9. In the list of faults for the instance in question, select the row that contains the word Recoverable in the Recovery column, and then click Recover in the screen area below the list.

10. In the Confirmation dialog box, click Yes.

    If the recovery process succeeds, the following message is displayed:

    Action completed successfully.
    

    Click OK to dismiss this message, which marks the completion of the recovery task.

    If the recovery process does not succeed, the list of faults with the word Recoverable in the Recovery column remains visible. This indicates that you have not succeeded in addressing the underlying cause of the original problem. Double-check your fix for the problem or try another approach. When you believe that you have addressed the problem successfully, return to the Flow Trace screen and repeat Step 8 through Step 10.

17.4.4 Creating and Packaging Incidents Manually

If you discover an issue with your Oracle Fusion applications that you want to gather data about, and if the system has not created an incident for it automatically, then it is recommended that you create an incident for it, yourself. For more information, see the "Creating an Incident Manually" section in the Oracle Fusion Middleware Administrator's Guide.

After creating the incident, you can manually add files to it. For example, you might want to add Oracle Fusion Applications diagnostic test results, Oracle Fusion Middleware diagnostic dumps, excerpts from downloaded log files, and Remote Diagnostic Agent (RDA) reports to the incident. You might also want to add a Readme.txt file containing information for Oracle Support. After you add files to an incident, you can package the incident into a zip file for transmission to Oracle Support. For more information about adding files to an incident and packaging the incident into a zip file, see the "Packaging an Incident" section in the Oracle Fusion Middleware Administrator's Guide.

17.4.5 Working with Automatically Created QuickTrace Dumps

As part of assisting you with resolving incidents, Oracle representatives often examine any dump files that were automatically created when the incident occurred, including QuickTrace dump files.

When a QuickTrace dump file is automatically created as part of an incident, the file is stored in the incident directory along with any other dump files that are related to that incident. QuickTrace dump file names begin with odl.quicktrace.

This section contains the following topics:

• Viewing Automatically Created QuickTrace Dump Files Using Oracle Weblogic Scripting Tool

• Viewing Automatically Created QuickTrace Dump Files Using the Automatic Diagnostic Repository Command Interpreter (adrci)

17.4.5.1 Viewing Automatically Created QuickTrace Dump Files Using Oracle Weblogic Scripting Tool

You can use the following standard Oracle Weblogic Scripting Tool commands to view information about automatically created QuickTrace dump files:

• Use the listIncidents command to view a list of incidents.

• Use the showIncident command to view details about a particular incident, including a list of dump files associated with that incident.

• Use the getIncidentFile command to view a specified dump file such as a QuickTrace dump file.

For more information about using these commands, see the "Viewing Problems and Incidents" section and the "Working with Diagnostic Dumps" section in the Oracle Fusion Middleware Administrator's Guide.

17.4.5.2 Viewing Automatically Created QuickTrace Dump Files Using the Automatic Diagnostic Repository Command Interpreter (adrci)

You can use the following standard Automatic Diagnostic Repository Command Interpreter (adrci) commands to view information about automatically created QuickTrace dump files:

• Use the show homes command to view a list of Automatic Diagnostic Repository (ADR) home directories. Depending on the configuration of your system, QuickTrace dumps may be located under one or more of these ADR home directories.

• Use the set home directory_path command to indicate which respository you want to work with, replacing directory_path with one of the locations from the output of the show home command.

• Use the show incdir -i incdir_incident_number command to view a list of files associated with the incident that incident_number identifies, including any QuickTrace dump files for that incident. QuickTrace dump file names start with odl_quicktrace and end with .dmp.

• Use the show tracefile -i incident_number command to view a list of just the dump files associated with the incident that incident_number identifies.

• Use the show trace filename command to view the contents of the QuickTrace dump file that you specify.

For more information about using adrci, see Section 17.4.2 and the chapter about the ADR Command Interpreter in Oracle Database Utilities.

17.4.6 Working with Manually Created QuickTrace Dumps

Oracle Support may occasionally ask you to perform a manual dump of QuickTrace buffers as part of work to resolve a problem that did not generate an incident automatically.

To create a QuickTrace dump manually:

1. Decide whether you want to associate the QuickTrace dump file with an existing incident at the time when you perform the dump:

   • If you want to associate a new QuickTrace dump file with an existing incident at the time when you perform the dump, make a note of the relevant incident ID. This can either be an automatically created incident or a manually created incident. For more information about creating incidents manually, see Section 17.4.4.

   • If you do not want to associate a new QuickTrace dump with an incident at the present time, decide where you will store the dump file.

2. Use the Oracle Weblogic Scripting Tool executeDump command to perform an odl.quicktrace dump, choosing appropriate syntax for your needs:

   • If you want to associate the dump file with an existing incident, use the id argument. This allows you to retrieve the dump file, later, using the getIncidentFile argument

   • If would rather associate the dump file with an incident at a later time, use the outputFile argument to specify an output file location at the time when you execute the dump.

   For more information about using the executeDump command and its arguments, see the "Working with Diagnostic Dumps" section in the Oracle Fusion Middleware Administrator's Guide.

17.4.7 Working with Other Diagnostic Dumps

In some troubleshooting situations, you may find it useful to locate or obtain Oracle Fusion Middleware diagnostic dumps such as Oracle WebLogic Server diagnostic images, Java Virtual Machine thread dumps, and Dynamic Monitoring Service metric dumps.

Oracle Fusion Applications incidents that are generated from Java code automatically execute these kinds of dumps and include the results in the incident data. Oracle Fusion Middleware diagnostic dumps are not available for Oracle Fusion Applications incidents that are generated from PL/SQL or C code.

If you troubleshoot an Oracle Fusion Applications problem that did not generate an incident automatically, it is recommended that you create an incident manually, then execute diagnostic dumps manually, and examine the dump files for information that may help you to resolve the problem. If you need to work with Oracle Support to resolve the problem, it is recommended that you include manually executed dump results in the incident data package that you send to Oracle for review. For information about how to list available diagnostic dumps, view dump descriptions, and manually execute selected dumps for Oracle Fusion Middleware, see the "Working with Diagnostic Dumps" section in the Oracle Fusion Middleware Administrator's Guide.

17.5 Troubleshooting Using Log Settings

You can use log settings to increase the amount of information that a particular Oracle Fusion application gathers about its own operations. This is particularly useful with problems that are easily reproducible—you can change log settings, reproduce the problem, then return log settings to their normal levels so that you do not use disk space for unnecessarily detailed information that is not relevant to the problem.

This section contains the following topics:

• Troubleshooting Using Standard Log Files

• Viewing Special Log Output for Oracle Fusion Incentive Compensation

• Correlating Log Messages Across Log Files and Components

• Downloading Log Files

When you are troubleshooting a problem in an Oracle Fusion Applications environment, you may also find it helpful to use the logging capabilities of Oracle Fusion Middleware. At times, Oracle Support may ask you to use certain of these capabilities when attempting to reproduce a problem, particularly if you have a nonproduction environment available for use in testing.

17.5.1 Troubleshooting Using Standard Log Files

Most code modules in Oracle Fusion Applications use a standard diagnostic logging framework in which profile options determine how much information is logged for a particular site or a particular user. For information about adjusting site-level profile options or adjusting user-level profile options as an administrator, see Section 13.7.5. For information about allowing selected users to adjust the level of information that is logged for their own sessions, see Section 17.2.1.

17.5.2 Viewing Special Log Output for Oracle Fusion Incentive Compensation

For information about configuring the special logging functionality for certain Oracle Fusion Incentive Compensation batch jobs, see Section 17.2.9. After you have enabled this functionality and attempted to replicate the Oracle Fusion Incentive Compensation problem that you are troubleshooting, you must perform some database queries to view the resulting log entries. For these queries, you must specify the name of the Oracle Fusion Incentive Compensation process that you ran when attempting to replicate the reported problem. The process names that you can use in the query are as follows:

• CALCULATION MAIN PROCESS

• CLASSIFICATION

• COLLECT_STATISTICS

• Crediting

• GEN_RULE_ENGINE

• GET_WINNERS

• PopulateSrpBatches

• PopulateSrpTables

• REVERT TRANSACTIONS

• ROLLUP

• Rollup within Crediting

Note:

These process names are case-sensitive. For more information about how Oracle Fusion Incentive Compensation processes operate, see product-specific documentation in Oracle Fusion Applications Help.

To view special log entries for Oracle Fusion Incentive Compensation batch jobs:

1. Use a SQL client to connect to Oracle Database and log in as a user that has permission to view the Fusion database schema.

2. Execute the following SQL command, replacing process_name with the name of the Oracle Fusion Incentive Compensation process that you ran when attempting to replicate the reported problem:

   select process_audit_id from cn_process_audits_all
   where process_type = process_name
   order by creation_date desc;
   

3. Make a note of the process audit ID that is returned in the first row of the output.

4. Execute the following SQL command, replacing selected_process_audit_ID with the value that you noted in Step 3.

   select message_type, message_text from cn_process_audit_lines_all
   where process_audit_id = selected_process_audit_ID
   order by process_audit_line_id asc;
   

5. If you have not already done so, place the output of Step 4 in a file that you can inspect with a text editor or send to Oracle Support if requested.

6. When you are finished gathering log entries, reset the value of the CN_DEBUG profile option setting to N.

   For information about setting the value of the CN_DEBUG profile option, see Section 17.2.9.

17.5.3 Correlating Log Messages Across Log Files and Components

Sometimes, as part of the process of troubleshooting Oracle Fusion Applications, you may need to find all of the log entries that result from a particular action. If the action involves multiple Oracle Fusion applications or multiple Oracle Fusion Middleware components, then the log entries are placed in multiple log files. You can use certain values related to the particular action to locate the relevant log messages in those log files.

This section contains the following topics:

• Values for Correlating Log Messages

• Correlating Log Messages for Oracle Enterprise Scheduler Jobs

• Correlating Log Messages for SOA Services and BPEL Processes

• Correlating Log Messages for Other Process Types

17.5.3.1 Values for Correlating Log Messages

You can use a particular action's Execution Context Identifier (ECID) and Relationship Identifier (RID) values to locate the log entries that result from that action.

A unique ECID value is assigned to each user request. However, depending on what is involved in the request, a log entry that is related to that request may have the original ECID of the request, or a different ECID of a child process spawned by the original request. For example, if you submit a job to Oracle Enterprise Scheduler, the request to schedule the job and the job, itself, always have different ECIDs.

For a particular ECID, if the request requires processing by multiple components, by multiple modules within a component, or using multiple threads, then a unique RID value may be assigned when the work passes to a new thread or process.

In log entries, the ECID and RID values are typically separated by a comma. ECID values can include alphanumeric characters, colons, periods, and hyphens. RID values can include numerals and colons. The RID value indicates the relationship of the current thread or process to its creator. The first RID created for a request is RID = 0. Each shift in generation is represented by a colon. For example, the RID value for the seventh child process of the third child process of the root context for a request is represented as 0:3:7.

17.5.3.2 Correlating Log Messages for Oracle Enterprise Scheduler Jobs

When a job is submitted to Oracle Enterprise Scheduler, the request to schedule the job and the job, itself, always have different Execution Context Identifiers (ECIDs).

The ECID for the job-scheduling request is displayed in the requesting user session at the time when the job is submitted. However, the ECID of the Oracle Enterprise Scheduler job, itself, is not assigned until the job starts, and it is not displayed in the requesting user session. To locate log entries related to the Oracle Enterprise Scheduler job, itself, you must use the job Request ID value.

To correlate log messages for Oracle Enterprise Scheduler jobs using Fusion Applications Control:

1. In the navigation pane, expand the Farm listing and then the Scheduling Services listing and then determine whether Scheduling Services contains Oracle Enterprise Scheduler clusters that contain individual Oracle Enterprise Scheduler servers, or individual Oracle Enterprise Scheduler servers without clusters.

2. To open the ESSAPP deployment administration page, click the listing for an Oracle Enterprise Scheduler cluster such as ESSAPP (ESSCluster), if one is available, or an individual Oracle Enterprise Scheduler server such as ESSAPP (ess_server1).

   Note:

   If you can select a cluster, that will allow you to search all servers in the cluster simultaneously. This is an advantage because each job request is logged only on the Oracle Enterprise Scheduler server where that request is actually processed.

3. If the job request for the action that you are troubleshooting appears in the list of Top 10 Running Job Requests, click its Request ID link to display the request details, then skip to Step 8.

4. In the context pane, from the dynamic Scheduling Service Group or Scheduling Service target menu, choose Job Requests > Search Job Request.

5. Use standard search techniques to locate the job request for the action that you are troubleshooting.

   For information about searching Oracle Enterprise Scheduler job requests, see Section 5.7.2.

6. In the search results, make a note of the Request ID value for the job that interests you.

7. In the search results, click the Request ID link to display the request details, then proceed to Step 8.

8. In the Request Details pane, select Request Log from the Action drop-down list.

   The Log Messages screen displays all the log messages for the ECID associated with the selected job request.

9. Click any Log File link to display a list of the related log messages in the View Log Messages screen.

   At this point, you can inspect the log messages on your screen or download the log file and search it for all messages containing the relevant ECID.

   Note:

   If the log level for a job request is set to FINER or FINEST when the job runs, then the diagnostic log entries for the job request are written to the log file specified by the AFLOG_FILENAME profile option, as well as to the request log file that is available to the user who submitted the job request.

17.5.3.3 Correlating Log Messages for SOA Services and BPEL Processes

If you need to correlate log messages that are related to SOA services or BPEL processes, you obtain the Execution Context Identifier (ECID) by first locating the relevant SOA composite instance.

To correlate log messages for SOA Services or BPEL processes using Fusion Applications Control:

1. In the navigation pane, expand the Farm listing, and then the SOA listing, and then the soa_infra listing, and then and select default.

2. In either pane, click the SOA composite instance name for which you want to locate log entries.

3. Make sure that your browser is set to allow the display of popup windows.

   For example, in Firefox, choose Options > Content from the Tools menu and either make sure that the Block pop-up windows checkbox is cleared or add any relevant hosts to Exceptions.

   Similarly, in Internet Explorer, fchoose Pop-up Blocker > Turn Off Pop-up Blocker from the Tools menu.

4. In the Recent Instances table, click the relevant Instance ID link to display the Flow Trace popup window.

   The ECID is displayed in the upper right corner of the window.

5. Make a note of the ECID value and close the Flow Trace popup window.

6. To view the relevant log entries using Fusion Applications Control, complete the following substeps. To view the log entries using the Oracle Weblogic Scripting Tool (WLST), skip to Step 7.

   1. In the navigation pane, select an appropriate target.

   2. From the dynamic target menu, choose Logs > View Log Messages.

   3. On the Log Messages page, in the Search area, click Add Field.

   4. In the popup list, select the ECID checkbox and click Add.

   5. Enter the ECID value in the ECID field.

   6. Adjust other search field values appropriately and click Search.

      Note:

      You can leave any search fields blank; blank fields are not used as search criteria.

      Skip the remainder of this procedure.

7. In the Oracle Weblogic Scripting Tool (WLST), enter a command like the following example, but substitute the ECID value that you previously noted.

   displayLogs(ecid='0000Hl9TwKUCslT6uBi8UH18lkWX000002')
   

17.5.3.4 Correlating Log Messages for Other Process Types

Many operations within Oracle Fusion applications do not use Oracle Enterprise Scheduler or SOA services or BPEL processes. For these operations, you can correlate log entries using the Execution Context Identifier (ECID).

To locate log entries that result from one particular Oracle Fusion application action that does not use Oracle Enterprise Scheduler or SOA services or BPEL processes:

1. Use Fusion Applications Control to search the log file that contains messages for the Oracle Fusion application in which the action was taken.

   Use any search criteria that will allow you to locate a log entry that is associated with the particular action. The associated ECID and RID values are included in the detailed log entry information. For more information about searching for particular information in an Oracle Fusion Applications log file, see Section 13.6.2.

2. When you have the ECID and RID values for the action, use one of the following methods to view all of the log entries associated with the action.

   • In Fusion Applications Control, on the Log Messages page, in the Search area, click Add Field to make sure that the ECID search field is displayed, then enter the ECID value in that field and click Search.

   • In Fusion Applications Control, on the page of Log Messages search results, with a relevant log message selected, select by ECID from the View Related Messages list.

   • In the Oracle Weblogic Scripting Tool (WLST), enter a command like the following example, but substitute the ECID value that you previously noted.

     displayLogs(ecid='0000Hl9TwKUCslT6uBi8UH18lkWX000002')
     

For more information about using ECIDs and RIDs to correlate log entries, see the "Correlating Messages Across Log Files and Components" section in the Oracle Fusion Middleware Administrator's Guide.

17.5.4 Downloading Log Files

As part of the process of troubleshooting Oracle Fusion Applications problems, an Oracle Support representative may occasionally request a copy of a particular log file. You can add downloaded log files to incidents in order to package those log files with incident information that you send to Oracle Support. For more information about adding files to an incident and packaging the incident into a zip file, see the "Packaging an Incident" section in the Oracle Fusion Middleware Administrator's Guide.

To download an Oracle Fusion Applications log file using Fusion Applications Control:

1. From the navigation pane, select the target for which you want to download a log file.

2. In the context pane, from the dynamic target menu, choose Logs > View Log Messages to display the log entries for the target you selected.

3. Use one of the following methods to locate the log file that you want to download:

   • Inspect the Log File column in the table of log entries that appears by default. If the column contains a link for the log file that you want to download, click the link and proceed to Step 4.

   • Perform a standard log file search using criteria that you expect to list the desired log file in the search results. When the Log File column contains a link for the log file that you want to download, click the link and proceed to Step 4.

   • Expand the Selected Targets section of the Search area and click the Target Log Files icon for the target most likely to be associated with the log file you want to download. In the list of log files that appears, select the file that you want to download and proceed to Step 4.

4. Click Download and specify where you want the log file to be saved.

17.6 Troubleshooting Using the Diagnostic Testing Framework

Some Oracle Fusion Applications diagnostic tests are designed to help you to monitor the health of your system. Other diagnostic tests are designed to help you troubleshoot problems or to gather data that Oracle Support may need when helping you to resolve a problem. For basic information about using Diagnostic Testing Framework features that are useful for both troubleshooting and monitoring system health, see Chapter 13. For information about the individual diagnostic tests that are provided with this release, see the Oracle Fusion Applications Common User Guide in Oracle Fusion Applications Help.

This section contains the following topics:

• Working with Automatically Launched Diagnostic Tests

• Registering Diagnostic Tests

• Sending Diagnostic Test Results to Oracle Support

• Purging the Results of Selected Diagnostic Test Runs

Note:

The user name that you use to sign in to an Oracle Fusion application affects which diagnostic operations are available to you. Be sure that you sign in using an account that is assigned to the job roles for the diagnostic operations that you need. For more information, see Section 13.9.1.

17.6.1 Working with Automatically Launched Diagnostic Tests

To make sure that certain diagnostic tests run automatically when certain error conditions occur, Oracle developers set the value of each test's APPS_MSG_ID tag to match the identifier of any error message that should trigger the automatic execution of that test. There is no configuration setting for disabling this automatic execution of diagnostic tests.

If an error has resulted in the automatic creation of an incident, then the results of any automatically run diagnostic tests are automatically associated with the incident. The identity of the user who received the error message is also recorded.

17.6.2 Registering Diagnostic Tests

Oracle supplies you with a number of diagnostic tests for Oracle Fusion Applications in seed data. Diagnostic tests that are supplied in seed data do not require you to register them.

However, if you report an Oracle Fusion application issue to Oracle Support, you may receive additional diagnostic tests to help resolve the issue. Depending on the exact circumstances, those diagnostic tests may or may not require registration. If registration is required for the additional diagnostic tests, Oracle will provide you with detailed registration instructions to follow.

17.6.3 Sending Diagnostic Test Results to Oracle Support

From time to time, you may run a diagnostic test and then want to send the results of the test to Oracle Support. To accomplish this, you attach the results of the test to the relevant incident.

Note:

You can also attach other types of files to incidents, such as log file excerpts or Readme.txt files containing information for Oracle Support. For more information about adding files other than diagnostic test results to incidents, see the "Packaging an Incident" section in the Oracle Fusion Middleware Administrator's Guide.

To attach diagnostic test results to an incident:

1. Sign in to the Oracle Fusion application that generated the incident, using an account that has the Diagnostic Advanced User duty role or the Diagnostic Administrator duty role.

2. In the Oracle Fusion application, choose Troubleshooting > Run Diagnostic Tests from the Help menu to display the Diagnostic Dashboard instance for that application. For more information, see Section 13.9.2.

3. If you are not already displaying the Diagnostic Test Run Status pane, expand the Tasks panel in the Regional area of your screen and click Run Status.

4. In the Diagnostic Test Run Status table, locate and select the test run that has the report that you want to add to an incident.

5. Click Add to Incident to display the Select Incident dialog box.

6. Click the Select ADR Home icon to the right of the ADR Home field.

7. In the Search and Select: ADR Home dialog box, locate and select the table row for the ADR_HOME and server that your Oracle Fusion application is currently using.

   If you wish, you can use any of the following techniques to locate the row that describes your ADR_HOME and server:

   • In the ADR Home field, enter any part of your ADR_HOME name and then click Search.

   • In the Server field, enter any part of the name of the server that is running your Oracle Fusion application and then click Search.

   • In both the ADR Home field and the Server field, enter the values you want to search for, then select either All (to display rows that match both search strings) or Any (to display rows that match either search string) and click Search.

   Note:

   In the Search and Select: ADR Home dialog box, search operations are case-sensitive.

8. With the correct table row highlighted, click OK to return to the Select Incident dialog box.

9. To display a list of known problems for your selected ADR home directory, click the Select Problem icon to the right of the Problem field.

10. In the Search and Select: Problem dialog box, locate and select the problem that describes the incident to which you want to add test results.

11. With the correct problem highlighted, click OK to return to the Select Incident dialog box.

12. From the Incident field, select the incident to which you want to add test results, and then click OK.

    The test report results are added to the incident and a confirmation message is displayed.

17.6.4 Purging the Results of Selected Diagnostic Test Runs

From time to time, you may want to remove diagnostic test run results from your database, to keep the Run Status table from becoming too large. For information about deleting diagnostic test results, see Section 13.10.11.